Three-dimensional scene reconstruction method and three-dimensional scene reconstruction apparatus

Abstract

The application discloses a three-dimensional scene reconstruction method and a three-dimensional scene reconstruction device. The three-dimensional scene reconstruction method comprises the following steps: acquiring first camera trajectory data, determining first direction representation data corresponding to each first observation view angle according to the first camera trajectory data, inputting each first direction representation data into a target image generation model to obtain a first generated image corresponding to each first observation view angle, generating target three-dimensional Gaussian splashing data of a target virtual scene based on first camera geometric parameters and the corresponding first generated image, and reconstructing the target virtual scene through the target three-dimensional Gaussian splashing data. In this way, a user only needs to configure a camera motion trajectory in a game engine to complete the whole process from two-dimensional image generation to three-dimensional scene reconstruction. Compared with a traditional artificial modeling method, the production efficiency of the three-dimensional scene is improved, and the generated image can be adapted to the game engine.

Description

Technical Field

[0001] This application relates to the field of 3D reconstruction technology, specifically to a 3D scene reconstruction method and a 3D scene reconstruction device. Background Technology

[0002] In traditional 3D scene reconstruction schemes, technicians typically first generate a set of multi-view images using a multi-view image generation model. These images are then input into camera pose estimation software to estimate the camera parameters for each image. Finally, the estimated parameters are used for 3D reconstruction. However, if the images output by the generation model have slight deviations in viewpoint consistency or geometric accuracy, the camera parameter estimation results will produce unpredictable drift, leading to distortion, misalignment, or holes in the reconstructed 3D scene. Summary of the Invention

[0003] This application provides a three-dimensional scene reconstruction method and a three-dimensional scene reconstruction device. By constraining the inference process of the image generation model with the native first camera geometric parameters provided by the game engine, the generated image is ensured to be compatible with the game engine, while improving the accuracy of three-dimensional reconstruction.

[0004] On one hand, embodiments of this application provide a three-dimensional scene reconstruction method, the method comprising: Acquire first camera trajectory data, wherein the first camera trajectory data includes first camera geometric parameters of the first virtual camera under multiple first observation views; Based on the first camera trajectory data, first direction characterization data corresponding to each first observation viewpoint is determined, wherein each first direction characterization data is used to indicate the direction in which the first virtual camera observes the virtual imaging plane through the corresponding first observation viewpoint; Input the first direction representation data corresponding to each first observation viewpoint into the pre-trained target image generation model to obtain the first generated image corresponding to each first observation viewpoint. Based on the first camera geometric parameters of each first observation viewpoint and the corresponding first generated image, target three-dimensional Gaussian splash data of the target virtual scene is generated, so as to reconstruct the target virtual scene through the target three-dimensional Gaussian splash data.

[0005] On the other hand, embodiments of this application provide a three-dimensional scene reconstruction apparatus, the apparatus comprising: The first acquisition module is used to acquire first camera trajectory data, wherein the first camera trajectory data includes the first camera geometric parameters of the first virtual camera under multiple first observation views; The first determining module is used to determine first direction characterization data corresponding to each first observation viewpoint based on the first camera trajectory data, wherein each first direction characterization data is used to indicate the direction of the first virtual camera when observing the virtual imaging plane through the corresponding first observation viewpoint. The first input module is used to input the first direction representation data corresponding to each first observation viewpoint into the pre-trained target image generation model to obtain the first generated image corresponding to each first observation viewpoint. The reconstruction module is used to generate target three-dimensional Gaussian splash data of the target virtual scene based on the first camera geometric parameters of each first observation view and the corresponding first generated image, so as to reconstruct the target virtual scene through the target three-dimensional Gaussian splash data.

[0006] On the other hand, embodiments of this application provide a computer-readable storage medium storing a computer program adapted for loading by a processor to execute the three-dimensional scene reconstruction method as described in any of the above embodiments.

[0007] On the other hand, embodiments of this application provide an electronic device, which includes a processor and a memory. The memory stores a computer program, and the processor executes the three-dimensional scene reconstruction method as described in any of the above embodiments by calling the computer program stored in the memory.

[0008] On the other hand, embodiments of this application provide a computer program product, including computer instructions, which, when executed by a processor, implement the three-dimensional scene reconstruction method as described in any of the above embodiments.

[0009] The 3D scene reconstruction method, 3D scene reconstruction device, computer-readable storage medium, electronic device, and computer program product provided in this application can acquire first camera trajectory data, determine first direction representation data corresponding to each first observation viewpoint based on the first camera trajectory data, input the first direction representation data corresponding to each first observation viewpoint into a pre-trained target image generation model to obtain a first generated image corresponding to each first observation viewpoint, and generate target 3D Gaussian splash data of the target virtual scene based on the first camera geometric parameters of each first observation viewpoint and the corresponding first generated image, so as to reconstruct the target virtual scene through the target 3D Gaussian splash data. This allows users to complete the entire process from 2D image generation to 3D scene reconstruction simply by configuring the camera motion trajectory within the game engine, improving the efficiency of 3D scene production compared to traditional manual modeling. Furthermore, by converting the native first camera geometric parameters provided by the game engine into first direction representation data and constraining the inference process of the image generation model with this first direction representation data, the compatibility of the generated image with the game engine is ensured. Meanwhile, the native first camera geometric parameters provided by the game engine can be directly reused in the 3D reconstruction process, which eliminates the step of using third-party software to infer the camera pose in the traditional solution, thereby avoiding the error caused by pose inference. This ensures the reconstruction accuracy of the 3D scene while taking into account the efficiency of 3D reconstruction. Attached Figure Description

[0010] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0011] Figure 1 This is one of the flowcharts illustrating the three-dimensional scene reconstruction method provided in the embodiments of this application.

[0012] Figure 2 This is the second flowchart illustrating the three-dimensional scene reconstruction method provided in the embodiments of this application.

[0013] Figure 3 This is the third flowchart illustrating the three-dimensional scene reconstruction method provided in the embodiments of this application.

[0014] Figure 4 This is a schematic diagram of the structure of the three-dimensional scene reconstruction device provided in the embodiments of this application.

[0015] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0016] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0017] This application provides a 3D scene reconstruction method, a 3D scene reconstruction device, a computer-readable storage medium, an electronic device, and a computer program product. Specifically, the 3D scene reconstruction method of this application can be executed by an electronic device, which can be a terminal or a server. The terminal can be a smartphone, tablet, laptop, smart TV, wearable smart device, smart vehicle terminal, etc. The terminal can also include a client, which can be a browser client, instant messaging client, or mini-program, etc. The server can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0018] It should be noted that, in this embodiment, the execution entity of the 3D scene reconstruction method can be a terminal device or a server. The terminal device can be a local terminal device or a client device in a cloud platform. This embodiment does not limit the type of execution entity.

[0019] For example, when this 3D scene reconstruction method runs on a terminal device, the terminal device may include a display screen and a processor. The display screen is used to present a graphical user interface (GUI) and receive instructions generated by the user interacting with the GUI. The processor is used to store applications, generate the GUI, respond to instructions, and control the display of the GUI on the display screen. When the user operates the GUI through the display screen, the GUI can control the local content of the terminal device in response to the received operation instructions. The terminal device can provide the GUI to the user in various ways, such as rendering it on the terminal device's display screen or presenting the GUI through holographic projection.

[0020] For example, when this 3D scene reconstruction method runs on a server, it can be implemented and executed based on a cloud system. The cloud system includes servers and client devices. The application's runtime and the graphical user interface (GUI) presentation are separate. The storage and execution of the 3D scene reconstruction method are completed on the server, while the GUI presentation is completed on the client. The client is mainly used for data reception, transmission, and GUI presentation. For example, the client can be a display device with data transmission capabilities located close to the user, such as a mobile terminal, television, computer, PDA, personal digital assistant, or head-mounted display device. However, the terminal device for data processing is the server in the cloud. During execution, the user operates the client to send instructions to the server. The server executes the instructions, encodes and compresses the GUI data, returns it to the client via the network, and finally, the client decodes and outputs the GUI.

[0021] It should be noted that, in this embodiment, the execution entity of the 3D scene reconstruction method can be a terminal device or a server. The terminal device can be a local terminal device or a client device in the aforementioned cloud system. This embodiment does not limit the type of execution entity.

[0022] The technical solution of this application will be described in detail below through specific embodiments. It should be noted that the following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.

[0023] 3D Gaussian Splatting (3DGS) technology represents a 3D scene as a collection of anisotropic 3D Gaussian distribution functions, each with its own attributes such as position, shape, color, and opacity. By performing differentiable rasterization rendering on these Gaussian functions, 3DGS can generate images from any viewpoint at extremely high frame rates while maintaining reconstruction quality comparable to mainstream methods such as neural radiation fields. This characteristic makes 3DGS particularly suitable for applications such as AI reconstruction of game scenes, virtual reality content creation, and large-scale realistic 3D modeling.

[0024] Traditional 3DGS reconstruction workflows typically begin with a set of images taken from multiple angles. Specifically, to achieve inverse optimization from a 2D image to a 3D Gaussian representation, the system must know the camera parameters for each image, namely the intrinsic and extrinsic parameter matrices. The intrinsic matrix determines the focal length and principal point, while the extrinsic matrix determines the camera's rotation and translation in the world coordinate system. In conventional practices, these camera parameters are not directly known but are derived from feature matching relationships between images using specialized motion reconstruction software such as COLMAP. The motion reconstruction software detects image feature points, establishes cross-viewpoint correspondences, triangulates to obtain a sparse point cloud, and then uses bundle adjustment to optimize and obtain the camera pose for each image. This step is not only computationally intensive—processing hundreds of images often takes hours or even longer—but also inevitably contains errors due to its dependence on the quality of image features. Furthermore, when the scene lacks texture, has drastic lighting changes, or contains a large number of repetitive patterns, the accuracy of the motion reconstruction software will significantly decrease, or even fail completely. Therefore, camera pose estimation is the most time-consuming and error-prone part of the entire 3DGS reconstruction process.

[0025] To reduce reliance on motion reconstruction software, researchers have begun exploring multi-view generation techniques based on video diffusion models. These include VideoFrom3D (3D Scene Video Generation via Image and Video Diffusion), StableVideo 3D (SV3D), Stable Virtual Camera (SEVA), 3D-Informed World-Consistent Video Generation with Precise Camera Control (GEN3C), and Create Anything in 4D with Multi-View Video Diffusion Models (CAT4D). The main idea behind these techniques is to train a diffusion model that can generate images of the scene from other specified viewpoints, given a single or a small number of input images.

[0026] However, these existing methods have revealed several serious flaws in practical applications. First, the way camera parameters are expressed is not precise enough. Taking SEVA as an example, this method uses Plücker ray coding to explicitly express the six-dimensional vector of each pixel ray, which theoretically has high spatial accuracy. However, this coding scheme encounters scale ambiguity problems when the input is a single view, that is, it cannot distinguish between the actual size of the scene and the actual distance to the camera, and must rely on manual scanning to determine a normalized scale range. More importantly, SEVA is designed entirely for general real-world shooting scenarios and does not consider the coordinate system conventions of game engines. Game engines typically use a left-handed coordinate system with the Y-axis pointing upwards, which differs from the right-handed coordinate system, Y-axis pointing downwards, or Z-axis pointing forwards commonly used in motion reconstruction software such as COLMAP. Therefore, directly using SEVA to process camera data output from game engines will introduce coordinate system mismatch errors, making it impossible for the generated frames to be precisely aligned with the engine coordinate system.

[0027] Furthermore, when distilling the generated multi-view frames into 3DGS representations using SEVA, iterative optimization of camera parameters is still required; known real camera parameters cannot be directly used to drive reconstruction. This means that a significant amount of computation has already been consumed in the generation stage, and then the camera must be re-estimated in the reconstruction stage, which is not only inefficient but also introduces additional optimization errors. Therefore, the geometric accuracy of the final 3DGS scene is affected by both the synthesis error of the generative model and the optimization error of the camera parameters.

[0028] Furthermore, the aforementioned multi-view generation techniques based on video diffusion models generally do not utilize real-world labeled data provided by game engines for training. For example, GEN3C is trained using real-world scene videos, and the depth information used during training comes from a monocular estimation network, making it difficult to guarantee the accuracy of the depth information. These techniques have not been specifically trained on data directly rendered by game engines, which has an unparalleled advantage: the engine can output the corresponding camera intrinsic and extrinsic parameter matrices and depth maps while generating each frame of the rendered image. Due to the lack of systematic learning from this high-quality data, the aforementioned multi-view generation techniques based on video diffusion models struggle to adapt to the variability of game visual styles. Whether it's a realistic style, a cartoon style, or a low-poly style, the models find it difficult to generate multi-view images that conform to the engine's distribution.

[0029] Furthermore, precise alignment between the generated frames and the engine coordinate system is almost always lacking in multi-view generation techniques based on video diffusion models. For example, VideoFrom3D proposed converting camera motion into an optical flow field to distort noise, thereby achieving conditional control in the latent space of the diffusion model. However, the proposers of VideoFrom3D pointed out that this operation of camera control in the compressed latent space suffers severe loss of precision, making it impossible to guarantee pixel-level alignment between the generated frames and the original engine coordinate system. If these generated frames are needed in subsequent 3D reconstruction, an additional coordinate system calibration step must be added, which is not only cumbersome but also further introduces errors.

[0030] In summary, regardless of whether the approach is through motion reconstruction software like COLMAP or combined with video diffusion models, traditional 3DGS 3D reconstruction solutions are affected by inaccurate camera parameters, mismatched coordinate systems, and the inability to utilize real-world annotation data. Among these, 3D reconstruction workflows implemented using motion reconstruction software are time-consuming and error-prone, while 3D reconstruction workflows based on multi-view generation can improve execution speed to some extent, but their accuracy in handling camera conditions is far from sufficient for high-quality reconstruction.

[0031] For the above issues, please refer to Figure 1 , Figure 1 This is a flowchart illustrating a three-dimensional scene reconstruction method provided in an embodiment of this application. It should be noted that the steps shown may be executed in a logical order different from that shown in the flowchart. The method includes: Step 110: Obtain first camera trajectory data, wherein the first camera trajectory data includes the first camera geometric parameters of the first virtual camera under multiple first observation views; Step 120: Based on the trajectory data of the first camera, determine the first direction representation data corresponding to each first observation viewpoint, wherein each first direction representation data is used to indicate the direction of the first virtual camera when observing the virtual imaging plane through the corresponding first observation viewpoint; Step 130: Input the first direction representation data corresponding to each first observation viewpoint into the pre-trained target image generation model to obtain the first generated image corresponding to each first observation viewpoint; Step 140: Based on the first camera geometric parameters of each first observation viewpoint and the corresponding first generated image, generate target three-dimensional Gaussian splash data of the target virtual scene, so as to reconstruct the target virtual scene through the target three-dimensional Gaussian splash data.

[0032] Specifically, when creating 3D assets for game scenes using manual modeling, the production cycle is long, heavily reliant on the modeling skills of art designers, and makes it difficult to quickly iterate through multiple versions of scene prototypes, failing to support the high-frequency, batch 3D scene review requirements during the game project initiation phase. If existing general artificial intelligence (AI) multi-view generation combined with 3DGS reconstruction is used for 3D production, the perspective and spatial arrangement of the AI-generated images rely on simple angle coding constraints, making it difficult to match the generated results with the engine's spatial coordinate system. The subsequent 3D reconstruction stage requires the use of third-party camera estimation software to infer the camera pose from the generated images. This inference process is time-consuming and inevitably introduces estimation errors. These errors propagate step-by-step, causing spatial distortion and coordinate misalignment in the reconstructed scene. Furthermore, the reconstructed 3D data requires additional coordinate system adaptation correction.

[0033] Based on this, this application provides a method for constraining the model inference process using camera trajectory data output by a game engine, so that the model inference process is constrained by the game engine's native data, thereby ensuring the effectiveness of the multi-view images output by the model, and finally completing the 3D scene reconstruction of 3D Gaussian splash reconstruction using multi-view images. Specifically, in this application embodiment, the electronic device can first extract the zero-error original camera trajectory data output by the game engine, and then transform the abstract camera geometric parameters into standardized and structured first-direction representation data to quantitatively define the camera observation lines corresponding to all pixels in a single virtual imaging plane. Then, the structured representation data is used as a constraint condition and fed into the pre-trained target image generation model, so that the model outputs the virtual scene simulation image obtained when the virtual camera observes the target virtual scene from various observation angles, which is the first generated image corresponding to each observation angle. Finally, the original camera geometric parameters are reused and combined with the generated image of the model to complete the 3D Gaussian splash reconstruction of the target virtual scene. Thus, the effectiveness of the multi-view images generated by the model can be ensured by the pre-constraint of the first-direction representation data, and the camera pose estimation is skipped by the native trajectory data output by the engine, thereby ensuring the efficiency of 3D reconstruction and effectively improving the quality of 3D scene generation.

[0034] In some embodiments, the first camera trajectory data can be understood as a dataset natively output by the game engine, which fully records the spatial motion trajectory information of the first virtual camera under multiple first observation perspectives within the target virtual scene, and includes at least the first camera geometric parameters corresponding to each perspective.

[0035] In some examples, for creating distant mountain scenes in a game engine, the first camera trajectory data output by the engine contains information on 24 shooting points evenly distributed along a circular trajectory, with each point accompanied by the corresponding raw values ​​of camera intrinsic and extrinsic parameters.

[0036] In some embodiments, a game engine can be understood as a game development engine software that has the ability to build virtual scenes and render real-time images, and can accurately export a complete set of geometric parameters of a built-in virtual camera while rendering the output image. The camera parameters natively output by the engine have no human estimation error.

[0037] In some examples, the game engine is Unreal Engine 5 (UE5), which uses a left-handed coordinate system and can export full-track camera parameters and accompanying depth data during the scene preview stage.

[0038] In some embodiments, the first virtual camera can be understood as a digital virtual shooting unit manually configured in the virtual editing environment of the game engine, which can freely define its spatial position, orientation, and framing focal length to define the framing boundaries of each first observation perspective.

[0039] In some examples, the first virtual camera is a standardized viewfinder camera configured in the UE5 mountain scene, arranged according to three framing rules: low elevation, eye level, and high elevation, forming a complete surround shooting group.

[0040] In some embodiments, the first observation perspective can be understood as an independent shooting position selected by the first virtual camera in the three-dimensional space of the target virtual scene. Each position corresponds to a unique framing orientation and image perspective rules, and multiple perspectives together constitute a complete scene surround shooting system.

[0041] In some examples, the Vision Mountain Project sets up a total of 24 first-view perspectives, including 12 eye-level perspectives, 8 low-angle perspectives, and 4 high-altitude perspectives.

[0042] In some embodiments, the first camera geometric parameters can be understood as a set of parameters that distinguish between the camera's optical properties and spatial position, and are divided into two categories: camera intrinsic parameters and camera extrinsic parameters. The intrinsic parameters define optical indicators such as the camera lens focal length and the principal point of imaging, while the extrinsic parameters define the camera's rotation and translation position information in the world coordinate system. All data is generated by the engine.

[0043] In some examples, the first camera geometry parameters corresponding to a single viewpoint include the intrinsic parameters of the focal length (fx=1200, fy=1200) and the principal imaging point parameters (cx=512, cy=512), and the extrinsic parameters of the camera rotation matrix around the three axes and the world space translation coordinates.

[0044] In some embodiments, the first direction characterization data can be understood as data that can define the camera observation line of sight corresponding to each pixel in the virtual imaging plane.

[0045] In some embodiments, the first direction characterization data can be generated based on the first camera geometric parameters of each first observation viewpoint, after coordinate system adaptation and pixel line-of-sight conversion.

[0046] In one example, the first direction representation data is the observation direction representation corresponding to each pixel on a virtual imaging plane defined by the geometric parameters of the first camera and a preset resolution. Exemplarily, the first direction representation data corresponding to a single viewpoint can be a structured image with H×W×6 dimensions. For example, when the image resolution is 1024×1024, an array format of 1024×1024×6 is generated, with each pixel location storing a set of six-dimensional line-of-sight representation values.

[0047] In some embodiments, the target image generation model can be understood as an AI video diffusion generation model fine-tuned and trained on an engine rendering dataset, capable of receiving first-direction representation data as generation constraints and generating scene images from the corresponding observation perspective according to the line-of-sight constraint logic.

[0048] In some examples, the target image generation model is based on the open-source Stable Virtual Camera (SEVA) diffusion model, and fine-tuned using UE5 realistic mountain rendering samples with low-rank adaptation (LoRA).

[0049] In some embodiments, the first generated image can be understood as a two-dimensional scene image output by the target image generation model after receiving the first direction representation data of the corresponding viewpoint, in accordance with the line-of-sight constraint rules. The perspective relationship and scene arrangement of the image can match the camera geometric parameters under the corresponding first observation viewpoint. It can also be understood that the first generated image is a simulation of the camera image taken by the virtual camera when observing the target virtual scene from the first observation viewpoint corresponding to the first direction representation data. In other words, the embodiments of this application can generate a simulation of the camera image taken by the virtual camera when observing the target virtual scene from the first observation viewpoint through the target image generation model. Thus, before the completion of the three-dimensional reconstruction of the target virtual scene, a sufficient number of image simulations can be obtained through the target image generation model to provide sufficient image support for the three-dimensional reconstruction of the target virtual scene.

[0050] In some examples, after 24 sets of view parameters are input into the model, 24 first generated images of distant mountains are output in sequence. The perspective of the mountains and the occlusion relationship of distant and near objects in each image are consistent with the actual images captured by the camera in UE5.

[0051] In some embodiments, the target virtual scene can be understood as a digital virtual environment built in a game engine that requires the use of multi-view images to reconstruct three-dimensional assets.

[0052] In some embodiments, the target virtual scene may include a main scene structure that has been designed, as well as blank expansion areas that need to be supplemented by design.

[0053] In some examples, the target virtual scene is a distant wilderness and mountainous area surrounding a completed main city in an open-world game, without a completed 3D model.

[0054] In some embodiments, 3D Gaussian splash data can be understood as 3D scene structured data with a massive number of Gaussian particles as the basic storage unit. Each Gaussian particle records attributes such as spatial coordinates, color, opacity, shape covariance, and color spherical harmonic coefficient. The 3D Gaussian splash data can be used to directly render a complete 3D scene in the engine.

[0055] In some examples, the 3D Gaussian splash data obtained after reconstructing the distant mountains consists of millions of Gaussian primitives, which can be directly imported into UE5 to enable 360° free rotation and browsing of the scene.

[0056] To more clearly illustrate the three-dimensional scene reconstruction method provided in the embodiments of this application, please refer to the following exemplary description: The electronic device first acquires the trajectory data of the first camera for 3D reconstruction. This data is directly exported by the game engine and includes a complete set of first camera geometric parameters corresponding to multiple first observation perspectives.

[0057] Then, the electronic device performs processing on the geometric parameters of the first camera, such as coordinate system matching transformation, pixel normalization mapping, and observation line solution, to generate first direction representation data corresponding to each viewpoint. This transforms the original numerical camera parameters of the engine into structured intermediate data that can quantify the pixel line direction, realizing the format conversion of the original engine parameters into conditional data readable by the AI ​​model.

[0058] Next, the first-direction representation data is input into the target image generation model. The model generates multi-view images under the constraints of the first-direction representation data, and finally outputs the first generated image corresponding to each viewpoint.

[0059] Finally, the geometric parameters of the first camera from each perspective obtained during the acquisition phase are reused, and combined with all the first generated images output by the model, the target virtual scene is initialized and iteratively optimized with three-dimensional Gaussian splash data. The original camera parameters are kept fixed throughout the process without optimization, and only the property parameters of the Gaussian particles themselves are adjusted. Finally, complete target three-dimensional Gaussian splash data is generated, and the three-dimensional reconstruction of the target virtual scene is completed through this data.

[0060] For example, in the UE5 open-world distant mountain project, the device reads first camera trajectory data from 24 surround view perspectives from the engine. This first camera trajectory data generates 24 first-direction representation data images of 1024×1024×6 pixels. These 24 first-direction representation data images are then input into an image generation model. The model generates 24 first-generation images, i.e., mountain images, that match the camera geometry in the first camera trajectory data. Finally, Gaussian point clouds are initialized using the engine's native camera parameters and the first-generation images. The Gaussian primitive properties are iteratively optimized, ultimately outputting 3D Gaussian splash scene data of mountains that can be directly imported into UE5.

[0061] Thus, in this embodiment, first camera trajectory data can be obtained, and based on the first camera trajectory data, first direction representation data corresponding to each first observation viewpoint can be determined. The first direction representation data corresponding to each first observation viewpoint is then input into a pre-trained target image generation model to obtain a first generated image corresponding to each first observation viewpoint. Based on the first camera geometric parameters of each first observation viewpoint and the corresponding first generated image, target 3D Gaussian splash data of the target virtual scene is generated. This allows the target virtual scene to be reconstructed using the target 3D Gaussian splash data. This enables users to complete the entire process from 2D image generation to 3D scene reconstruction simply by configuring the camera motion trajectory within the game engine. Compared to traditional manual modeling, the efficiency of 3D scene production is improved. Furthermore, by converting the native first camera geometric parameters provided by the game engine into first direction representation data, and using this first direction representation data to constrain the inference process of the image generation model, the compatibility between the generated image and the game engine is ensured. Meanwhile, the native first camera geometric parameters provided by the game engine can be directly reused in the 3D reconstruction process, which eliminates the step of using third-party software to infer the camera pose in the traditional solution, thereby avoiding the error caused by pose inference. This ensures the reconstruction accuracy of the 3D scene while taking into account the efficiency of 3D reconstruction.

[0062] Please see Figure 2 , Figure 2 This is a flowchart illustrating a three-dimensional scene reconstruction method provided in an embodiment of this application. It should be noted that the steps shown may be executed in a logical order different from that shown in the flowchart. Specifically, in some embodiments provided in this application, the first camera geometric parameters include first camera intrinsic parameters and first camera extrinsic parameters. Therefore, step 120 above includes: Step 1200: Perform coordinate system matching processing on the first coordinate system corresponding to the first camera trajectory data so that the coordinate axes, rotation, origin position and scale of the first coordinate system are matched with the predetermined second coordinate system to obtain the third coordinate system; Step 1201: Determine the first direction representation data corresponding to each first observation viewpoint based on the first camera intrinsic parameters corresponding to each first observation viewpoint and the first camera extrinsic parameters corresponding to each first observation viewpoint in the third coordinate system.

[0063] Specifically, considering that different types of game engines have different spatial coordinate system specifications, the native coordinate systems of various engines have inherent differences in coordinate axis orientation, left-hand and right-hand judgment rules, scene world measurement scale and the standard coordinate system used with the image generation model. If the original camera intrinsic and extrinsic parameters under the engine's native coordinate system are directly used to calculate the observation direction, the inconsistency between the two coordinate references will lead to systematic numerical deviations and ultimately distort the generated direction representation data.

[0064] Based on this, in some embodiments provided in this application, for the native coordinate system of the game engine, that is, the first coordinate system corresponding to the first camera trajectory data, the electronic device can perform matching transformation on the coordinate axes, rotation, origin position and scale of the first coordinate system to generate a third coordinate system that is consistent with the specifications of the second coordinate system, and convert the original camera extrinsic parameters output by the engine into the third coordinate system. Then, by jointly calculating the camera intrinsic parameters under the same observation view and the camera extrinsic parameters after coordinate transformation, the first direction characterization data is obtained, thus avoiding calculation errors caused by coordinate system conflicts from the source.

[0065] In some embodiments, the first camera intrinsic parameters The inherent imaging parameters of the first virtual camera are determined by the equivalent optical imaging characteristics of the camera. The values ​​do not change with the spatial position and orientation of the camera. They can be used to establish the mapping relationship between pixel coordinates and camera imaging space. The parameters mainly include focal length, principal point coordinates of the imaging plane, etc.

[0066] In some examples, when Unreal Engine and Unity Engine generate virtual cameras internally, they automatically generate the first camera intrinsics for the corresponding camera. , K It includes fx, fy, cx, and cy, where fx and fy represent the imaging focal lengths in two orthogonal directions, and cx and cy represent the principal point offset coordinates of the virtual imaging plane.

[0067] In some embodiments, the first camera extrinsic parameters The rotation matrix is ​​used to characterize the pose information of the virtual camera in the scene space. R Translation vector t It consists of components that change numerically in real time as the virtual camera's viewing angle moves, and is used to mark the camera's spatial position and orientation in the corresponding coordinate system.

[0068] In some examples, when the game engine switches the shooting position of the virtual camera, it will update the camera's rotation and translation data simultaneously. This updated data is the first camera extrinsic parameter for the corresponding observation viewpoint.

[0069] In some embodiments, the first coordinate system can be understood as the native spatial coordinate system preset by the game engine, and each game engine customizes the coordinate axis arrangement, chirality rules and spatial measurement units.

[0070] In some examples, Unreal Engine 5 uses a left-handed coordinate system with the X-axis pointing forward, the Y-axis pointing to the right, and the Z-axis pointing upward by default, while the Unity engine uses a left-handed coordinate system with the Z-axis pointing forward.

[0071] In some embodiments, coordinate system matching processing can be understood as a series of spatial transformation operations performed to address the differences between the source coordinate system and the target coordinate system, in order to align the source coordinate system to the target reference coordinate system and eliminate the differences between the two in coordinate axes, rotation direction, origin, and scale (or scale).

[0072] In some examples, coordinate system matching processing may include four operations: coordinate axis permutation, rotation transformation, origin translation, and scale normalization. For example, the coordinate axis order is rearranged using a permutation matrix, the left-handed coordinate system is converted to a right-handed coordinate system using a reflection matrix, the origin is aligned to the center of the scene bounding box using a translation vector, and the scene scale is normalized to a uniform range using a scaling factor.

[0073] In some embodiments, the second coordinate system can be understood as a pre-determined standard visual reference coordinate system, which is a unified spatial reference used for training the orientation representation encoding algorithm and the target image generation model, and usually follows the general coordinate conventions in the field of computer vision.

[0074] In some examples, the second coordinate system can adopt the right-handed coordinate system convention commonly used in the field of computer vision, with the Z-axis pointing forward, the X-axis pointing to the right, and the Y-axis pointing downward.

[0075] In some embodiments, the third coordinate system can be understood as a unified coordinate system obtained after coordinate system matching. The coordinate axes, rotation, origin position, and spatial scale of the third coordinate system are completely consistent with those of the second coordinate system, and it serves as a unified spatial reference for subsequent calculation of directional representation data.

[0076] In some examples, the third coordinate system can achieve standardized transformation between different engine native coordinate systems. Regardless of the convention of the input engine coordinate system, the final output third coordinate system follows a unified standard specification.

[0077] To more clearly illustrate the implementation process of solving the direction representation data based on coordinate system transformation in the embodiments of this application, please refer to the following exemplary description: First, through the coordinate axis permutation matrix Adjust the arrangement of the three axes of the first coordinate system to match the coordinate axis directions of the second coordinate system.

[0078] Then, using the reflection matrix This completes the rotation (or chirality) transformation of the first coordinate system, converting it from a left-handed to a right-handed system.

[0079] Then, world-scale normalization scaling is performed using scene bounding boxes to make the scale (or proportion) of the first coordinate system the same as that of the second coordinate system, thus completing the matching process between the two coordinate systems.

[0080] Finally, all the original first camera extrinsic parameters were converted to the third coordinate system, thus unifying the extrinsic parameter coordinate reference.

[0081] As an example, the camera extrinsic parameters in the third coordinate system can be expressed by formula (1), that is: (1) in, The extrinsic parameters of the first camera in the third coordinate system. R ' is the rotation matrix in the third coordinate system. t ' is the translation vector in the third coordinate system.

[0082] Thus, in this embodiment of the application, the first coordinate system corresponding to the first camera trajectory data can be subjected to coordinate system matching processing so that the coordinate axes, rotation, origin position and scale of the first coordinate system are matched with the predetermined second coordinate system to obtain the third coordinate system. Based on the first camera intrinsic parameters corresponding to each first observation view and the first camera extrinsic parameters corresponding to each first observation view in the third coordinate system, the first direction representation data corresponding to each first observation view is determined. This can unify the coordinate calculation benchmark of the camera intrinsic and extrinsic parameters, thereby eliminating the direction representation data calculation error caused by the difference in specifications between the engine coordinate system and the model standard coordinate system.

[0083] In some embodiments provided in this application, step 1201 includes: determining the mapping position of each pixel in the virtual imaging plane in the third coordinate system based on the intrinsic parameters of the first camera corresponding to the first observation viewpoint; determining the observation direction characterization parameter corresponding to each pixel in the virtual imaging plane based on the mapping position of each pixel in the virtual imaging plane and the extrinsic parameters of the first camera in the third coordinate system; and obtaining the first direction characterization data corresponding to each first observation viewpoint based on the observation direction characterization parameter corresponding to each pixel in the virtual imaging plane corresponding to each first observation viewpoint. The observation direction characterization parameter is used to indicate the observation direction of the first virtual camera when observing a pixel at the corresponding first observation viewpoint.

[0084] Specifically, to ensure the detail accuracy and effectiveness of the orientation representation data, in some embodiments provided in this application, the electronic device can first complete the point mapping from the virtual imaging plane pixels to the third coordinate system through the camera intrinsic parameters, then combine the standardized camera extrinsic parameters to solve the observation orientation representation parameters pixel by pixel, and finally integrate the full pixel parameters to form complete first orientation representation data. This pixel-refined calculation method improves the accuracy and effectiveness of the orientation representation data.

[0085] In some embodiments, the virtual imaging plane is the equivalent two-dimensional imaging carrier corresponding to the first virtual camera. It is the reference plane for the projection imaging of the three-dimensional scene in the virtual imaging system, and all scene content projected by the camera will fall on the pixel points of this plane.

[0086] In some examples, the virtual camera configuration in Unreal Engine allows for the creation of a custom rectangular virtual imaging plane with a resolution of 1920×1080, which serves as the imaging medium for the scene images.

[0087] In some embodiments, the pixel mapping position refers to the coordinates of a single two-dimensional pixel on the virtual imaging plane, which, after imaging projection conversion, corresponds to the three-dimensional coordinate point in the three-dimensional space of the third coordinate system, thereby realizing the point conversion of the pixel from the two-dimensional imaging space to the unified three-dimensional coordinate system.

[0088] In some examples, a pixel with coordinates (200, 300) in the virtual imaging plane can be mapped to a pixel position (x=1.2, y=2.5, z=6.3) in the third coordinate system after conversion via the intrinsic parameter matrix.

[0089] In some embodiments, the observation direction characterization parameter is a parameter information used to quantify the spatial observation ray corresponding to a single pixel. It can record the spatial direction from the camera's optical center to the pixel mapping point in a standardized mathematical form, and can mark the spatial line of sight from the camera to that pixel.

[0090] In some examples, the observation direction representation parameter can be stored in the form of a six-dimensional vector to store spatial ray information, recording the ray direction and moment vector respectively, representing the observation direction corresponding to a single pixel.

[0091] To more clearly illustrate the process of calculating the first direction characterization data step by step from internal and external parameters in the embodiments of this application, please refer to the following exemplary description: First, retrieve the intrinsic parameters of the first camera corresponding to the current viewpoint. Then, using the pinhole imaging projection rules, perform inverse projection operations on all pixels in the 1920×1080 resolution virtual imaging plane one by one, converting the two-dimensional plane coordinates of each pixel into three-dimensional coordinates in the third coordinate system, and obtaining the pixel mapping positions corresponding to all pixels in sequence.

[0092] Then, the extrinsic parameters of the first camera stored in the third coordinate system from the same perspective are retrieved. The optical center coordinates of the current virtual camera in the third coordinate system are solved by the rotation and translation matrices in the extrinsic parameters. The optical center is used as the starting point of the ray and the mapping position corresponding to each pixel is used as the ending point of the ray. After vector normalization, the observation direction characterization parameters of each pixel are calculated one by one.

[0093] Finally, following the original row and column pixel arrangement order of the virtual imaging plane, the observation direction representation parameters of all pixels under the current observation view are systematically spliced ​​and integrated to form structured data that matches the image resolution. This structured data is the first direction representation data corresponding to the current observation view.

[0094] Thus, in this embodiment, the mapping position of each pixel in the virtual imaging plane in the third coordinate system can be determined according to the intrinsic parameters of the first camera corresponding to the first observation viewpoint. The observation direction representation parameters corresponding to each pixel in the virtual imaging plane can be determined according to the mapping position of each pixel in the virtual imaging plane and the extrinsic parameters of the first camera in the third coordinate system. The first direction representation data corresponding to each first observation viewpoint can be obtained according to the observation direction representation parameters corresponding to each pixel in the virtual imaging plane corresponding to each first observation viewpoint. This achieves fine calculation of pixel-level direction representation data, and the accuracy and effectiveness of the first direction representation data can be guaranteed.

[0095] In some embodiments provided in this application, the step of determining the mapping position of each pixel in the virtual imaging plane in the third coordinate system based on the intrinsic parameters of the first camera corresponding to the first observation viewpoint includes: normalizing the position of each pixel in the virtual imaging plane based on the intrinsic parameters of the first camera corresponding to the first observation viewpoint to obtain the normalized position of each pixel in the virtual imaging plane; and mapping each pixel in the virtual imaging plane to the third coordinate system based on the normalized position of each pixel in the virtual imaging plane to obtain the mapping position of each pixel in the virtual imaging plane.

[0096] Specifically, the imaging resolution and screen size of different virtual cameras in the game engine are significantly different. The original position of pixels on the virtual imaging plane is marked in units of pixel grids. The actual physical scale of pixel coordinates is different under different resolutions. If the unprocessed original pixel coordinates are directly used to perform spatial mapping to the third coordinate system, the inconsistency of coordinate scale will cause conversion deviation and result in the error in calculating the pixel mapping position.

[0097] Based on this, in some embodiments provided in this application, the original coordinates of all pixels are first normalized using the intrinsic parameters of the first camera at the current viewpoint to eliminate scale differences caused by resolution and generate normalized positions of uniform specifications; then, the normalized positions are used as the calculation benchmark to complete the spatial mapping of pixels from the two-dimensional imaging plane to the third coordinate system and obtain the mapping position corresponding to each pixel, thereby avoiding conversion deviations caused by inconsistent coordinate scales.

[0098] In some embodiments, the position of a pixel in the virtual imaging plane can be understood as the original coordinates of the pixel on the two-dimensional virtual imaging plane. The upper left corner of the image or a preset origin is used as the coordinate reference, and two-dimensional coordinate values ​​are generated by the image row and column index. The coordinate measurement unit is a single pixel.

[0099] In some examples, for a 1920×1080 resolution engine-rendered image, the original pixel position of the first pixel in the top left corner of the image can be recorded as coordinates (0,0), and the horizontal last pixel position can be recorded as (1919,1079).

[0100] In some embodiments, normalization is a mathematical transformation of the original pixel coordinates using the focal length and principal point parameters in the intrinsic parameters of the first camera. This process is used to remove scale interference caused by image resolution and unify the range of coordinate values.

[0101] In some examples, the original pixel coordinates of different frame sizes are converted to the [-1,1] interval using the camera intrinsic parameters fx, fy focal length and cx, cy principal point values. This conversion process is called normalization.

[0102] In some embodiments, the normalized position is a dimensionless two-dimensional coordinate obtained by normalizing the original pixel coordinates. It is not affected by image resolution or image size, and the pixel coordinates of all images are uniformly within the same standard value range.

[0103] In some examples, after normalization, the center pixels of images with resolutions of 1920×1080 and 1280×720 will eventually have completely consistent normalized position coordinates.

[0104] In some embodiments, the mapped position is a three-dimensional spatial coordinate in a third coordinate system obtained by combining the normalized two-dimensional pixel coordinates with imaging geometry rules, representing the spatial starting point of the pixel line of sight in a unified coordinate system.

[0105] In some examples, after spatial projection transformation, the normalized coordinates of a pixel are transformed into three-dimensional coordinates (2.3, 1.1, 5.6) in the third coordinate system. These three-dimensional coordinates are the mapped position of the pixel.

[0106] To more clearly illustrate the process of obtaining the pixel mapping position in the embodiments of this application, please refer to the following exemplary description: Assuming a 1920×1080 resolution scene rendering generated by Unreal Engine is used as the processing object, the electronic device first extracts the pixel positions of all pixels p within the virtual imaging plane. .

[0107] Subsequently, the intrinsic parameters of the first camera corresponding to that observation angle were retrieved. The original pixel coordinates of the entire image are batch-normalized using the focal length and principal point parameters included in the intrinsic parameters, eliminating the scale constraints imposed by the 1920×1080 image size and generating normalized positions for all pixels with a unified standard. The process can be shown in formula (2), that is: (2) After obtaining the normalized positions of all pixels, the spatial coordinates of the normalized positions in the two-dimensional format are converted using the principle of pinhole imaging projection geometry. Each pixel is then mapped to the third coordinate system after coordinate unification, and finally the three-dimensional mapping position of each pixel in the third coordinate system is output.

[0108] Thus, in this embodiment, the position of each pixel in the virtual imaging plane can be normalized according to the intrinsic parameters of the first camera corresponding to the first observation viewpoint to obtain the normalized position of each pixel in the virtual imaging plane. Based on the normalized position of each pixel in the virtual imaging plane, each pixel in the virtual imaging plane is mapped to the third coordinate system to obtain the mapped position of each pixel in the virtual imaging plane. In this way, the pixel calculation benchmark of images with different resolutions is unified through the normalization operation, avoiding the coordinate scale deviation caused by the difference in image size.

[0109] In some embodiments provided in this application, the step of determining the observation direction characterization parameter corresponding to each pixel in the virtual imaging plane based on the mapping position of each pixel in the virtual imaging plane and the first camera extrinsic parameters in the third coordinate system includes: determining the optical center position of the first virtual camera corresponding to the first observation viewpoint based on the first camera extrinsic parameters corresponding to the first observation viewpoint in the third coordinate system; constructing a direction from the optical center position of the first virtual camera to the mapping position of each pixel based on the mapping position of each pixel in the virtual imaging plane corresponding to the first observation viewpoint and the optical center position of the first virtual camera corresponding to the first observation viewpoint, thereby obtaining the observation direction characterization parameter corresponding to each pixel in the virtual imaging plane.

[0110] Specifically, to avoid deviations in the calculated observation direction characterization parameters due to camera optical center coordinate offset, in some embodiments provided in this application, the electronic device can accurately solve the spatial coordinates of the camera optical center based on the first camera extrinsic parameters after coordinate system correction, thereby determining the starting point of all imaging rays. Then, using the camera optical center coordinates and the mapping points of each pixel in the third coordinate system as two endpoints, a spatial ray pointing from the optical center to the pixel point is constructed through spatial vector operations. The observation direction characterization parameters corresponding to each pixel are generated through this ray data, ensuring the accuracy of parameter calculation from the source of the imaging ray.

[0111] In some embodiments, the optical center position of the first virtual camera can be understood as the three-dimensional coordinates corresponding to the optical imaging center point of the virtual camera in the three-dimensional space of the third coordinate system. It is the unified starting origin of all pixel observation rays and can be obtained by inverse operation from the standardized camera extrinsic parameters.

[0112] In some examples, after Unreal Engine exports camera parameters and completes coordinate transformation to form a third coordinate system, the rotation and translation matrices in the extrinsic parameters can be used to perform reverse operations to obtain three-dimensional coordinates in the format of (x,y,z). These coordinate values ​​are the optical center positions of the first virtual camera in the current viewpoint.

[0113] In some embodiments, the observation direction characterization parameter can be understood as numerical information used to quantitatively characterize the imaging line of sight corresponding to a single pixel, which can completely describe the three-dimensional ray spatial orientation that originates from the camera optical center and passes through the corresponding pixel.

[0114] In some examples, the observation direction characterization parameters can be stored in the form of a normalized three-dimensional direction vector or a six-dimensional Plück ray vector format to fully record the spatial orientation information of the ray.

[0115] In some embodiments, constructing the direction from the optical center position to the pixel can be understood as a data processing operation that generates a mathematical expression for the spatial ray between two points by using two sets of spatial points: the three-dimensional coordinates of the optical center and the three-dimensional coordinates of the pixel mapping.

[0116] In some examples, the original spatial vector between two points can be generated by subtracting the camera optical center coordinates from the pixel mapping coordinates using vector subtraction, and then the orientation is constructed after normalization.

[0117] To more clearly illustrate the process of obtaining the observation direction characterization parameters in the embodiments of this application, please refer to the following exemplary description: Assuming Unreal Engine 5 is a game engine, and the matching and conversion from the engine's native first coordinate system to the second coordinate system has been completed, generating a unified standard third coordinate system, and the primary camera extrinsic parameters of each observation point have been converted to the third coordinate system, then the electronic device can select any primary observation point and retrieve the primary camera extrinsic parameters of that point stored in the third coordinate system. Relying on the rotation matrix inside the extrinsic parameters R 'and translation vector t 'Perform reverse coordinate calculations to obtain the three-dimensional spatial coordinates o of the optical center of the first virtual camera in the third coordinate system from this perspective, as shown in formula (3), that is: (3) Then, the pixel mapping position data of the whole image from this perspective, which was calculated in the previous steps, is retrieved. This type of data consists of the three-dimensional coordinates d of each pixel in the third coordinate system, as shown in formula (4), that is: (4) Each pixel in the image is traversed sequentially. The camera optical center coordinates o are used as the starting point of the vector, and the mapping position coordinates d corresponding to a single pixel are used as the ending point of the vector. The spatial ray direction between the two points is calculated through spatial vector calculation. Then, the generated vector is normalized to generate the observation direction characterization parameter corresponding to the current pixel, as shown in formula (5). (5) The above traversal calculation process is continuously repeated until the parameters of all pixels on the virtual imaging plane H×W under the current observation view are solved. The sum of all pixel parameters is then used to obtain the Plücker ray map P, where P∈R. H×W×6 That is, the first direction of the data.

[0118] Thus, in this embodiment, the optical center position of the first virtual camera corresponding to the first observation viewpoint can be determined based on the extrinsic parameters of the first camera corresponding to the first observation viewpoint in the third coordinate system. Based on the mapping position of each pixel in the virtual imaging plane corresponding to the first observation viewpoint and the optical center position of the first virtual camera corresponding to the first observation viewpoint, the direction from the optical center position of the first virtual camera to the mapping position of each pixel is constructed, thereby obtaining the observation direction characterization parameter corresponding to each pixel in the virtual imaging plane. Thus, the observation direction parameter can be calculated using reliable optical center parameters, ensuring the accuracy of the calculation of the observation direction parameter from the starting point of the imaging ray.

[0119] In some embodiments provided in this application, step 130 includes: concatenating the first direction representation data with the input feature map of the first layer of the denoising network of the target image generation model along the channel dimension; and injecting the global pooling features of the first direction representation data into at least one residual block of the denoising network through an adaptive layer normalization module.

[0120] Specifically, when the target image generation model is a diffusion model, in order to ensure the model inference effect, in some embodiments provided in this application, a dual-branch parallel feature injection strategy is adopted. One path concatenates the first direction representation data and the first layer input feature map at the input end of the denoising network to achieve pixel-level fine-grained conditional embedding; the other path extracts global pooling features from the first direction representation data and embeds global geometric constraints into the residual block of the denoising network through the adaptive layer normalization module, thus completing the full network injection of direction representation data from two dimensions: local pixels and global features.

[0121] In some embodiments, the denoising network is the backbone network of the target image generation model, which completes the iterative removal of noise features through multiple layers of convolution and residual units.

[0122] In some examples, the overall architecture of the diffusion model used for multi-view image generation uses U-Net as the denoising network, and the entire image denoising and feature extraction operations are completed within this network.

[0123] In some embodiments, the input feature map of the first layer of the denoising network can be understood as the original tensor data received by the first layer of the denoising network, which is generally the noisy features in the latent space and stored in the form of a three-dimensional tensor with height, width, and multiple channels.

[0124] In some examples, the latent variable features sampled by the diffusion model at each denoising time step are preprocessed and used as the input feature map of the first layer of the denoising network.

[0125] In some embodiments, channel dimension stitching can be understood as superimposing information only in the channel direction of the feature, without changing the original height and width of the feature map, which allows the original image features and camera geometric features to be bound one-to-one at the same pixel position.

[0126] In some examples, the input feature map has a size of H×W×C1, the first direction representation data feature has a size of H×W×C2, and the channel concatenation results in a fused feature with a size of H×W×(C1+C2).

[0127] In some embodiments, a global pooling feature can be understood as a one-dimensional vector obtained by performing a global pooling operation on the entire first-direction representation feature map.

[0128] In some examples, global average pooling is performed on the H×W×6 dimensional orientation representation feature map, and the final output is a one-dimensional vector of length 6, which is the global pooling feature.

[0129] In some embodiments, the Adaptive Layer Normalization (AdaLN) module can be understood as a type of normalization functional component that can dynamically generate scaling coefficients and offsets based on the conditional features of external inputs, and rely on parameters to adjust the data distribution pattern of features within the network in real time.

[0130] In some examples, the AdaLN module receives global pooling features and dynamically generates mean and variance correction parameters to adjust the normalization results of features within the residual block.

[0131] In some embodiments, residual blocks are the basic computational units stacked in the denoising network. They alleviate the gradient vanishing problem in deep networks by using skip connection structures and are responsible for extracting abstract features from the middle layers of the image.

[0132] In some examples, the underlying modules that are repeatedly stacked at the U-Net encoder and decoder are residual blocks.

[0133] To more clearly illustrate the implementation process of dual-path injection of directional representation data in the embodiments of this application, please refer to the following exemplary description: Assuming the first direction representation data is the aforementioned Plücker ray map P, and the U-Net within the target image generation model is a denoising network, the model generates noisy latent variables during the denoising sampling stage. z .

[0134] Execute the first injection branch, and compare the Plücker ray map P with the diffusion time step. τ Noisy latent variables at the location Directly stitching along the channel dimension, resulting in fused features. The image is directly fed into the first convolutional layer of the denoising network to achieve pixel-by-pixel binding of observation direction information with image features. The stitching process can be illustrated by formula (6), i.e.: = (6) Simultaneously (or subsequently), the second injection branch is executed, performing global average pooling on the Plücker ray graph P, and then linearly projecting the globally average pooled Plücker ray graph P to obtain the condition vector c. cam Conditional vector c cam The AdaLN module is injected into each residual block of U-Net to constrain the feature normalization scale and offset during the feature computation process of the residual blocks.

[0135] Thus, in this embodiment, the first direction representation data and the input feature map of the first layer of the denoising network of the target image generation model can be concatenated along the channel dimension, and the global pooling features of the first direction representation data can be injected into at least one residual block of the denoising network through the adaptive layer normalization module. This avoids the camera constraints provided by the first direction representation data from weakening or disappearing during the iteration process under single-mode injection, thereby ensuring the model inference effect.

[0136] In some embodiments provided in this application, step 130 includes: determining each first direction representation data as a conditional attribute of the target image generation model, and inputting each first direction representation data into the target image generation model; based on the target image generation model, performing denoising sampling processing on the initial noise data corresponding to each first observation viewpoint to obtain denoised representation data corresponding to each first observation viewpoint; and performing decoding processing on the denoised representation data corresponding to each first observation viewpoint to obtain a first generated image corresponding to each first observation viewpoint.

[0137] Specifically, to avoid misalignment between the viewpoint of the generated image and the preset observation direction of the virtual camera, in the embodiments provided in this application, the electronic device can set the first direction representation data of each observation viewpoint as a conditional attribute of the target image generation model and complete the model input, thereby constraining the model's image generation logic. In the image generation stage, the model first performs denoising sampling on the initial noise data corresponding to each viewpoint, extracts the denoised representation data in the latent space, and finally performs decoding and transformation on the denoised representation data of the intermediate layer, ultimately outputting the first generated image whose viewpoint matches the camera's observation direction.

[0138] In some embodiments, conditional attributes can be understood as guiding feature parameters that provide camera geometric constraint benchmarks throughout the entire computation cycle of model denoising and decoding, used to constrain the generation logic of the target image generation model, so that the model can generate corresponding images according to the observation direction of the virtual camera.

[0139] In some examples, the aforementioned Plücker ray feature data can be used as conditional attributes to constrain the feature generation logic of the model.

[0140] In some embodiments, the initial noise data can be understood as the basic input data of the diffusion-type image generation model, which can be a random noise tensor that satisfies a standard normal distribution. Each first observation view is configured with corresponding noise data that is independent of each other.

[0141] In some examples, for 24 different observation perspectives in a certain game scene, the system will randomly generate 24 sets of distinct Gaussian noise tensors with dimensions matching the image resolution, as independent initial noise data for each perspective.

[0142] In some embodiments, denoising sampling processing can be understood as a computational process that uses the reverse diffusion algorithm logic of the diffusion model to gradually remove useless random noise from the initial noise and extract the latent space features of the virtual scene from the noise through multiple rounds of iterative operations.

[0143] In some examples, the DDIM sampling strategy is used to perform denoising sampling, setting a fixed number of sampling steps for iterative denoising, gradually stripping away random noise, and extracting the spatial and color latent features of the scene from the disordered noise.

[0144] In some embodiments, denoised representation data can be understood as intermediate latent variable data that has undergone denoising sampling processing, removing most of the random noise and carrying the latent space features of the image. It can be stored in the latent space of the model and cannot be directly converted into a visual image.

[0145] In some examples, the low-dimensional latent vector tensor output after 30 steps of DDIM denoising sampling is the denoised representation data. This type of data is stored in the VAE latent space and cannot be directly parsed and displayed by electronic devices.

[0146] In some embodiments, the decoding process is a data transformation operation that uses the model’s built-in decoding network to map and transform the denoised representation data in the latent space format to the pixel image space, thus converting the latent variables into a visualized digital image.

[0147] In some examples, the built-in Variational Autoencoder (VAE) decoder of the target image generation model is invoked to read the denoised representation data in the latent space. After dimension restoration and pixel mapping operations, a standard digital image with three channels of red, green, and blue (RGB) is output.

[0148] To more clearly illustrate the generation process of the first generated image in the embodiments of this application, please refer to the following exemplary description: Taking an open-world mountain scene built with Unreal Engine 5 as an example, the designers pre-set 24 different pitch and azimuth first observation views inside the engine. After the electronic device calculates the first direction representation data corresponding to each of the 24 views, it uniformly marks these 24 first direction representation data as the conditional attributes of the target image generation model. After the data format is adapted, it is input into the target image generation model that has been fine-tuned by the engine data, so that the camera geometric constraint information is embedded in the model calculation link.

[0149] Subsequently, the electronic device generates a set of independent Gaussian initial noise data for each of the 24 observation perspectives. All noise data are then sent into the model in batches. Under the continuous constraints of the conditional attributes, the model uses the Denoising Diffusion Implicit Models (DDIM) sampling rule to perform multiple rounds of denoising sampling processing on each set of initial noise data, thereby obtaining 24 denoised representation data corresponding to each perspective.

[0150] Finally, the model's built-in decoding module reads each denoised representation data in sequence and performs decoding processing, completing the format conversion of latent space data to pixel images one by one, and finally outputting 24 first generated images that are precisely matched with the corresponding observation viewpoints.

[0151] Thus, in this embodiment, each first direction representation data can be determined as a conditional attribute of the target image generation model, and each first direction representation data can be input into the target image generation model. Based on the target image generation model, the initial noise data corresponding to each first observation viewpoint is subjected to denoising sampling processing to obtain denoised representation data corresponding to each first observation viewpoint. The denoised representation data corresponding to each first observation viewpoint is then decoded to obtain the first generated image corresponding to each first observation viewpoint. This enables the camera observation geometry information to constrain the image generation process, and to a certain extent avoids the misalignment between the viewpoint of the generated image and the preset observation direction of the virtual camera, ensuring the pixel alignment accuracy between the generated image and the observation viewpoint of the virtual camera.

[0152] In some embodiments provided in this application, the 3D scene reconstruction method further includes: acquiring a reference image corresponding to at least one reference observation viewpoint and second camera geometric parameters corresponding to the reference observation viewpoint; determining second direction representation data corresponding to the reference observation viewpoint based on the second camera geometric parameters; and encoding the reference image to obtain reference image encoded data. The second camera geometric parameters include second camera intrinsic parameters and second camera extrinsic parameters.

[0153] Correspondingly, the steps described above, which determine each first direction representation data as a conditional attribute of the target image generation model and input each first direction representation data into the target image generation model, include: determining the reference image encoding data, the second direction representation data, and each first direction representation data as conditional attributes, and inputting the reference image encoding data, the second direction representation data, and each first direction representation data into the target image generation model.

[0154] Specifically, if only the representation data of the first direction is used as the model conditional attribute, the generated model only has the geometric constraint information of the target viewpoint, lacks the prior knowledge of the real picture content of the virtual scene and the geometric anchoring information corresponding to the reference viewpoint, which can easily lead to problems such as object deformation, texture loss, and inconsistent color style in different viewpoints.

[0155] Based on this, in some embodiments provided in this application, additional data from the reference observation perspective can be introduced. In other words, the electronic device can collect the reference image and the geometric parameters of the second camera, and convert them into the second direction representation data and the reference image encoding data respectively. The two types of new data and the original first direction representation data are merged and uniformly used as the conditional attribute input model of the target image generation model. Thus, the image generation process can be restricted by combining the prior knowledge of the image content with geometric constraints.

[0156] In some embodiments, a reference observation viewpoint can be understood as a pre-selected fixed observation point that can completely capture the main content of the target virtual scene, and the imaging content of this viewpoint serves as a reference for other viewpoint images to be generated.

[0157] In some examples, when rebuilding game building assets, the camera position directly in front of the asset is selected as the reference observation view, and the remaining multiple cameras arranged around it are all used as the first observation view to be generated.

[0158] In some embodiments, a reference image can be understood as the original image captured by a virtual camera from a reference observation perspective, and the image contains real object texture, structure and color information of the scene.

[0159] In some examples, the high-resolution original image of the reference view, directly rendered and output by the game engine, serves as the reference image.

[0160] In some embodiments, the geometric parameters of the second camera can be understood as the camera configuration parameters corresponding to the reference observation view, including two types of parameters: the intrinsic parameters of the second camera and the extrinsic parameters of the second camera.

[0161] In some examples, the reference camera focal length, principal point coordinates, and rotation and translation matrices of the camera in the world coordinate system, derived from the game engine, together form a set of second camera geometric parameters.

[0162] In some embodiments, the second direction characterization data can be understood as feature data generated by converting the geometric parameters of the second camera through a reference observation viewpoint, used to characterize the spatial observation ray direction corresponding to each pixel under the reference viewpoint.

[0163] In some examples, the H×W×6 Plücker ray atlas generated after the reference viewpoint parameters are adapted and transformed to the coordinate system is the second-direction representation data.

[0164] In some embodiments, reference image encoded data can be understood as feature data generated after the original reference image is compressed and mapped to the latent space by an encoder.

[0165] In some examples, the low-dimensional tensor data output by the VAE encoder built into the reference image input model is the reference image encoded data.

[0166] To more clearly illustrate the process of obtaining and inputting multiple types of conditional attributes in the embodiments of this application, please refer to the following exemplary description: Taking the reconstruction of cartoon building assets created with the Unity engine as an example, the project sets up one set of frontal camera positions as the reference observation view, and the remaining 12 surrounding camera positions as the first observation view to be generated.

[0167] First, the electronic device acquires the reference image corresponding to the reference observation viewpoint from the engine side, as well as the second camera geometric parameters consisting of the second camera intrinsic parameters and the second camera extrinsic parameters associated with that viewpoint. Using the second camera geometric parameters, coordinate transformation and pixel ray solving are performed to generate the second direction representation data corresponding to the reference viewpoint. At the same time, the acquired reference image is sent to the encoding network built into the target image generation model. After feature compression and spatial mapping, reference image encoding data in latent space format is obtained.

[0168] Then, the electronic device sets the reference image encoding data, the second-direction representation data, and 12 sets of first-direction representation data as conditional attributes of the target image generation model, and inputs these data into the model. Under the common constraints of the three types of conditional attributes, the model performs DDIM denoising sampling on the independent initial noise data corresponding to the 12 first observation viewpoints to obtain denoised representation data, which is then decoded to output a multi-view video frame sequence. This outputs 12 first generated images. In the denoising sampling process, the input for each step can be represented as:

[0169] in, For the target perspective (i.e., the first observation perspective) at the diffusion time step Noisy latent variables at the location, VAE encoding for the conditional perspective (i.e., the reference observation perspective). and The Plücker ray diagrams (i.e., first-direction characterization data and second-direction characterization data) are respectively from the target and condition perspectives.

[0170] Thus, in this embodiment, at least one reference image corresponding to a reference observation viewpoint and the geometric parameters of a second camera corresponding to the reference observation viewpoint can be obtained. Based on the geometric parameters of the second camera, second-direction representation data corresponding to the reference observation viewpoint is determined. The reference image is encoded to obtain reference image encoded data. The reference image encoded data, the second-direction representation data, and each first-direction representation data are all determined as conditional attributes. These data are then input into the target image generation model. This allows the second-direction representation data and the first-direction representation data to jointly constitute geometric constraints, thereby improving the content consistency and geometric alignment between multi-view images generated by the model. Furthermore, the introduction of reference image encoded data can, to some extent, improve the situation where the model lacks real-world content references, thereby improving issues such as detail loss and color deviation in the generated images.

[0171] In some embodiments provided in this application, step 140 includes: constructing initial point cloud data of the target virtual scene based on the first camera geometric parameters corresponding to each first observation viewpoint; initializing the three-dimensional Gaussian splash data of the target virtual scene based on the initial point cloud data to obtain initial three-dimensional Gaussian splash data; updating the initial three-dimensional Gaussian splash data based on the first camera geometric parameters and the first generated image corresponding to each first observation viewpoint, and determining the updated initial three-dimensional Gaussian splash data as the target three-dimensional Gaussian splash data. During the update process, the parameter values ​​of the first camera geometric parameters remain unchanged.

[0172] Specifically, in some embodiments provided in this application, the electronic device can construct an initial point cloud of the scene using known and accurate first camera geometric parameters, eliminating the need for camera inverse solving steps; then, the initialization of 3D Gaussian splashing is completed based on the initial point cloud, generating unoptimized initial 3D Gaussian splashing data; finally, by combining the camera parameters of each viewpoint with the matched first generated image, the Gaussian elements in the initial 3D Gaussian splashing data are iteratively optimized under the premise of fixed camera parameters, and the updated and optimized data is used as the final usable target 3D Gaussian splashing data.

[0173] In some embodiments, the initial point cloud data can be understood as a dataset consisting of a large number of discrete coordinate points in three-dimensional space, which can sparsely represent the spatial contour and structure of the target virtual scene.

[0174] In some examples, multi-view depth information output by Unreal Engine is combined with precise camera parameters to back-project image pixels into three-dimensional space, summarize all projected points and remove redundant noise, and after downsampling, an initial point cloud data containing hundreds of thousands of spatial points is formed.

[0175] In some embodiments, the initial 3D Gaussian splatter data refers to the dataset formed by creating Gaussian elements one by one centered on each 3D coordinate within the initial point cloud. Each Gaussian element is pre-configured with initial parameters such as position, opacity, covariance, and spherical harmonic color, and has not yet undergone iterative optimization and correction.

[0176] In some examples, all 3D points within the initial point cloud are traversed, and a basic 3D Gaussian cell is generated for each spatial coordinate. The cell's parameters are randomly initialized, and all Gaussian cells are integrated and stored to obtain the initial 3D Gaussian splash data.

[0177] In some embodiments, the update process is an iterative optimization operation that repeatedly adjusts the various attribute parameters of Gaussian primitives and continuously reduces the pixel error between the Gaussian rendered image and the generated image, while keeping the camera geometric parameters unchanged and using the first generated image as a real-world monitoring signal.

[0178] In some examples, the images of each viewpoint are rendered forward using the current Gaussian data. The L1 loss (L1) and Structural Similarity Index Loss (SSIM) loss of the rendered results and the first generated image are compared. The spatial position, transparency and color parameters of each Gaussian unit are fine-tuned by relying on the backpropagation algorithm. Multiple iterations are performed until the loss converges, and the update process is completed.

[0179] In some embodiments, the target 3D Gaussian splatter data refers to the final Gaussian dataset after multiple rounds of updates and optimizations, where the Gaussian meta-parameters converge and stabilize, and the data can accurately reproduce the spatial structure and surface color information of the scene. This dataset can be directly used for 3D scene rendering and reconstruction.

[0180] In some examples, after hundreds of iterations and updates, and once the rendering error reaches a preset threshold, the entire set of Gaussian primitives can be imported into the game engine as the target 3D Gaussian splash data to complete the scene preview.

[0181] Thus, in this embodiment, initial point cloud data of the target virtual scene can be constructed based on the first camera geometric parameters corresponding to each first observation viewpoint. The initial point cloud data is then used to initialize the three-dimensional Gaussian splash data of the target virtual scene to obtain initial three-dimensional Gaussian splash data. Furthermore, the initial three-dimensional Gaussian splash data is updated based on the first camera geometric parameters and the first generated image corresponding to each first observation viewpoint. The updated initial three-dimensional Gaussian splash data is then determined as the target three-dimensional Gaussian splash data. This eliminates the need for camera pose estimation using motion reconstruction software, improving execution efficiency. Additionally, the accuracy of the initial point cloud points generated by the camera parameters is higher, thereby reducing the positional deviation of the Gaussian initialization and reducing the number of subsequent optimization iterations.

[0182] In some embodiments provided in this application, the step of constructing initial point cloud data of the target virtual scene based on the geometric parameters of the first camera corresponding to each first observation view includes: obtaining a first depth map of the target virtual scene corresponding to each first observation view through a game engine; projecting pixels in the first generated image corresponding to each first observation view onto the three-dimensional space corresponding to the target virtual scene according to the first camera geometric parameters and the first depth map to obtain a three-dimensional point cloud corresponding to each first observation view; and merging the three-dimensional point clouds corresponding to all first observation views to obtain initial point cloud data.

[0183] Specifically, to ensure the accuracy and efficiency of the initial point cloud generation, in some embodiments provided in this application, the game engine can output the first depth map of each observation viewpoint. This allows the use of the engine's native depth data to avoid errors caused by depth estimation through algorithms or models in traditional solutions. After obtaining the depth data output by the engine, the pixels of the generated image are reverse-mapped to three-dimensional space using known camera geometric parameters to obtain three-dimensional point clouds for each viewpoint. Finally, the point cloud data from all viewpoints are integrated and merged to generate initial point cloud data that can be directly used for three-dimensional Gaussian splash initialization.

[0184] In some embodiments, the first depth map can be understood as a depth data image generated synchronously by the game engine during the screen rendering stage. The pixel value of each pixel in the image represents the real physical depth from the corresponding screen position to the optical center of the virtual camera. Since it is directly generated by the game engine, there is no numerical error caused by algorithm estimation.

[0185] In some examples, when rendering a scene using Unreal Engine 5, the engine can export a first depth map with the same resolution and viewpoint at the same time as outputting the rendered frame, with the depth values ​​calibrated according to the standard units of the engine's world coordinate system.

[0186] In some embodiments, a three-dimensional point cloud can be understood as a set of three-dimensional coordinate points formed by spatial back projection of two-dimensional image pixels under a single observation viewpoint, with the points accompanied by spatial coordinates and pixel color information.

[0187] In some examples, a single 1920×1080 resolution first generated image, after pixel projection calculation, can generate millions of spatial coordinate points, and all points are aggregated to form an independent 3D point cloud corresponding to that viewpoint.

[0188] In some embodiments, the merging process can be understood as performing an integration operation on the scattered 3D point clouds from multiple perspectives, including coordinate unification, redundant point removal, and point location simplification, to eliminate duplicate coordinate data and form a uniformly distributed complete point cloud set.

[0189] In some examples, 24 independent point clouds generated from different observation perspectives were integrated to generate a sparse and uniform global initial point cloud after coordinate alignment, duplicate point filtering, and appropriate downsampling.

[0190] To more clearly illustrate the process of acquiring the initial point cloud in the embodiments of this application, please refer to the following exemplary description: Taking an open-world distant mountain scene built with Unreal Engine 5 as an example, after obtaining 24 first generated images corresponding to the viewpoint through the target image generation model, the electronic device connects to the Unreal Engine's data output interface and simultaneously retrieves 24 corresponding first depth maps from the engine during the rendering stage of each viewpoint.

[0191] Subsequently, the electronic device retrieves the first camera geometric parameters corresponding to each viewpoint, uses the camera intrinsic parameters to normalize the pixel coordinates, and combines the camera extrinsic parameters with the depth values ​​in the depth map to convert the pixels of each first generated image to the virtual scene world coordinate system, thereby generating 24 sets of independent single-view 3D point clouds.

[0192] Finally, all 24 sets of 3D point clouds were unified into the same spatial coordinate system, redundant coordinate points with overlapping spatial positions were filtered out, and the point clouds were appropriately downsampled and simplified according to the reconstruction requirements. After merging, the initial point cloud data of the entire scene was obtained.

[0193] Thus, in this embodiment, the first depth map of the target virtual scene corresponding to each first observation view can be obtained through the game engine, and the pixels in the first generated image corresponding to each first observation view can be projected into the three-dimensional space corresponding to the target virtual scene according to the first camera geometric parameters and the first depth map to obtain the three-dimensional point cloud corresponding to each first observation view. The three-dimensional point clouds corresponding to all first observation views are then merged to obtain the initial point cloud data. Thus, the initial point cloud can be constructed using the native depth data output by the game engine. Compared with the traditional method of obtaining depth through estimation algorithms or models and constructing the initial point cloud accordingly, depth estimation errors can be avoided, thereby improving the quality of the initial point cloud.

[0194] Furthermore, the high-quality initial point cloud used for 3D Gaussian splash initialization ensures that the initial position of the Gaussian elements closely matches the real scene structure, saving reconstruction computational costs and shortening computation time.

[0195] In some embodiments provided in this application, the step of updating the initial three-dimensional Gaussian splash data based on the first camera geometric parameters and the first generated image corresponding to each first observation view includes: rendering a rendering depth map corresponding to each first observation view based on the initial three-dimensional Gaussian splash data; determining a depth consistency parameter based on the first depth map and the rendering depth map corresponding to each first observation view; and updating the initial three-dimensional Gaussian splash data based on the depth consistency parameter.

[0196] Specifically, traditional 3D Gaussian splash optimization updates mostly rely solely on pixel photometric losses in the generated image to iterate parameters, which can easily lead to distortions in the spatial structure of the reconstructed 3D scene and misalignment of Gaussian patches. Therefore, in some embodiments provided in this application, the initial, unoptimized 3D Gaussian splash data is used in conjunction with camera parameters to render rendering depth maps from various viewpoints. The first depth map output by the engine is used as a ground truth reference, and the two types of depth maps are compared to calculate a depth consistency parameter. This parameter is then used as a depth constraint condition, combined with the original photometric loss, to iteratively update the initial 3D Gaussian splash data, constraining the Gaussian pixel distribution from a spatial dimension.

[0197] In some embodiments, the rendering depth map can be understood as a depth image generated by using the initial three-dimensional Gaussian splash data of the current iteration stage, combined with the geometric parameters of the first camera from the corresponding viewpoint, and using differentiable rasterization rendering. The value of each pixel in the image represents the spatial distance from the camera to the corresponding point in the scene, thereby reflecting the scene depth distribution fitted by the current Gaussian model.

[0198] In some examples, for a certain observation point in a mountain scene created by the game engine, all Gaussian metadata and the intrinsic and extrinsic parameters of the camera in this iteration are retrieved, and a single rendering depth map with a resolution of 1920×1080 is generated through raster rendering, with the pixels storing floating-point depth distance values.

[0199] In some embodiments, the depth consistency parameter can be understood as a quantized parameter obtained by comparing the difference between the rendered depth and the real depth pixel by pixel using the first depth map from the same viewpoint as the real depth benchmark, removing invalid pixels, and integrating the calculations. This parameter can be converted into a depth loss term to constrain the optimization direction of the three-dimensional Gaussian splash parameter.

[0200] In some examples, after filtering out invalid pixels whose depth exceeds the scene distance threshold, the depth error of all pixels in the valid area is summed using L1 weighting, and the resulting value is the depth consistency parameter corresponding to that viewpoint.

[0201] To more clearly illustrate the update process of the initial three-dimensional Gaussian splash data in the embodiments of this application, please refer to the following exemplary description: Assume that the multiple first generated images are the aforementioned multi-view sequence frames. Then it can be based on the initial point cloud and multi-view sequence frames. Combining known Ground Truth Camera Parameters (GT), namely the intrinsic and extrinsic parameters of the camera recorded by the game engine, and multi-view frame sequences... With multi-view sequence frames To monitor performance, a 3DGS optimization process was executed to calculate the photometric loss as shown in Equation (7). And minimize that loss.

[0202] (7) in, For L1 color loss, For structural similarity loss, For weight hyperparameters, .

[0203] In the 3DGS optimization process, the optimization object is the set of attributes of Gaussian basic elements, that is, the three-dimensional position of the Gaussian basic elements. Opacity Covariance matrix Color sphere harmonic coefficient The GT camera parameters are fixed and not included in the optimization.

[0204] After obtaining the depth map generated by the game engine At that time, calculate the depth consistency regularization term as shown in formula (8). This is the depth consistency parameter, and convergence is accelerated through the depth consistency parameter.

[0205] (8) in, From the perspective of the current Gaussian field The rendering depth.

[0206] Thus, in this embodiment of the application, a rendering depth map corresponding to each first observation view can be obtained by rendering the initial three-dimensional Gaussian splash data, and a depth consistency parameter can be determined based on the first depth map and the rendering depth map corresponding to each first observation view. The initial three-dimensional Gaussian splash data can be updated based on the depth consistency parameter. This adds a depth dimension constraint in addition to the traditional photometric loss, restricting the arrangement of Gaussian primitives in space and avoiding structural distortion and floating of patches in scene reconstruction.

[0207] Furthermore, the regularization constraints brought about by depth consistency can accelerate model convergence to a certain extent, reduce the number of iterations, and effectively reduce the computational power consumption during the 3D reconstruction process. At the same time, by using the engine's natively accurate first depth map as a benchmark, the optimized 3D Gaussian splash data maintains consistency with the original virtual scene in terms of scale and coordinate system, eliminating the need for additional scale and coordinate calibration steps after reconstruction, and effectively reducing the manual correction costs for subsequent deployment of game 3D assets.

[0208] In some embodiments provided in this application, the step of constructing initial point cloud data of the target virtual scene based on the geometric parameters of the first camera corresponding to each first observation view includes: inputting the first generated image corresponding to each first observation view into a pre-trained depth estimation model, so that the depth estimation model estimates the depth of the target virtual scene based on each first generated image, and obtains the depth estimation result corresponding to each first generated image; and constructing initial point cloud data based on the geometric parameters of the first camera corresponding to each first observation view and the depth estimation result corresponding to each first generated image.

[0209] Specifically, in actual operation, some game project resources are encrypted and packaged, and the engine disables the export permission of depth data, making it impossible to directly obtain the first depth map from the game engine, and thus impossible to construct a point cloud from the depth map generated by the engine. Therefore, in cases where it is impossible to construct a point cloud from the depth map generated by the engine, to improve the quality of the initial point cloud, in some embodiments provided in this application, the electronic device can input the first generated images from each viewpoint into a pre-trained depth estimation model. The model automatically predicts the depth estimation result corresponding to a single image, and then, combined with the known and accurate first camera geometric parameters, projects the image pixels back into three-dimensional space, summarizing the spatial points of all viewpoints, and finally generating initial point cloud data for three-dimensional Gaussian splash initialization.

[0210] In some embodiments, the depth estimation model can be understood as a deep learning network that has been trained offline using massive amounts of game scene images and corresponding ground truth depth samples, and can infer the depth value of each pixel in the image from a single two-dimensional image.

[0211] In some examples, the depth estimation model can use a monocular depth network based on the Transformer architecture, and mix in various types of game screen data such as realistic, cartoon, and low-poly graphics during the training phase to adapt to the depth inference work of the first generated image with different styles.

[0212] In some embodiments, the depth estimation result can be understood as a structured depth matrix output by the depth estimation model. The matrix size corresponds one-to-one with the resolution of the first generated image input, and each value in the matrix represents the spatial depth of the corresponding pixel from the optical center of the first virtual camera.

[0213] In some examples, when a first generated image with a resolution of 1280×720 is input into the depth estimation model, the model will output a floating-point depth matrix of the same size, and each floating-point value in the matrix is ​​the estimated depth of the corresponding pixel.

[0214] Thus, in this embodiment, the first generated image corresponding to each first observation viewpoint can be input into a pre-trained depth estimation model. This allows the depth estimation model to estimate the depth of the target virtual scene based on each first generated image, obtaining a depth estimation result for each first generated image. Furthermore, based on the first camera geometric parameters corresponding to each first observation viewpoint and the depth estimation result for each first generated image, initial point cloud data is constructed, thereby generating the initial point cloud data. Moreover, compared to randomly generated point clouds, point clouds generated through image-based depth prediction more closely resemble the real spatial structure of the scene, thereby reducing the initialization error of the 3D Gaussian splashing, shortening the subsequent iterative optimization cycle, and reducing equipment computing power consumption.

[0215] Please see Figure 3 , Figure 3 This is a flowchart illustrating a three-dimensional scene reconstruction method provided in an embodiment of this application. It should be noted that the steps shown may be executed in a logical order different from that shown in the flowchart. Specifically, as... Figure 3 As shown, the training steps for the target image generation model include: Step 210: Obtain multiple training samples, wherein each training sample includes multiple image samples obtained by the second virtual camera of the reference game scene observing the reference game scene from at least two third observation perspectives, as well as the third camera geometric parameters and second depth map of the second virtual camera in each third observation perspective. The second depth map is generated by the game engine. Step 220: Determine the second direction representation data corresponding to the third observation viewpoint based on the geometric parameters of the third camera; Step 230: Input the second direction representation data into the initial image generation model to be trained, so that the initial image generation model generates a second generated image corresponding to the third observation viewpoint based on the second direction representation data; Step 240: Train the initial image generation model based on the second generated image, image samples, second depth map, and second orientation representation data to obtain the target image generation model.

[0216] Specifically, traditional image generation models are mostly pre-trained using real-world video footage. The camera geometry and scene depth information used for training require reverse engineering using external pose estimation software, inevitably introducing systematic errors during the estimation process. Furthermore, traditional training datasets typically consist mainly of real-world footage, lacking the unique visual data specific to the gaming industry, such as cartoon, realistic, and low-poly graphics. This makes it difficult for the models to adapt to the coordinate system rules and visual characteristics of mainstream game engines. Therefore, if pre-trained models are directly used for multi-view image generation in game scenes, the generated images cannot be aligned pixel-wise with the engine's coordinate system. Subsequent 3D Gaussian splash reconstruction still requires significant time for iterative optimization of camera parameters, making it difficult to guarantee reconstruction efficiency and final accuracy.

[0217] Based on this, in the embodiments provided in this application, the electronic device can construct training samples using the ground truth error data natively output by the game engine, use the accurate third camera geometric parameters within the samples to calculate the second direction representation data and use it as the model training condition, constrain the initial image generation model to output the generated image from the corresponding viewpoint, and combine the native image ground truth and depth ground truth of the engine to jointly construct the loss function to complete the model iterative training, thereby obtaining the target image generation model that adapts to the parameter rules and visual style of the game engine, and making up for the defects of poor parameter adaptation and misaligned image generation of the existing model from the training source.

[0218] In some embodiments, training samples can be understood as multimodal data units used to iteratively optimize the image generation model. Each sample corresponds to multiple observation views of the same reference game scene and includes image samples, camera parameters, depth... Figure 3 Class data.

[0219] In some examples, a single training sample can contain data from 8 to 32 perspectives in the same scene, and the camera trajectory composed of multiple observation perspectives within a single sample can cover various modes such as circular orbits, straight-line fly-throughs, and random multi-viewpoints.

[0220] In some embodiments, the reference game scene can be understood as a virtual 3D scene built within the game engine, which can support configuration of different art styles such as realistic, cartoon, and science fiction, and can directly render and output the screen, depth and camera parameter data by the engine.

[0221] In some examples, the reference game scenes can be a Q-version cartoon town scene created using a game engine, or a realistic wilderness open world scene built using UE5. These two types of scenes are used for specialized training of models with cartoon style and realistic texture, respectively.

[0222] In some embodiments, the second virtual camera can be understood as an engine-built-in virtual camera used only for material acquisition during the model training phase and instantiated and deployed within the reference game scene, with the engine directly outputting a complete set of camera geometric parameters with zero error.

[0223] In some examples, the second virtual camera can be a perspective virtual camera configured in the game engine's scene editor, supporting batch arrangement along circular or straight trajectories, and exporting full-view parameters and rendering materials.

[0224] In some embodiments, the third observation perspective can be understood as the spatial position and lens orientation of the second virtual camera in the reference game scene during the training acquisition phase. Each perspective uniquely corresponds to a set of camera parameters, a rendered image, and a depth map.

[0225] In some examples, the third observation perspective can be 16 shooting points evenly arranged in a ring around the center of the scene, with an azimuth angle interval of 22.5° between the points and a pitch angle set at ±15°, forming a standardized multi-view sampling scheme.

[0226] In some embodiments, the image sample can be understood as a baseline image directly rendered by the game engine from a third observation perspective by a second virtual camera, serving as the ground truth standard for the model generation result.

[0227] In some examples, the image samples may be renderings of architectural assets output by the game engine using global illumination, with the colors and lighting being the engine's native true values.

[0228] In some embodiments, the geometric parameters of the third camera can be understood as the camera intrinsic and extrinsic parameter matrices of the viewpoint corresponding to the second virtual camera, which are synchronously output by the game engine. These parameters include focal length, principal point, spatial rotation and translation information, and the parameter values ​​have no estimation error.

[0229] In some examples, the third camera geometry parameters can be a 3×3 intrinsic matrix and a 3×4 extrinsic matrix derived from the game engine, with the matrix values ​​calculated and generated by the game engine.

[0230] In some embodiments, the second depth map can be understood as a pixel-level depth map generated synchronously by the game engine when rendering image samples, where each pixel in the map stores the corresponding real depth distance in three-dimensional space.

[0231] In some examples, the second depth map can be a grayscale depth map output by the engine's rasterization, with smaller pixel depth values ​​for nearby objects and larger pixel depth values ​​for distant objects, which can be directly used for 3D point cloud back projection calculations.

[0232] In some embodiments, the second direction characterization data can be understood as a structured feature map generated by the geometric parameters of the third camera after coordinate system adaptation and pixel ray conversion, which can characterize the camera observation ray information corresponding to all pixels of the virtual imaging plane under a single view.

[0233] In some examples, the second-direction representation data can be a Plücker ray feature map of H×W×6 dimensions, with a six-dimensional ray vector stored at each pixel position, and connected to the diffusion model network through two methods: channel concatenation and adaptive normalization.

[0234] In some embodiments, the initial image generation model can be understood as a basic video diffusion network that has not been specifically optimized for game scene data. It has the basic ability to generate general multi-view images, but it is not adapted to the game engine coordinate system and screen style.

[0235] In some examples, the initial image generation model is a video diffusion model backbone network based on SEVA pre-trained weights. During training, a LoRA low-rank fine-tuning scheme is used to fine-tune only the attention layer weights, while retaining the model's original general generation priors.

[0236] In some embodiments, the second generated image can be understood as the predicted image output by the initial image generation model after noise reduction and decoding, after the second direction representation data is used as a condition, and is used to compare with the same viewpoint image sample.

[0237] In some examples, the second generated image can be a multi-view rendering of a cartoon building generated by the model based on ray conditions. The output image contrasts with the original image from the engine in terms of composition and object outline, and is used for loss calculation.

[0238] To more clearly illustrate the complete training process of the target image generation model in the embodiments of this application, please refer to the following exemplary description: First, within the reference game scene of the game engine, the electronic device is configured with multiple sets of third-viewpoints from a second virtual camera. Each set contains multiple third-viewpoints, collectively forming a camera trajectory. The game engine renders and generates corresponding image samples based on each third-viewpoint, simultaneously outputting the corresponding third-camera geometric parameters and a second depth map. Image samples, camera parameters, and depth maps from multiple views within the same scene are paired and encapsulated to form a single training sample. This process is repeated to generate multiple training samples covering various scene styles and trajectory patterns, forming a training set.

[0239] Subsequently, for each training sample in the dataset, second-direction representation data corresponding to the third observation viewpoint is generated based on the third camera's geometric parameters. It is understood that the process of determining the second-direction representation data corresponding to the third observation viewpoint based on the third camera's geometric parameters is consistent with the aforementioned process of determining the first-direction representation data corresponding to the first observation viewpoint based on the first camera's geometric parameters. For details, please refer to the relevant explanations above; to avoid repetition, they will not be elaborated upon here.

[0240] Subsequently, the generated second-direction representation data is set as a conditional attribute of the initial image generation model and input into the initial image generation model. The initial image generation model uses Gaussian random noise as the initial latent variable and completes diffusion denoising sampling through conditional constraints. The second generated image corresponding to the third observation viewpoint is then output via the built-in VAE decoder. It can be understood that the specific method of inputting the second-direction representation data into the initial image generation model is consistent with the specific method of inputting the first-direction representation data into the target image generation model as described above. For details, please refer to the relevant explanations above. To avoid repetition, it will not be elaborated here.

[0241] Then, the electronic device retrieves image samples, second depth maps, and second orientation representation data from the same sample, and calculates the loss function value together with the second generated image output by the model.

[0242] Finally, the gradient values ​​of backpropagation are calculated based on the loss function values, and the LoRA strategy is used to fine-tune the network weights of the initial image generation model. That is, while freezing the backbone weights of the initial image generation model, only the weights of the cross-attention layer are adjusted. , , The projection matrix adds low-rank bypasses, and the weights of these low-rank bypasses are updated.

[0243] The entire process of parameter conversion, image generation, and weight update is repeatedly executed by retrieving all training samples until the overall dataset loss converges to a preset threshold. The training process is then terminated, and a target image generation model that can be used for subsequent 3D reconstruction is finally obtained.

[0244] Thus, in this embodiment, multiple training samples can be obtained, and based on the third camera geometric parameters in the training samples, the second direction representation data corresponding to the third observation viewpoint can be determined. This second direction representation data is then used as a conditional attribute and input into the initial image generation model to be trained. This enables the initial image generation model to generate a second generated image corresponding to the third observation viewpoint. Based on the second generated image, the second direction representation data, and the image samples and second depth map in the training samples, the initial image generation model is trained to obtain the target image generation model. This achieves model training based on the engine's native data. Furthermore, the trained target image generation model can receive camera data input from the engine, generate multi-view images pixel-level aligned with the engine's coordinate system, and the visual distribution of the generated images conforms to the engine's native rendering style.

[0245] In some embodiments provided in this application, step 240 includes: determining a target loss based on the second generated image corresponding to each of the third observation viewpoints in the training samples, the image sample, the second depth map, and the second orientation representation data; and training the initial image generation model based on the target loss to obtain a target image generation model. The target loss is used to indicate the differences between the second generated images corresponding to each of the third observation viewpoints.

[0246] Specifically, in the training process of multi-view image generation models, if the loss is calculated using single-view images to train the model, the loss can only constrain the visual difference between the generated image and the ground truth image from a single viewpoint. It lacks effective constraints on the geometric consistency between the multi-view generation results, which leads to problems such as content misalignment and geometric structure drift in the different viewpoint images output by the model, and cannot directly meet the input requirements of high-precision 3D reconstruction.

[0247] Based on this, in some embodiments provided in this application, during the model training phase, the electronic device can calculate the target loss that comprehensively reflects the differences between multi-view generated images by using the second generated image, image sample, second depth map and second orientation representation data contained in the training samples. Then, using the target loss as the optimization objective, the network parameters of the initial image generation model are iteratively updated through backpropagation to complete the model iterative training. This allows the model to learn not only the knowledge of generating ground truth images from a single viewpoint during the training phase, but also the geometric consistency of different images under different viewpoints, ultimately obtaining a target image generation model with multi-view alignment capabilities.

[0248] In some embodiments, the target loss can be understood as a numerical index calculated from the second generated image, image sample, second depth map, and second direction representation data corresponding to different third observation perspectives within a single sample. It can evaluate the difference between the results (i.e., the second generated image) generated by the model corresponding to different third observation perspectives and the true benchmark (i.e., image sample and second depth map), and can also evaluate the degree of difference between the generated results (i.e., the second generated image) corresponding to different third observation perspectives. The magnitude of the target loss directly reflects the current generation accuracy and consistency level of the model.

[0249] Thus, in this embodiment of the application, the target loss can be determined based on the second generated image, the image sample, the second depth map, and the second direction representation data corresponding to each third observation view in the training sample, and the initial image generation model can be trained based on the target loss to obtain the target image generation model. This allows the model to learn the geometric alignment relationship of multi-view generated images during the training phase, thereby avoiding content drift and structural misalignment caused by viewpoint switching when the model generates images.

[0250] In some embodiments provided in this application, the step of determining the target loss based on the second generated image, image sample, second depth map, and second direction representation data corresponding to each third observation view includes: projecting pixels in the second generated image corresponding to the third observation view onto the three-dimensional space corresponding to the reference game scene based on the second depth map and second direction representation data corresponding to the third observation view to obtain pixel projection points corresponding to the third observation view; projecting the pixel projection points corresponding to each third observation view onto the virtual imaging plane corresponding to the second virtual camera based on the second direction representation data corresponding to each third observation view to obtain pixel projection positions corresponding to each third observation view; and determining the target loss based on the difference between the pixel value of the pixel projection position corresponding to each third observation view and the corresponding pixel value in the image sample or second generated image corresponding to another third observation view.

[0251] Specifically, to ensure that the generative model can learn the mapping relationship between pixels and three-dimensional space, in some embodiments provided in this application, the electronic device first projects the pixels of the single-view image backward to the three-dimensional space to generate pixel projection points through the second depth map and the second direction representation data output by the game engine, and then projects the three-dimensional points forward to the virtual imaging plane of other observation views to obtain the pixel projection positions. Based on the statistical deviation of the pixel color difference at the projection position, the target loss is calculated, and the model is trained through the target loss, so that the model can learn the mapping relationship between pixels and three-dimensional space.

[0252] In some embodiments, a pixel projection point can be understood as a three-dimensional spatial coordinate point obtained by converting the pixel coordinates of a two-dimensional image to the three-dimensional world coordinate system of the scene through a second depth map and second direction representation data.

[0253] In some examples, in realistic mountain scenes generated by the game engine, after the coordinates of the surface depth of a mountain recorded by a certain pixel are converted, the resulting three-dimensional spatial coordinates of the mountain are the pixel projection point corresponding to that pixel.

[0254] In some embodiments, the pixel projection position can be understood as the two-dimensional pixel coordinate point obtained by projecting the already generated three-dimensional pixel projection point, combined with the second direction representation data corresponding to other viewpoint cameras, onto the imaging plane of the corresponding camera.

[0255] In some examples, the 3D projection points of the mountain range are projected onto the side-view camera image surrounding the mountain. The horizontal and vertical pixel coordinates that fall on the image are the pixel projection positions corresponding to the 3D points.

[0256] To more clearly illustrate the specific calculation process of the target loss and the model training process in the embodiments of this application, please refer to the following exemplary description: Make the game engine target the first i The second depth map generated from the third observation perspective is The initial image generation model is based on the first i The second generated image is generated from the second direction representation data corresponding to the third observation perspective. Then electronic devices can The pixels in the image are back-projected into a 3D point cloud, and then projected onto the image with the pixel in the image. i A third observation perspective with different views j The virtual imaging plane corresponding to each third observation viewpoint is used to calculate and generate frames. The difference in pixel color.

[0257] Specifically, through the first i Intrinsic parameter matrix of the geometric parameters of the third camera from a third observation perspective K i The pixel coordinates are back-projected into the camera space, and then through the first... i Rotation matrix in the geometric parameters of the third camera from a third observation perspective The pixel coordinates projected into camera space are transformed to the world coordinate system, thereby obtaining... Located in ( u , v The pixel projection point corresponding to the pixel at position ) This process can be expressed as formula (9), that is: (9) Then, Project to the j The virtual imaging plane corresponding to each third observation viewpoint is used to obtain the corresponding pixel projection position. This process can be expressed as formula (10), that is: (10) Then, calculate middle Pixel value at position and middle Difference between pixel values ​​at Specifically, it can be calculated using formula (11), that is: (11) in, V ij For the first i From a third observational perspective, the first j The third observation perspective corresponds to The set of pixels that are visible in the image.

[0258] Then, a weighted average of all viewpoint pairs is calculated to obtain the multiview reprojection loss. This is the target loss. The specific calculation method is shown in formula (12), that is: (12) Finally, the values ​​of each loss function are combined to obtain the joint loss. The initial image generation model is then updated using a joint loss. The joint loss... It can be calculated using formula (13), that is: (13) in, The temporal consistency loss obtained through optical flow consistency or inter-frame feature distance is used to apply smoothing constraints to the generation results of adjacent frames; Standard diffusion denoising loss; and All of these are weighted hyperparameters.

[0259] To avoid due to Larger The poor prediction quality leads to multi-view reprojection loss. The corresponding gradient degradation is addressed by reducing the weights during the initial training phase. To mitigate the impact of this loss, the weight value is gradually increased as the number of training steps increases.

[0260] also, The generation process includes: Suppose that during model training, the first... i The second-direction representation data corresponding to the third observation perspective are used as conditional attributes, and the model operates at the diffusion time step. Output noise prediction Then, the clear latent variables are predicted by inversely using the noise prediction. , This can be expressed as formula (14), that is: (14) Then, the model is tested via VAE. Decoding can be performed to obtain the i-th second generated image. .

[0261] Thus, in this embodiment, pixels in the second generated image corresponding to the third observation view can be projected onto the three-dimensional space corresponding to the reference game scene based on the second depth map and second direction representation data corresponding to the third observation view, to obtain pixel projection points corresponding to the third observation view. Furthermore, pixel projection points corresponding to each third observation view are projected onto the virtual imaging plane corresponding to the second virtual camera based on the second direction representation data corresponding to each third observation view, to obtain pixel projection positions corresponding to each third observation view. Finally, the target loss is determined based on the difference between the pixel value at each pixel projection position corresponding to the third observation view and the corresponding pixel value in the image sample or the second generated image corresponding to another third observation view. This allows the target loss to reflect the degree of difference between camera geometric parameters in multi-view images, enabling the model to accurately parse the camera parameters provided by the game engine after training, thereby outputting an image aligned with the engine coordinate system pixels.

[0262] In some embodiments provided in this application, the step of determining the target loss based on the second generated image, image sample, second depth map, and second direction representation data corresponding to each third observation viewpoint includes: determining the visibility mask of effective pixels in the second generated image based on the second depth map; determining the target pixels in the second generated image that are within a preset depth range based on the visibility mask; and determining the target loss based on the image sample, second depth map, second direction representation data, and target pixels in the second generated image corresponding to each third observation viewpoint. Wherein, the depth value corresponding to the effective pixel is greater than a preset threshold, and the visibility mask is used to indicate whether the pixel is within the preset depth range.

[0263] Specifically, when using depth information to calculate the loss, if all pixels of the image are directly selected for the loss calculation, the game scene depth map will contain a large number of abnormal depth pixels corresponding to the sky, near-section occlusion, and extremely distant empty scenes. The depth values ​​of these pixels do not conform to the actual spatial relationships of the scene. Furthermore, the participation of these pixels in the loss will generate erroneous pixel offset differences, causing abnormal fluctuations in the loss value. After backpropagation, erroneous gradients will be generated, ultimately leading to a decrease in the model's learning accuracy and a deterioration in the geometric alignment of the generated image.

[0264] Based on this, in some embodiments provided in this application, the electronic device generates a visibility mask by combining the second depth map output by the engine with a preset depth threshold. Invalid pixels with depths outside the range are filtered out by the mask, and only target pixels that meet the depth range requirements are selected to participate in subsequent loss calculations. This filters out the calculation errors caused by abnormal pixels, ensures the accuracy of target loss calculation, and optimizes the model training effect.

[0265] In some embodiments, the preset threshold can be understood as a pre-set depth threshold value, including a lower depth threshold and an upper depth threshold, used to define the boundary conditions for the effective depth of a pixel.

[0266] In some examples, when conducting training on architectural prop scenes, a lower threshold of 0.2m for near-section depth and an upper threshold of 300m for distant view are set. This set of values ​​is a set of preset thresholds.

[0267] In some embodiments, the visibility mask can be understood as a binary marker map with the same resolution size as the image and depth map, which distinguishes whether a pixel falls within the effective depth range by different marker values.

[0268] In some examples, a scene depth map with a resolution of 512×512 generates a mask of the same size, where a pixel value of 1 represents a valid pixel and 0 represents an invalid pixel to be discarded. This map is the visibility mask.

[0269] In some embodiments, an effective pixel can be understood as a pixel whose depth value is between a preset upper and lower depth threshold, corresponding to scene entity content and image pixels that can participate in reprojection calculation.

[0270] In some examples, the pixel depth corresponding to the cartoon house model is 3.5m, which is within the threshold range of 0.2m to 300m, and this pixel is considered a valid pixel.

[0271] In some embodiments, the preset depth range can be understood as the effective depth range formed by the combination of two preset thresholds above and below, used to determine whether a pixel can participate in the loss calculation.

[0272] In some examples, a continuous numerical range consisting of a minimum depth of 0.2m and a maximum depth of 300m corresponds to the preset depth range of the training scenario.

[0273] In some embodiments, the target pixel can be understood as all valid pixels that are retained after being filtered by a visibility mask, and these pixels will participate in the subsequent target loss calculation.

[0274] In some examples, after a building rendering is masked and the blank pixels of the sky and distant view are removed, the set of all pixels corresponding to the remaining walls, doors and windows is the target pixel.

[0275] To more clearly illustrate the complete process of filtering target pixels based on visibility masks and calculating target loss in the embodiments of this application, please refer to the following exemplary description: First, the electronic device exports training data from 16 different observation perspectives of the same scene through the game engine. Each set of data includes image samples, a second depth map, and third camera geometric parameters. After coordinate transformation, second orientation representation data corresponding to each perspective is generated. The orientation representation data is then fed into the initial image generation model for inference to obtain the second generated image.

[0276] Subsequently, the second depth map corresponding to each viewpoint is retrieved, matched with the pre-configured preset depth threshold, and the depth value is compared with the preset depth range pixel by pixel. Based on the comparison result, a visibility mask with the same size as the second generated image is generated.

[0277] Next, the generated visibility mask is used to filter pixels in the second generated image, removing invalid pixels such as sky and distant empty scenes that exceed the depth limit, and retaining target pixels that fall within the preset depth range to form a set. V That is, the set of visible pixels in the aforementioned formula (11). V .

[0278] Finally, through the set of visible pixels V For each target pixel, combined with image samples from the corresponding viewpoint, the second depth map, and the second orientation representation data, the target loss is calculated. Then, the weights of the initial image generation model are updated based on the target loss through backpropagation, completing a single training iteration. The above steps of selection, loss calculation, and parameter update are repeated until the model converges.

[0279] Thus, in this embodiment, the visibility mask of valid pixels in the second generated image can be determined based on the second depth map, and the target pixels in the second generated image within a preset depth range can be determined based on the visibility mask. The target loss can be determined based on the image sample corresponding to each third observation view, the second depth map, the second direction representation data, and the target pixels in the second generated image. This achieves the filtering of invalid pixels, avoids invalid pixels interfering with the loss calculation, and ensures the model training effect.

[0280] To facilitate better implementation of the 3D scene reconstruction method of this application, this application also provides a 3D scene reconstruction apparatus. Please refer to... Figure 4 , Figure 4 This is a schematic diagram of the structure of a three-dimensional scene reconstruction device provided in an embodiment of this application. The three-dimensional scene reconstruction device 300 can display a graphical user interface (GUI) via a terminal device. The GUI includes a virtual scene. The three-dimensional scene reconstruction device 300 may include: The first acquisition module 310 is used to acquire first camera trajectory data, wherein the first camera trajectory data includes the first camera geometric parameters of the first virtual camera of the game engine under multiple first observation perspectives; The first determining module 320 is used to determine the first direction characterization data corresponding to each first observation viewpoint based on the first camera trajectory data, wherein each first direction characterization data is used to indicate the direction of the first virtual camera when observing the virtual imaging plane through the corresponding first observation viewpoint. The first input module 330 is used to input the first direction representation data corresponding to each first observation viewpoint into the pre-trained target image generation model to obtain the first generated image corresponding to each first observation viewpoint. The reconstruction module 340 is used to generate target three-dimensional Gaussian splash data of the target virtual scene based on the first camera geometric parameters of each first observation view and the corresponding first generated image, so as to reconstruct the target virtual scene through the target three-dimensional Gaussian splash data.

[0281] In some embodiments, the first camera geometric parameters include first camera intrinsic parameters and first camera extrinsic parameters. Furthermore, the first determining module 320 is further configured to perform coordinate system matching processing on the first coordinate system corresponding to the game engine, so that the coordinate axes, chirality attributes, and world scale of the first coordinate system match with a pre-determined second coordinate system to obtain a third coordinate system. And based on the first camera intrinsic parameters corresponding to each first observation viewpoint and the first camera extrinsic parameters corresponding to each first observation viewpoint in the third coordinate system, it determines the first direction representation data corresponding to each first observation viewpoint. Here, the chirality attribute is either left-handed or right-handed.

[0282] In some embodiments, the first determining module 320 is further configured to determine the mapping position of each pixel in the virtual imaging plane in the third coordinate system based on the intrinsic parameters of the first camera corresponding to the first observation viewpoint; determine the observation direction characterization parameter corresponding to each pixel in the virtual imaging plane based on the mapping position of each pixel in the virtual imaging plane and the extrinsic parameters of the first camera in the third coordinate system; and obtain the first direction characterization data corresponding to each first observation viewpoint based on the observation direction characterization parameter corresponding to each pixel in the virtual imaging plane corresponding to each first observation viewpoint. The observation direction characterization parameter is used to indicate the observation direction of the first virtual camera when observing a pixel at the corresponding first observation viewpoint.

[0283] In some embodiments, the first determining module 320 is further configured to normalize the position of each pixel in the virtual imaging plane according to the first camera intrinsic parameters corresponding to the first observation viewpoint, to obtain the normalized position of each pixel in the virtual imaging plane, and to map each pixel in the virtual imaging plane to the third coordinate system according to the normalized position of each pixel in the virtual imaging plane, to obtain the mapped position of each pixel in the virtual imaging plane.

[0284] In some embodiments, the first determining module 320 is further configured to determine the optical center position of the first virtual camera corresponding to the first observation viewpoint based on the first camera extrinsic parameters corresponding to the first observation viewpoint in the third coordinate system, and to construct the direction from the optical center position of the first virtual camera to the mapping position of each pixel based on the mapping position of each pixel in the virtual imaging plane corresponding to the first observation viewpoint and the optical center position of the first virtual camera corresponding to the first observation viewpoint, thereby obtaining the observation direction characterization parameters corresponding to each pixel in the virtual imaging plane.

[0285] In some embodiments, the first input module 330 is further configured to concatenate the first direction representation data with the input feature map of the first layer of the denoising network of the target image generation model along the channel dimension, and to inject the global pooling features of the first direction representation data into at least one residual block of the denoising network through the adaptive layer normalization module.

[0286] In some embodiments, the first input module 330 is further configured to determine each first direction representation data as a conditional attribute of the target image generation model, input each first direction representation data into the target image generation model, and based on the target image generation model, perform denoising sampling processing on the initial noise data corresponding to each first observation viewpoint to obtain denoised representation data corresponding to each first observation viewpoint, and perform decoding processing on the denoised representation data corresponding to each first observation viewpoint to obtain the first generated image corresponding to each first observation viewpoint.

[0287] In some embodiments, the 3D scene reconstruction apparatus 300 further includes a second acquisition module, a second determination module, and an encoding module. The second acquisition module is used to acquire a reference image corresponding to at least one reference observation viewpoint and the geometric parameters of a second camera corresponding to the reference observation viewpoint, wherein the second camera geometric parameters include second camera intrinsic parameters and second camera extrinsic parameters. The second determination module is used to determine second direction representation data corresponding to the reference observation viewpoint based on the second camera geometric parameters. The encoding module is used to encode the reference image to obtain reference image encoded data. The first input module 330 is further used to determine the reference image encoded data, the second direction representation data, and each first direction representation data as conditional attributes, and input the reference image encoded data, the second direction representation data, and each first direction representation data into the target image generation model.

[0288] In some embodiments, the reconstruction module 340 is further configured to construct initial point cloud data of the target virtual scene based on the first camera geometric parameters corresponding to each first observation view, and initialize the three-dimensional Gaussian splash data of the target virtual scene based on the initial point cloud data to obtain initial three-dimensional Gaussian splash data, and update the initial three-dimensional Gaussian splash data based on the first camera geometric parameters and the first generated image corresponding to each first observation view, and determine the updated initial three-dimensional Gaussian splash data as the target three-dimensional Gaussian splash data, wherein the parameter values ​​of the first camera geometric parameters remain unchanged during the update process.

[0289] In some embodiments, the reconstruction module 340 is further configured to obtain a first depth map of the target virtual scene corresponding to each first observation view through the game engine, and to project the pixels in the first generated image corresponding to each first observation view onto the three-dimensional space corresponding to the target virtual scene according to the first camera geometric parameters and the first depth map to obtain a three-dimensional point cloud corresponding to each first observation view, and to merge the three-dimensional point clouds corresponding to all first observation views to obtain initial point cloud data.

[0290] In some embodiments, the reconstruction module 340 is further configured to render a rendering depth map corresponding to each first observation view based on the initial three-dimensional Gaussian splash data, and determine a depth consistency parameter based on the first depth map and the rendering depth map corresponding to each first observation view, and update the initial three-dimensional Gaussian splash data based on the depth consistency parameter.

[0291] In some embodiments, the reconstruction module 340 is further configured to input the first generated image corresponding to each first observation viewpoint into a pre-trained depth estimation model, so that the depth estimation model estimates the depth of the target virtual scene based on each first generated image, obtains the depth estimation result corresponding to each first generated image, and constructs initial point cloud data based on the first camera geometric parameters corresponding to each first observation viewpoint and the depth estimation result corresponding to each first generated image.

[0292] In some embodiments, the three-dimensional scene reconstruction device 300 further includes a third acquisition module, a third determination module, a third input module, and a generation module.

[0293] The third acquisition module acquires multiple training samples, each of which includes multiple image samples obtained by a second virtual camera observing the reference game scene from multiple third viewing angles, as well as the third camera geometric parameters and a second depth map of the second virtual camera in each third viewing angle. The second depth map is generated by the game engine. The third determination module determines the second orientation representation data corresponding to the third viewing angle based on the third camera geometric parameters. The third input module inputs the second orientation representation data into the initial image generation model to be trained, so that the initial image generation model generates a second generated image corresponding to the third viewing angle based on the second orientation representation data. The generation module trains the initial image generation model based on the second generated image, image samples, second depth map, and second orientation representation data to obtain a target image generation model.

[0294] In some embodiments, the generation module is further configured to determine a target loss based on the second generated image corresponding to each of the third observation viewpoints in the training samples, the image sample, the second depth map, and the second orientation representation data, and to train the initial image generation model based on the target loss to obtain a target image generation model. The target loss is used to indicate the differences between the second generated images corresponding to each of the third observation viewpoints.

[0295] In some embodiments, the generation module is further configured to: project pixels in the second generated image corresponding to the third observation viewpoint onto the three-dimensional space corresponding to the reference game scene based on the second depth map and second direction characterization data corresponding to the third observation viewpoint, to obtain pixel projection points corresponding to the third observation viewpoint; and project pixel projection points corresponding to each third observation viewpoint onto the virtual imaging plane corresponding to the second virtual camera based on the second direction characterization data corresponding to each third observation viewpoint, to obtain pixel projection positions corresponding to each third observation viewpoint; and determine the target loss based on the difference between the pixel value of the pixel projection position corresponding to each third observation viewpoint and the corresponding pixel value in the image sample or the second generated image corresponding to another third observation viewpoint.

[0296] In some embodiments, the generation module is further configured to determine a visibility mask for valid pixels in the second generated image based on the second depth map, and to determine target pixels in the second generated image that are within a preset depth range based on the visibility mask, and to determine a target loss based on the image samples corresponding to each third observation viewpoint, the second depth map, the second direction representation data, and the target pixels in the second generated image. Wherein, the depth value corresponding to a valid pixel is greater than a preset threshold, and the visibility mask is used to indicate whether the pixel is within the preset depth range.

[0297] Each unit in the aforementioned 3D scene reconstruction device 300 can be implemented entirely or partially through software, hardware, or a combination thereof. Each unit can be embedded in or independent of the processor in the electronic device in hardware form, or stored in the memory of the electronic device in software form, so that the processor can call and execute the operations corresponding to each unit.

[0298] The 3D scene reconstruction device 300 can be integrated into a terminal or server that has memory and a processor and thus computing power, or the 3D scene reconstruction device 300 can be the terminal or server.

[0299] Optionally, this application also provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0300] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device may be a terminal or a server. Figure 5 As shown, the electronic device 400 includes a processor 401 with one or more processing cores, a memory 402 with one or more computer-readable storage media, and a computer program stored in the memory 402 and executable on the processor. The processor 401 and the memory 402 are electrically connected. Those skilled in the art will understand that the electronic device structure shown in the figures does not constitute a limitation on the electronic device, and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0301] The processor 401 is the control center of the electronic device 400. It connects various parts of the electronic device 400 through various interfaces and lines. By running or loading software programs and / or modules stored in the memory 402, and calling data stored in the memory 402, it executes various functions of the electronic device 400 and processes data, thereby performing overall processing of the electronic device 400.

[0302] In this embodiment, the processor 401 in the electronic device 400 loads the instructions corresponding to the processes of one or more computer programs into the memory 402 according to the following steps, and the processor 401 runs the computer programs stored in the memory 402 to realize various functions: Acquire first camera trajectory data, wherein the first camera trajectory data includes the first camera geometric parameters of the first virtual camera under multiple first observation views; Based on the trajectory data of the first camera, first direction characterization data corresponding to each first observation viewpoint is determined, wherein each first direction characterization data is used to indicate the direction of the first virtual camera when observing the virtual imaging plane through the corresponding first observation viewpoint; Input the first direction representation data corresponding to each first observation viewpoint into the pre-trained target image generation model to obtain the first generated image corresponding to each first observation viewpoint; Based on the first camera geometric parameters and the corresponding first generated image for each first observation viewpoint, target three-dimensional Gaussian splash data of the target virtual scene is generated, so as to reconstruct the target virtual scene through the target three-dimensional Gaussian splash data.

[0303] For details on the implementation of each of the above operations, please refer to the previous examples, which will not be repeated here.

[0304] Optional, such as Figure 5 As shown, the electronic device 400 also includes: a display screen 403, a radio frequency circuit 404, an audio circuit 405, an input unit 406, and a power supply 407. The processor 401 is electrically connected to the display screen 403, the radio frequency circuit 404, the audio circuit 405, the input unit 406, and the power supply 407. Those skilled in the art will understand that... Figure 5 The electronic device structure shown does not constitute a limitation on the electronic device and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0305] The display screen 403 can be used to display a graphical user interface (GUI) and receive operation commands generated by the user interacting with the GUI. The display screen 403 may include a display panel and a touch panel. The display panel can be used to display information input by the user or information provided to the user, as well as various graphical user interfaces of the electronic device. These graphical user interfaces can be composed of graphics, text, icons, video, and any combination thereof. The touch panel can be used to collect touch operations performed by the user on or near it (such as operations performed by the user using a finger, stylus, or any suitable object or accessory on or near the touch panel), generate corresponding operation commands, and execute the corresponding program. Optionally, the touch panel may include a touch detection device and a touch controller. The touch detection device detects the user's touch location and the signal generated by the touch operation, and transmits the signal to the touch controller. The touch controller receives touch information from the touch detection device, converts it into touch point coordinates, sends it to the processor 401, and can receive and execute commands from the processor 401. The touch panel can cover the display panel. When the touch panel detects a touch operation on or near it, it transmits the information to the processor 401 to determine the type of touch event. Subsequently, the processor 401 provides corresponding visual output on the display panel according to the type of touch event. In this embodiment, the touch panel and the display panel can be integrated into the display screen 403 to achieve input and output functions. However, in some embodiments, the touch panel and the display screen 403 can be implemented as two independent components to achieve input and output functions. That is, the display screen 403 can also be used as part of the input unit 406 to achieve input functions.

[0306] The radio frequency circuit 404 can be used to transmit and receive radio frequency signals to establish wireless communication with network devices or other electronic devices, and to transmit and receive signals with network devices or other electronic devices.

[0307] Audio circuit 405 can be used to provide an audio interface between a user and an electronic device via a speaker and a microphone. Audio circuit 405 can convert received audio data into electrical signals and transmit them to the speaker, where the speaker converts them into sound signals for output. Conversely, the microphone converts collected sound signals into electrical signals, which are then received by audio circuit 405, converted back into audio data, and then processed by processor 401 before being transmitted via radio frequency circuit 404 to, for example, another electronic device, or output to memory 402 for further processing. Audio circuit 405 may also include an earphone jack to provide communication between peripheral headphones and electronic devices.

[0308] The input unit 406 can be used to receive input numbers, characters, or object feature information (such as fingerprints, iris, facial information, etc.), and to generate keyboard, mouse, joystick, optical, or trackball signal inputs related to user settings and function control.

[0309] Power supply 407 is used to supply power to various components of electronic device 400. Optionally, power supply 407 can be logically connected to processor 401 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system. Power supply 407 may also include one or more DC or AC power supplies, recharging systems, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components.

[0310] although Figure 5 As not shown in the diagram, the electronic device 400 may also include a camera, sensor, wireless fidelity module, Bluetooth module, etc., which will not be described in detail here.

[0311] This application also provides a computer-readable storage medium for storing a computer program. This computer-readable storage medium can be applied to a computer device, and the computer program causes the computer device to execute the corresponding processes in the three-dimensional scene reconstruction method described in the embodiments of this application; for brevity, these will not be elaborated upon here.

[0312] This application also provides a computer program product including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the corresponding processes in the 3D scene reconstruction method described in the embodiments of this application. For simplicity, further details are omitted here.

[0313] This application also provides a computer program comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the corresponding processes in the 3D scene reconstruction method of this application. For brevity, these details are not elaborated here.

[0314] It should be understood that the processor in this application may be an integrated circuit chip with signal processing capabilities. In implementation, the steps of the above method embodiments can be completed by integrated logic circuits in the processor's hardware or by instructions in software form. The processor described above can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules can be located in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. This storage medium is located in memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method.

[0315] It is understood that the memory in the embodiments of this application can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDR SDRAM), Enhanced Synchronous DRAM (ESDRAM), Synchlink DRAM (SLDRAM), and Direct Rambus RAM (DR RAM). It should be noted that the memory used in the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0316] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0317] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0318] In the embodiments of this application, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0319] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0320] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0321] In addition, the functional units in this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0322] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer or a server) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, ROM, RAM, magnetic disks, or optical disks.

[0323] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for reconstructing a three-dimensional scene, characterized in that, The method includes: Acquire first camera trajectory data, wherein the first camera trajectory data includes first camera geometric parameters of the first virtual camera under multiple first observation views; Based on the first camera trajectory data, first direction characterization data corresponding to each first observation viewpoint is determined, wherein each first direction characterization data is used to indicate the direction of the first virtual camera when observing the virtual imaging plane through the corresponding first observation viewpoint; Input the first direction representation data corresponding to each first observation viewpoint into the pre-trained target image generation model to obtain the first generated image corresponding to each first observation viewpoint. Based on the first camera geometric parameters of each first observation viewpoint and the corresponding first generated image, target three-dimensional Gaussian splash data of the target virtual scene is generated, so as to reconstruct the target virtual scene through the target three-dimensional Gaussian splash data.

2. The three-dimensional scene reconstruction method according to claim 1, characterized in that, The first camera geometric parameters include first camera intrinsic parameters and first camera extrinsic parameters. The step of determining the first direction representation data corresponding to each first observation viewpoint based on the first camera trajectory data includes: A coordinate system matching process is performed on the first coordinate system corresponding to the first camera trajectory data so that the coordinate axes, rotation, origin position and scale of the first coordinate system are matched with the predetermined second coordinate system to obtain a third coordinate system. Based on the intrinsic parameters of the first camera corresponding to each first observation viewpoint and the extrinsic parameters of the first camera corresponding to each first observation viewpoint in the third coordinate system, the first direction representation data corresponding to each first observation viewpoint is determined.

3. The three-dimensional scene reconstruction method according to claim 2, characterized in that, The step of determining the first direction representation data corresponding to each first observation viewpoint based on the first camera intrinsic parameters corresponding to each first observation viewpoint and the first camera extrinsic parameters corresponding to each first observation viewpoint in the third coordinate system includes: Based on the first camera intrinsic parameters corresponding to the first observation viewpoint, determine the mapping position of each pixel in the virtual imaging plane in the third coordinate system; Based on the mapping position of each pixel in the virtual imaging plane and the first camera extrinsic parameters in the third coordinate system, the observation direction characterization parameter corresponding to each pixel in the virtual imaging plane is determined, wherein the observation direction characterization parameter is used to indicate the observation direction of the first virtual camera when observing the pixel at the corresponding first observation view. Based on the observation direction representation parameters of each pixel in the virtual imaging plane corresponding to each first observation viewpoint, the first direction representation data corresponding to each first observation viewpoint is obtained.

4. The three-dimensional scene reconstruction method according to claim 3, characterized in that, The step of determining the mapping position of each pixel in the virtual imaging plane in the third coordinate system based on the first camera intrinsic parameters corresponding to the first observation viewpoint includes: Based on the first camera intrinsic parameters corresponding to the first observation viewpoint, the position of each pixel in the virtual imaging plane is normalized to obtain the normalized position of each pixel in the virtual imaging plane. Based on the normalized position of each pixel in the virtual imaging plane, each pixel in the virtual imaging plane is mapped to the third coordinate system to obtain the mapped position of each pixel in the virtual imaging plane.

5. The three-dimensional scene reconstruction method according to claim 3, characterized in that, The step of determining the observation direction characterization parameter corresponding to each pixel in the virtual imaging plane based on the mapped position of each pixel in the virtual imaging plane and the first camera extrinsic parameters in the third coordinate system includes: Based on the first camera extrinsic parameters corresponding to the first observation viewpoint in the third coordinate system, determine the optical center position of the first virtual camera corresponding to the first observation viewpoint; Based on the mapping position of each pixel in the virtual imaging plane corresponding to the first observation viewpoint and the optical center position of the first virtual camera corresponding to the first observation viewpoint, a direction from the optical center position of the first virtual camera to the mapping position of each pixel is constructed to obtain the observation direction characterization parameter corresponding to each pixel in the virtual imaging plane.

6. The three-dimensional scene reconstruction method according to claim 1, characterized in that, The step of inputting the first direction representation data corresponding to each first observation viewpoint into the pre-trained target image generation model includes: The first direction representation data is concatenated with the input feature map of the first layer of the denoising network of the target image generation model along the channel dimension. The global pooling features of the first direction representation data are injected into at least one residual block of the denoising network through an adaptive layer normalization module.

7. The three-dimensional scene reconstruction method according to claim 1, characterized in that, The step of inputting the first direction representation data corresponding to each first observation viewpoint into a pre-trained target image generation model to obtain a first generated image corresponding to each first observation viewpoint includes: Each first direction representation data is determined as a conditional attribute of the target image generation model, and each first direction representation data is input into the target image generation model; Based on the target image generation model, the initial noise data corresponding to each first observation view is subjected to denoising sampling processing to obtain the denoised representation data corresponding to each first observation view. The denoised representation data corresponding to each first observation viewpoint is decoded to obtain the first generated image corresponding to each first observation viewpoint.

8. The three-dimensional scene reconstruction method according to claim 7, characterized in that, The method further includes: Acquire a reference image corresponding to at least one reference observation viewpoint and the second camera geometric parameters corresponding to the reference observation viewpoint, wherein the second camera geometric parameters include second camera intrinsic parameters and second camera extrinsic parameters; Based on the geometric parameters of the second camera, determine the second direction characterization data corresponding to the reference observation angle; The reference image is encoded to obtain reference image encoded data; The step of determining each first direction representation data as a conditional attribute of the target image generation model and inputting each first direction representation data into the target image generation model includes: The reference image encoding data, the second direction representation data, and each of the first direction representation data are all determined as the conditional attributes, and the reference image encoding data, the second direction representation data, and each of the first direction representation data are input into the target image generation model.

9. The three-dimensional scene reconstruction method according to claim 1, characterized in that, The step of generating target 3D Gaussian splash data of the target virtual scene based on the first camera geometric parameters of each first observation viewpoint and the corresponding first generated image, and reconstructing the target virtual scene using the target 3D Gaussian splash data, includes: Based on the geometric parameters of the first camera corresponding to each first observation viewpoint, the initial point cloud data of the target virtual scene is constructed; Based on the initial point cloud data, the three-dimensional Gaussian splash data of the target virtual scene is initialized to obtain the initial three-dimensional Gaussian splash data; Based on the first camera geometric parameters and the first generated image corresponding to each first observation viewpoint, the initial three-dimensional Gaussian splash data is updated, and the updated initial three-dimensional Gaussian splash data is determined as the target three-dimensional Gaussian splash data. During the update process, the parameter values ​​of the first camera geometric parameters remain unchanged.

10. The three-dimensional scene reconstruction method according to claim 9, characterized in that, The step of constructing the initial point cloud data of the target virtual scene based on the geometric parameters of the first camera corresponding to each first observation viewpoint includes: The first depth map of the target virtual scene corresponding to each of the first observation perspectives is obtained through the game engine. Based on the first camera geometric parameters and the first depth map, the pixels in the first generated image corresponding to each first observation view are projected onto the three-dimensional space corresponding to the target virtual scene to obtain the three-dimensional point cloud corresponding to each first observation view. The three-dimensional point clouds corresponding to all the first observation viewpoints are merged to obtain the initial point cloud data.

11. The three-dimensional scene reconstruction method according to claim 10, characterized in that, The step of updating the initial 3D Gaussian splash data based on the first camera geometric parameters and the first generated image corresponding to each first observation viewpoint includes: The rendering depth map corresponding to each first observation viewpoint is obtained by rendering the initial three-dimensional Gaussian splash data. Determine depth consistency parameters based on the first depth map and the rendered depth map corresponding to each first observation viewpoint; The initial three-dimensional Gaussian splash data is updated based on the depth consistency parameter.

12. The three-dimensional scene reconstruction method according to claim 9, characterized in that, The step of constructing the initial point cloud data of the target virtual scene based on the geometric parameters of the first camera corresponding to each first observation viewpoint includes: The first generated image corresponding to each first observation viewpoint is input into a pre-trained depth estimation model, so that the depth estimation model estimates the depth of the target virtual scene based on each first generated image, and obtains the depth estimation result corresponding to each first generated image; The initial point cloud data is constructed based on the first camera geometric parameters corresponding to each first observation viewpoint and the depth estimation results corresponding to each first generated image.

13. The three-dimensional scene reconstruction method according to claim 1, characterized in that, The training steps of the target image generation model include: Multiple training samples are acquired, wherein each training sample includes multiple image samples obtained by a second virtual camera observing the reference game scene from multiple third observation perspectives, as well as the third camera geometric parameters and a second depth map of the second virtual camera in each of the third observation perspectives, the second depth map being generated by the game engine; Based on the geometric parameters of the third camera, determine the second direction representation data corresponding to the third observation viewpoint; The second direction representation data is input into the initial image generation model to be trained, so that the initial image generation model generates a second generated image corresponding to the third observation viewpoint based on the second direction representation data; The initial image generation model is trained based on the second generated image, the image sample, the second depth map, and the second orientation representation data to obtain the target image generation model.

14. The three-dimensional scene reconstruction method according to claim 13, characterized in that, The step of training the initial image generation model based on the second generated image, the image sample, the second depth map, and the second orientation representation data to obtain the target image generation model includes: Based on the second generated image, the image sample, the second depth map, and the second orientation representation data corresponding to each third observation view in the training samples, a target loss is determined, wherein the target loss is used to indicate the difference between the second generated images corresponding to each third observation view. The initial image generation model is trained based on the target loss to obtain the target image generation model.

15. The three-dimensional scene reconstruction method according to claim 14, characterized in that, The step of determining the target loss based on the second generated image corresponding to each third observation viewpoint in the training samples, the image sample, the second depth map, and the second orientation representation data includes: Based on the second depth map and the second direction representation data corresponding to the third observation view, the pixels in the second generated image corresponding to the third observation view are projected onto the three-dimensional space corresponding to the reference game scene to obtain the pixel projection points corresponding to the third observation view. Based on the second direction characterization data corresponding to each third observation view, the pixel projection points corresponding to each third observation view are projected onto the virtual imaging plane corresponding to the second virtual camera to obtain the pixel projection position corresponding to each third observation view. The target loss is determined based on the difference between the pixel value at the pixel projection position corresponding to each of the third observation views and the corresponding pixel value in the image sample or the second generated image corresponding to another third observation view.

16. The three-dimensional scene reconstruction method according to claim 14, characterized in that, The step of determining the target loss based on the second generated image corresponding to each third observation viewpoint in the training samples, the image sample, the second depth map, and the second orientation representation data includes: Based on the second depth map, a visibility mask for valid pixels in the second generated image is determined, wherein the depth value corresponding to the valid pixel is greater than a preset threshold, and the visibility mask is used to indicate whether the pixel is within a preset depth range. Based on the visibility mask, target pixels in the second generated image that are within the preset depth range are determined; The target loss is determined based on the image sample corresponding to each third observation viewpoint, the second depth map, the second orientation representation data, and the target pixel in the second generated image.

17. A three-dimensional scene reconstruction device, characterized in that, The device includes: The first acquisition module is used to acquire first camera trajectory data, wherein the first camera trajectory data includes the first camera geometric parameters of the first virtual camera under multiple first observation views; The first determining module is configured to determine first direction characterization data corresponding to each first observation viewpoint based on the first camera trajectory data, wherein each first direction characterization data is used to indicate the direction of the first virtual camera when observing the virtual imaging plane through the corresponding first observation viewpoint. The first input module is used to input the first direction representation data corresponding to each first observation viewpoint into the pre-trained target image generation model to obtain the first generated image corresponding to each first observation viewpoint. The reconstruction module is used to generate target three-dimensional Gaussian splash data of the target virtual scene based on the first camera geometric parameters of each first observation view and the corresponding first generated image, so as to reconstruct the target virtual scene through the target three-dimensional Gaussian splash data.

Images