A three-dimensional scene reconstruction method, device, system, terminal and storage medium

Abstract

This invention provides a method, apparatus, system, terminal, and storage medium for 3D scene reconstruction. The method includes: responding to a rendering task execution command, acquiring the current field of view of several user terminals, and evaluating the rendering task volume within the current field of view of each user terminal; based on the rendering task volume, modeling a cross-layer optimization problem based on end-to-end latency and rendering quality constraints with the goal of minimizing system energy consumption; solving the cross-layer optimization problem to obtain optimization variables, including a target rendering model selected from a set of candidate rendering models; and based on the optimization variables, distributing the rendering task to several distributed nodes so that the distributed nodes can perform collaborative rendering to reconstruct the 3D scene. This application, by considering the differences in field of view, constructs a cross-layer optimization problem and ensures that end-to-end latency and rendering quality constraints are met, reduces the energy consumption of the rendering task and solves the resource optimization problem of distributed 3D reconstruction.

Description

Technical Field

[0001] This invention relates to the field of three-dimensional reconstruction technology, and in particular to a three-dimensional scene reconstruction method, apparatus, system, terminal and storage medium. Background Technology

[0002] XR (Extended Reality) is an immersive rendering technology. With the development of Extended Reality (XR) technology, 3D scene reconstruction technology has become a key component of XR systems. In particular, 3D Gaussian Splatting (3DGS) technology has provided a new paradigm for immersive rendering. However, this 3DGS-based XR service is usually accompanied by an explosive increase in computational and communication data volume, making it difficult for the system to simultaneously meet users' comprehensive needs for energy consumption, end-to-end latency, and rendering quality under limited communication and computing resources. Especially in near-field communication scenarios, although a high-speed, low-latency immersive experience can be achieved, the comprehensive requirements of users for the system pose a significant challenge to resource-constrained XR systems.

[0003] Traditional 3D reconstruction methods mainly use implicit scene representation techniques. The scenes generated by this method cannot be spatially segmented independently because the content from any viewpoint involves panoramic information. Therefore, it cannot fully utilize the advantages of explicit spatial representation to solve the resource optimization problem of distributed 3D reconstruction.

[0004] Therefore, existing technologies have shortcomings and need to be improved and developed. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to provide a three-dimensional scene reconstruction method, device, system, terminal and storage medium to address the above-mentioned defects of the prior art. The aim is to solve the problem of resource optimization in distributed three-dimensional reconstruction, since the content of any viewpoint involves panoramic information, it is impossible to fully utilize the advantages of explicit spatial representation.

[0006] The technical solution adopted by this invention to solve the technical problem is as follows: A three-dimensional scene reconstruction method, comprising: Respond to rendering task execution instructions, obtain the current field of view of several user terminals, and evaluate the amount of rendering task within the current field of view of each user terminal; Based on the rendering workload, with the goal of minimizing system energy consumption, a cross-layer optimization problem based on end-to-end latency and rendering quality constraints is modeled. Solving the cross-layer optimization problem yields optimization variables, which include the target rendering model selected from the candidate rendering model set. Based on the optimization variables, the rendering task is distributed to several distributed nodes so that the distributed nodes can perform collaborative rendering to reconstruct the 3D scene.

[0007] In one embodiment of this application, based on the rendering task volume and with the objective of minimizing system energy consumption, a cross-layer optimization problem based on end-to-end latency and rendering quality constraints is modeled, including: Based on the amount of rendering tasks, determine the rendering energy consumption and transmission energy consumption corresponding to several distributed nodes, and obtain the system energy consumption based on the rendering energy consumption and transmission energy consumption. Determine the constraints for rendering task allocation, transmission power, rendering quality, and latency of real-time rendering; With the goal of minimizing system energy consumption, and with constraints such as rendering task allocation, transmission power, rendering quality, and real-time rendering latency, a cross-layer optimization problem based on end-to-end latency and rendering quality constraints is modeled.

[0008] In one embodiment of this application, determining the rendering energy consumption and transmission energy consumption corresponding to several distributed nodes based on the rendering task volume includes: The rendering energy consumption of the distributed nodes is obtained by acquiring and using the effective switched capacitors, computational efficiency, computational resources, rendering task volume on the user side, data size corresponding to each Gaussian ellipsoid in the rendering task volume, and task allocation variables on the user side. The downlink transmission delay of the user terminal is obtained by acquiring and based on the data size of the image stream when the user terminal transmits the completed rendering task, the channel bandwidth, the transmission rate, the factor of the data size of adjacent Gaussian ellipsoids, the amount of rendering task of the user terminal, and the data size corresponding to each Gaussian ellipsoid. Obtain the beamforming vector at the user end, and obtain the transmission power consumption corresponding to the distributed node based on the downlink transmission delay and the beamforming vector.

[0009] In one embodiment of this application, determining rendering task allocation constraints, transmission power constraints, rendering quality constraints, and real-time rendering latency constraints includes: Obtain and obtain rendering task allocation constraints based on the user-side task allocation variables; Obtain a preset transmission power threshold and the beamforming vector of the user terminal, and obtain a transmission power constraint based on the transmission power threshold and the beamforming vector of the user terminal; Obtain the voxel size and preset minimum point cloud density corresponding to the current field of view of the user terminal, and obtain the rendering quality constraints based on the rendering task volume, voxel size and preset minimum point cloud density; Obtain the rendering latency at the user end, and derive the latency constraint for real-time rendering based on the rendering latency at the user end and the downlink transmission latency.

[0010] In one embodiment of this application, solving the cross-layer optimization problem to obtain optimization variables includes: The cross-layer optimization problem is decomposed into several sub-problems, including: updating task allocation variables, updating transmission beamforming vectors, and model selection. Solve the subproblems to update the task assignment variables, the signals received by the user, the beamforming vectors, and the target rendering model; The task allocation variables, signals received by the user, beamforming vectors, and target rendering models are continuously updated iteratively until the stopping condition is met. The task allocation variables obtained after stopping updates, the signals received by the user, the beamforming vector, and the target rendering model are used as optimization variables for distributed nodes.

[0011] In one embodiment of this application, solving various sub-problems to update task allocation variables, signals received by the user terminal, beamforming vectors, and target rendering models includes: The subproblem of updating the task allocation variable is transformed into a standard integer linear programming problem. An integer programming solver is used to obtain the optimal solution corresponding to the subproblem of updating the task allocation variable, so as to update the task allocation variable. The subproblem of updating the transmission beamforming vector is transformed into an unequal transformation subproblem using AM-GM unequal transformation. The unequal transformation subproblem is solved to obtain a closed-form solution. The signal received by the user terminal is updated based on the closed-form solution. The unequal transformation subproblem is transformed into an approximate convex problem. The approximate convex problem is solved to obtain the stationary point solution. The beamforming vector is updated based on the closed solution and the stationary point solution. Solve the closed-form optimal solution to the subproblem of model selection, and update the target rendering model based on the closed-form optimal solution.

[0012] This application also provides a three-dimensional scene reconstruction device, including: The response module is used to respond to rendering task execution instructions, obtain the current field of view of several user terminals, and evaluate the amount of rendering task within the current field of view of each user terminal. The modeling module is used to model a cross-layer optimization problem based on end-to-end latency and rendering quality constraints, with the goal of minimizing system energy consumption, according to the rendering task volume. The solution module is used to solve the cross-layer optimization problem and obtain optimization variables, including the target rendering model selected from the candidate rendering model set. The reconstruction module is used to allocate the rendering task to several distributed nodes based on the optimization variables, so that the several distributed nodes can perform collaborative rendering to reconstruct the 3D scene.

[0013] This application also provides a three-dimensional scene reconstruction system, including: several user terminals, a central node, and distributed nodes; The user terminal is used for near-field wireless communication with the central node and distributed nodes, and for sending rendering tasks to the central node. The central node is used to respond to rendering task execution instructions, obtain the current field of view of several user terminals, and evaluate the rendering task volume within the current field of view of each user terminal; based on the rendering task volume, with the goal of minimizing system energy consumption, a cross-layer optimization problem based on end-to-end latency and rendering quality constraints is modeled; the cross-layer optimization problem is solved to obtain optimization variables, which include the target rendering model selected from the candidate rendering model set; based on the optimization variables, the rendering task is allocated to several distributed nodes; Several distributed nodes are used for collaborative rendering to reconstruct the 3D scene.

[0014] This application also provides a terminal, including: a memory, a processor, and a three-dimensional scene reconstruction program stored in the memory and executable on the processor, wherein the three-dimensional scene reconstruction program, when executed by the processor, implements the steps of the three-dimensional scene reconstruction method as described above.

[0015] This application also provides a computer-readable storage medium storing a computer program that can be executed to implement the steps of the three-dimensional scene reconstruction method described above.

[0016] This invention provides a method, apparatus, system, terminal, and storage medium for 3D scene reconstruction. The method includes: responding to a rendering task execution command, acquiring the current field of view of several user terminals, and evaluating the rendering task volume within the current field of view of each user terminal; based on the rendering task volume, modeling a cross-layer optimization problem based on end-to-end latency and rendering quality constraints with the goal of minimizing system energy consumption; solving the cross-layer optimization problem to obtain optimization variables, including a target rendering model selected from a set of candidate rendering models; and based on the optimization variables, distributing the rendering task to several distributed nodes so that the distributed nodes can perform collaborative rendering to reconstruct the 3D scene. This application considers the differences in field of view, constructs a cross-layer optimization problem, comprehensively considers multi-dimensional resources and decisions such as rendering model selection, transmission precoding, and rendering task offloading, and ensures that end-to-end latency and rendering quality constraints are met, thereby reducing the energy consumption of the rendering task, fully utilizing the advantages of explicit spatial representation, and solving the resource optimization problem of distributed 3D reconstruction. Attached Figure Description

[0017] Figure 1This is a flowchart of a preferred embodiment of the three-dimensional scene reconstruction method in this invention.

[0018] Figure 2 This is a system schematic diagram of a preferred embodiment of the three-dimensional scene reconstruction method in this invention.

[0019] Figure 3 This is the pseudocode of the overall algorithm of a preferred embodiment of the three-dimensional scene reconstruction method in this invention.

[0020] Figure 4 This is a graph showing the convergence performance of the algorithm of this invention under different settings.

[0021] Figure 5 This is a performance comparison diagram of different rendering schemes.

[0022] Figure 6(a) is a visualization of the corresponding images of the real scene, high-precision rendering and low-precision rendering based on the open source dataset in this invention.

[0023] Figure 6(b) is a visualization of the energy consumption and latency corresponding to high-precision rendering and low-precision rendering based on the open-source dataset in this invention.

[0024] Figure 7 This is a functional principle block diagram of a preferred embodiment of the three-dimensional scene reconstruction device in this invention.

[0025] Figure 8 This is a functional principle block diagram of the terminal in this invention. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of this invention clearer and more explicit, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0027] Traditional 3D reconstruction methods primarily utilize implicit scene representation techniques. Scenes generated by this method cannot be independently spatially segmented because content from any viewpoint involves panoramic information, thus failing to fully leverage the advantages of explicit spatial representation. Furthermore, existing systems typically employ relatively simple task allocation strategies, such as random assignment or assigning all tasks to the nodes with the strongest computational power.

[0028] On the one hand, traditional 3D reconstruction methods, due to their use of implicit scene representation, cannot be designed and optimized for resources based on fine-grained explicit spatial segmentation. On the other hand, existing distributed rendering systems lack resource allocation optimization schemes based on explicit 3D representation rendering such as 3DGS (3D Gaussian Splatting). 3DGS is a 3D reconstruction method based on deep learning. Especially in near-field XR systems, existing technologies have failed to fully consider user interactivity, differences in field of view (FoV), and the collaborative optimization of communication and computing resources. Finally, existing technologies lack a systematic solution to handle cross-layer resource allocation problems in 3DGS-driven near-field XR systems, failing to adequately consider rendering content from different perspectives for multiple users and matching heterogeneous resources, thus failing to fully utilize the rendering capabilities of distributed systems.

[0029] This invention presents the first near-field XR system designed and implements distributed rendering based on optimized 3DGS technology. It proposes a novel energy-minimizing framework to reduce system energy consumption as much as possible while ensuring a real-time user experience. The invention constructs a cross-layer optimization problem, comprehensively considering multi-dimensional resources and decisions such as rendering model selection, transport precoding, and rendering task offloading, while ensuring end-to-end latency and rendering quality constraints are met. To solve this mixed-integer non-convex optimization problem, this invention proposes an efficient algorithm based on a block coordinate descent (BCD) framework, which combines integer programming, arithmetic-geometric mean (AM-GM) algorithm, and a closed-form optimal solution strategy. Through this invention, the energy consumption of 3DGS-driven XR systems can be significantly reduced without sacrificing user experience, providing a theoretical solution for resource allocation in distributed 3D reconstruction. Users can design future indoor scene XR systems from a new perspective and enhance the delivery capabilities of 3DGS-based scene reconstruction services by optimizing the system.

[0030] The following description, with reference to the accompanying drawings, illustrates a three-dimensional scene reconstruction method, apparatus, system, terminal, and storage medium according to embodiments of this application. Addressing the problem mentioned in the background art that, due to the involvement of panoramic information from any viewpoint, the advantages of explicit spatial representation cannot be fully utilized to solve the resource optimization challenge of distributed three-dimensional reconstruction, this application provides a three-dimensional scene reconstruction method. In this method, in response to a rendering task execution command, the current field of view of several user terminals is obtained, and the rendering task volume within the current field of view of each user terminal is evaluated. Based on the rendering task volume, with the goal of minimizing system energy consumption, a cross-layer optimization problem based on end-to-end latency and rendering quality constraints is modeled. The cross-layer optimization problem is solved to obtain optimization variables, which include a target rendering model selected from a set of candidate rendering models. Based on the optimization variables, the rendering task is allocated to several distributed nodes, enabling the distributed nodes to collaboratively render and reconstruct the three-dimensional scene. This application constructs a cross-layer optimization problem by considering the differences in the field of view, and comprehensively considers multi-dimensional resources and decisions such as rendering model selection, transport precoding and rendering task offloading, while ensuring that end-to-end latency and rendering quality constraints are met, thereby reducing the energy consumption of rendering tasks and making full use of the advantages of explicit spatial representation, thus solving the resource optimization problem of distributed 3D reconstruction.

[0031] Please see Figure 1 , Figure 1 This is a flowchart of the three-dimensional scene reconstruction method in this invention. For example... Figure 1 As shown, the three-dimensional scene reconstruction method described in this embodiment of the invention includes the following steps: Step S100: Respond to the rendering task execution command, obtain the current field of view of several user terminals, and evaluate the amount of rendering task within the current field of view of each user terminal.

[0032] For example, such as Figure 2As shown, the system of this application includes user terminals, indoor wireless access points equipped with N antennas, and a pre-trained candidate model set. The user terminals can be U single-antenna XR user terminals. The user terminals requiring 3D scene reconstruction form a user terminal set, which may also contain only one user terminal, representing only one user terminal requiring 3D scene reconstruction. Each user terminal communicates with the indoor access point via a near-field wireless channel, using spatial multiplexing for communication scheduling. The indoor access point is an indoor wireless access point equipped with N antennas, comprising one central computing node and A distributed computing nodes. The distributed nodes and the central node can communicate directly, jointly forming a distributed rendering system. The candidate model set can be a 3DGS model library, which includes pre-trained multi-level 3DGS (3D Gaussian Splatting) models. 3DGS models are a point cloud representation technology based on a three-dimensional Gaussian distribution, used for efficient 3D scene reconstruction and real-time rendering; each model has a different number of Gaussian ellipsoids and rendering quality, and these models can be directly accessed by the central node and distributed nodes. This application does not limit specific user types, access point configurations, or the number of distributed nodes.

[0033] This application proposes for the first time a near-field XR system architecture based on 3D Gaussian sputtering (3DGS) technology. It leverages the explicit spatial representation characteristics of 3DGS to achieve fine-grained distributed rendering task allocation. Furthermore, a dynamic 3DGS model selection mechanism based on view angle evaluation adaptively selects the optimal model according to the view angle characteristics of different user terminals and system resource conditions, achieving a balance between rendering quality and system efficiency. This application is not limited to 3DGS technology but is also applicable to other 3D representation technologies.

[0034] like Figure 1 As shown, the three-dimensional scene reconstruction method further includes the following steps: Step S200: Based on the amount of rendering tasks, with the goal of minimizing system energy consumption, model a cross-layer optimization problem based on end-to-end latency and rendering quality constraints.

[0035] First, a mathematical model of the near-field communication and 3DGS distributed rendering system of this application is presented. A near-field multipath channel model is adopted, and it is assumed that there are L scatterers in the signal propagation path. This application defines a set of... For the user's index set, This is the set of indices for the access point antennas. Defined as a set of indexes for distributed nodes. This indicates the position of the nth access point antenna. This indicates the position of the u-th user terminal. This indicates the position of the l-th point scatterer.

[0036] The channel of the u-th user terminal is modeled as: ;in, Represents a complex vector of dimension N; This indicates the direct path channel, determined by the user terminal location information; This represents a non-direct path channel, determined by the transmission path gain. Where: ; ; in, Indicates the gain of the direct path channel at the user end. The gain of the channel through the non-direct path of the scatterer l is represented by... This represents the carrier wavelength used in the communication. Furthermore, j represents the imaginary unit. Specifically, the near-field response vector is represented as: (1) The signal received by the u-th user terminal is: (2) in, This represents the beamforming vector of the user terminal u. This represents the beamforming vector of user terminal i. This represents the signal transmitted from the base station to the user terminal u. This represents the signal transmitted from the base station to user terminal i. The superscript H indicates taking the conjugate transpose of this object. Assume this signal has unit energy, i.e. The superscript * indicates taking the conjugate of the object. This represents noise, which follows a pattern with a mean of 0 and a variance of . The complex Gaussian distribution. This application models the rendering task of each user terminal separately. When executing the rendering task of a certain user terminal u, all user terminals in the user terminal set except user terminal u interfere with user terminal u. Therefore, user terminal i represents all user terminals in the user terminal set except user terminal u.

[0037] The signal transmission rate obtained by the u-th user terminal is: (3) Furthermore, this invention models the performance evaluation metrics for all stages of a distributed real-time 3D reconstruction operation. Specifically, during distributed rendering, the central node first obtains the user's current field of view (FoV), which is the spatial coordinate corresponding to the field of view. It then evaluates the rendering workload (i.e., rendering task volume) within each user's FoV, which is determined by the number of Gaussian ellipsoids within the FoV. This indicates that the central node is derived from the candidate model set. Select the optimal 3DGS model corresponding to the rendering task, and then assign the task to the distributed nodes. The rendering latency of the user-side u is: (4) in, and These are the task unloading decision variables (i.e., task allocation variables) for user terminal u and user terminal i, respectively. This represents the computational efficiency of distributed node a. This represents the computing resources of distributed node a. This represents the data size corresponding to each Gaussian ellipsoid (these data typically include location, covariance, spherical harmonics, transparency, etc.). This represents the amount of rendering tasks on user i. Specifically, This indicates that the rendering task for the u-th client is delegated to the a-th node. Conversely. The meanings of the values ​​are similar.

[0038] In this embodiment of the application, step S200 specifically includes: Step S210: Determine the rendering energy consumption and transmission energy consumption corresponding to several distributed nodes based on the rendering task volume, and obtain the system energy consumption based on the rendering energy consumption and transmission energy consumption. Step S220: Determine the rendering task allocation constraints, transmission power constraints, rendering quality constraints, and real-time rendering latency constraints. Step S230: With the goal of minimizing system energy consumption, and with constraints such as rendering task allocation constraints, transmission power constraints, rendering quality constraints, and real-time rendering latency constraints, model a cross-layer optimization problem based on end-to-end latency and rendering quality constraints.

[0039] This invention constructs a cross-layer optimization framework that comprehensively considers model selection and the allocation of communication and computing resources, minimizing system energy consumption while meeting end-to-end latency and rendering quality constraints.

[0040] In one embodiment of this application, step S210, "determining the rendering energy consumption and transmission energy consumption corresponding to several distributed nodes based on the rendering task volume," specifically includes: The rendering energy consumption of the distributed nodes is obtained by acquiring and using the effective switched capacitors, computational efficiency, computational resources, rendering task volume on the user side, data size corresponding to each Gaussian ellipsoid in the rendering task volume, and task allocation variables on the user side. The downlink transmission delay of the user terminal is obtained by acquiring and based on the data size of the image stream when the user terminal transmits the completed rendering task, the channel bandwidth, the transmission rate, the factor of the data size of adjacent Gaussian ellipsoids, the amount of rendering task of the user terminal, and the data size corresponding to each Gaussian ellipsoid. Obtain the beamforming vector at the user end, and obtain the transmission power consumption corresponding to the distributed node based on the downlink transmission delay and the beamforming vector.

[0041] Specifically, the rendering energy consumption of distributed nodes is as follows: ; in, The effective switched capacitors are determined by the hardware of the distributed nodes. This refers to the amount of rendering tasks on the user-side u.

[0042] Once the rendering task is complete, the indoor access point will transmit these rendered 3D images to the user. This invention employs a hybrid transmission mode to balance transmission efficiency and interactivity; therefore, the downlink transmission latency experienced by the u-th user terminal is: (5) Where d represents the data size of the image stream, It is a factor that takes into account the data size of adjacent Gaussian ellipsoids. It is the channel bandwidth of the u-th user terminal. It is the transmission rate of the u-th user terminal.

[0043] Transmission energy consumption is: .

[0044] In this embodiment of the application, step S220 specifically includes: Obtain and obtain rendering task allocation constraints based on the user-side task allocation variables; Obtain a preset transmission power threshold and the beamforming vector of the user terminal, and obtain a transmission power constraint based on the transmission power threshold and the beamforming vector of the user terminal; Obtain the voxel size and preset minimum point cloud density corresponding to the current field of view of the user terminal, and obtain the rendering quality constraints based on the rendering task volume, voxel size and preset minimum point cloud density; Obtain the rendering latency at the user end, and derive the latency constraint for real-time rendering based on the rendering latency at the user end and the downlink transmission latency.

[0045] Specifically, this application formulates the system optimization problem as minimizing the total system energy consumption while satisfying end-to-end latency and rendering quality constraints, i.e.:

[0046] In the aforementioned cross-layer optimization problem, constraints (6) and (7) are constraints on rendering task allocation; constraint (8) defines the upper limit of transmission power; constraints (9) and (10) control the rendering quality; and constraint (11) is a latency constraint for real-time rendering to ensure the smoothness of the XR experience. This indicates the preset transmission power threshold. This represents the latency requirement for real-time rendering by the u-th user. This indicates the voxel size corresponding to the user's FoV. Minimum point cloud density to ensure rendering quality.

[0047] The optimization model corresponding to the cross-layer optimization problem in this application is not limited to a specific energy consumption model or channel model. It only needs to satisfy the optimization objective of minimizing system energy consumption and the constraints of latency and rendering quality requirements.

[0048] like Figure 1 As shown, the three-dimensional scene reconstruction method further includes the following steps: Step S300: Solve the cross-layer optimization problem to obtain optimization variables, including the target rendering model selected from the candidate rendering model set.

[0049] In this embodiment of the application, step S300 specifically includes: Step S310: Decompose the cross-layer optimization problem into several sub-problems, including: updating task allocation variables, updating transmission beamforming vectors, and model selection; Step S320: Solve each sub-problem to update the task allocation variables, the signal received by the user terminal, the beamforming vector, and the target rendering model; Step S330: Continuously iterate and update the task allocation variables, the signals received by the user terminal, the beamforming vector, and the target rendering model until the stopping condition is met and then stop. Step S340: Use the task allocation variables obtained after stopping the update, the signals received by the user terminal, the beamforming vector, and the target rendering model as optimization variables for the distributed nodes.

[0050] Specifically, to address the non-convex mixed integer problem of cross-layer optimization, this invention proposes an algorithm based on the Block Coordinate Descent (BCD) framework, which updates each variable sequentially. That is, based on the divisibility of the original problem P, this invention decomposes it into multiple smaller, more easily solvable sub-problem modules for sequential solving, and stops the overall algorithm after satisfying a preset cutoff condition.

[0051] In this embodiment of the application, step S320 specifically includes: Step S321: Convert the subproblem of updating the task allocation variable into a standard integer linear programming problem, and use an integer programming solver to obtain the optimal solution corresponding to the subproblem of updating the task allocation variable, so as to update the task allocation variable; Step S322: The subproblem of updating the transmission beamforming vector is transformed into an unequal transformation subproblem using AM-GM unequal transformation, the unequal transformation subproblem is solved to obtain a closed-form solution, and the signal received by the user terminal is updated based on the closed-form solution; Step S323: Convert the unequal transformation subproblem into an approximate convex problem, solve the approximate convex problem to obtain the stationary point solution, update the beamforming vector based on the closed solution, and update the beamforming vector based on the stationary point solution. Step S324: Solve the closed-form optimal solution of the subproblem of model selection, and update the target rendering model based on the closed-form optimal solution.

[0052] For example, the subproblem of updating task assignment variables in this application is used to update task assignment variables. Specifically, if other variables remain fixed, the subproblem of updating the task allocation variable is:

[0053] To optimize this subproblem, this invention transforms the quadratic constraint (12) into a linear constraint using the Big M method, i.e.

[0054] This transforms problem P1 into a standard integer linear programming problem, and the optimal solution to P1 can be obtained using an integer programming solver, such as Gurobi (an optimization problem solver).

[0055] For example, the subproblem of updating the transmit beamforming vector in this application is used to update the transmit beamforming vector. Specifically, with other variables fixed, the transmission beamforming subproblem is transformed into a convex optimization problem through serial updates and an arithmetic-geometric-mean (AM-GM) inequality transformation, establishing a concave lower bound. AM-GM is an algorithm applied to minimizing fractional programming problems. The final solution is obtained using the CVX tool. Specifically, the subproblem is:

[0056] Furthermore, subproblem P2 can be solved by updating only one at a time. And maintain other beamforming vectors Optimize by keeping it unchanged; specifically, for vectors... The optimization problem is given by P2-u below:

[0057] Furthermore, this invention employs the AM-GM inequality transformation to convert P2-u into a more easily processed P2-u. ,Right now:

[0058] Furthermore, in the above P2-u In this invention, all are updated sequentially. and auxiliary variables This achieves an equivalent optimization of P2-u. Among them, The update is based on the closed-form solution: ; Furthermore, the present invention establishes P2-u objective function f( and functions in constraints ( The concave lower bound of ) thus in an approximate P2-u Updating variables in convex optimization problems Specifically, the function f( )and ( The lower bound of ) is:

[0059] in: ; .

[0060] In particular, This indicates that the variable is updated during the nth iteration. The value of f( ) represents the corresponding function f( The lower bound of ) ( ) represents the corresponding function ( The lower bound of ). This indicates that the gradient of the object is being retrieved. This indicates retrieving the Hessian matrix of the object, and the superscript T indicates retrieving the transpose of the object. This indicates taking the F-norm of the object.

[0061] Furthermore, the present invention substitutes the above equations (16) and (17) into problem P2-u Thus, we obtain the final approximate convex problem, namely: ; Since the problem is convex, it can be efficiently solved using CVX (Convex Optimization Toolbox, a solver for convex optimization problems), thus providing the subproblem P2 with respect to the variables. The solution to the stationing point.

[0062] For example, the model selection sub-problem in this application is used to update variables. Specifically, with other variables fixed, the model selection subproblem is:

[0063] This subproblem has a closed-form optimal solution, that is .

[0064] The overall algorithm of this invention iteratively updates the three modules mentioned above until the relative change in the objective function is less than a preset threshold, at which point the algorithm converges. This algorithm guarantees that the objective function value monotonically decreases and converges to a stable value.

[0065] The overall algorithm for solving the system optimization problem in this application is as follows: Figure 3 As shown, the specific steps are summarized below: Step 1: Initialize all optimization variables.

[0066] Step 2: Execute the outer loop.

[0067] Step 3: Update P1 by solving it using an integer programming solver. .

[0068] Step 4: Update based on the closed-form solution provided in this invention And solve P2-u using CVX appr Update Repeat step 4; the inner loop stops when the stopping condition is met.

[0069] Step 5: Update the closed-form solution according to the present invention. .

[0070] Step 6: Repeat steps 2 to 5 until the outer loop meets the stopping condition.

[0071] This application proposes an efficient optimization algorithm based on block coordinate descent (BCD). Through integer programming, arithmetic-geometric mean transformation and closed-form optimal solution strategy, it effectively solves the non-convex mixed integer optimization problem and solves the resource optimization problem of distributed 3D reconstruction system.

[0072] like Figure 1 As shown, the three-dimensional scene reconstruction method further includes the following steps: Step S400: Based on the optimization variables, the rendering task is allocated to several distributed nodes so that the several distributed nodes can perform collaborative rendering to reconstruct the 3D scene.

[0073] The performance of the algorithm proposed in this invention has been verified through multiple sets of experiments, and the results are as follows: Figure 4 , Figure 5 As shown in Figures 6(a) and 6(b).

[0074] Figure 4 The convergence performance of the algorithm under different settings is demonstrated. The results show that regardless of the frequency of user interaction or the differences in the computing power of distributed nodes (…), the algorithm can achieve convergence. The proposed algorithm can converge quickly within a few iterations (representing the average computing power of nodes), which proves the effectiveness and stability of the algorithm.

[0075] Figure 5 The performance of different rendering schemes was compared. Compared with energy-saving rendering, greedy rendering, random rendering, and fixed-model rendering, the scheme proposed in this invention achieves the lowest system power consumption while ensuring a high success rate of real-time rendering. The advantages of this scheme become even more apparent when the number of users increases.

[0076] Figure 6(a) shows the visualization results of real-world scenes, high-precision rendering, and low-precision rendering based on the Tank-and-Temple dataset (an open-source dataset for 3D reconstruction tasks); Figure 6(b) shows the visualization results of energy consumption and latency corresponding to high-precision and low-precision rendering based on the Tank-and-Temple dataset. This demonstrates that selecting an appropriate 3DGS model based on user FoV and system resources can reduce the rendering precision of background areas while preserving key object details, thereby achieving efficient real-time rendering in resource-constrained XR systems.

[0077] This application provides a distributed rendering system architecture combining 3DGS technology and near-field communication, including a central node field-of-view evaluation, distributed node task allocation, and a collaborative rendering mechanism. It also models a cross-layer optimization problem with the goal of minimizing system energy consumption, while considering end-to-end latency and rendering quality constraints. Furthermore, this application provides a system optimization algorithm based on the BCD framework, including an integer programming solution for the task allocation subproblem, an AM-GM transform and convex approximation method for the transmission beamforming subproblem, a closed-form optimal solution derivation method for the model selection subproblem, and a rendering quality evaluation and assurance mechanism based on Gaussian ellipsoid density control. This mechanism achieves fine-grained rendering quality control by adjusting the Gaussian ellipsoid density within the field of view.

[0078] Furthermore, such as Figure 7As shown, based on the above-described three-dimensional scene reconstruction method, the present invention also provides a three-dimensional scene reconstruction device, comprising: The response module 100 is used to respond to rendering task execution instructions, obtain the current field of view of several user terminals, and evaluate the amount of rendering task within the current field of view of each user terminal. Modeling module 200 is used to model a cross-layer optimization problem based on end-to-end latency and rendering quality constraints, with the goal of minimizing system energy consumption, according to the rendering task volume. The solver module 300 is used to solve the cross-layer optimization problem and obtain optimization variables, including the target rendering model selected from the candidate rendering model set. The reconstruction module 400 is used to allocate the rendering task to several distributed nodes based on the optimization variables, so that the several distributed nodes can perform collaborative rendering to reconstruct the 3D scene.

[0079] It should be noted that the foregoing explanation of the three-dimensional scene reconstruction method embodiment also applies to the three-dimensional scene reconstruction device of this embodiment, and will not be repeated here.

[0080] This invention discloses a 3D scene reconstruction device. By responding to rendering task execution commands, it acquires the current field of view of several user terminals and evaluates the rendering task volume within the current field of view of each user terminal. Based on the rendering task volume, and with the goal of minimizing system energy consumption, it models a cross-layer optimization problem based on end-to-end latency and rendering quality constraints. The cross-layer optimization problem is solved to obtain optimization variables, which include a target rendering model selected from a set of candidate rendering models. Based on the optimization variables, the rendering task is allocated to several distributed nodes, enabling these nodes to collaboratively render and reconstruct the 3D scene. This application considers the differences in field of view, constructs a cross-layer optimization problem, comprehensively considers multi-dimensional resources and decisions such as rendering model selection, transmission precoding, and rendering task unloading, and ensures that end-to-end latency and rendering quality constraints are met. This reduces the energy consumption of the rendering task, fully utilizes the advantages of explicit spatial representation, and solves the resource optimization problem of distributed 3D reconstruction.

[0081] Furthermore, based on the above-mentioned three-dimensional scene reconstruction method, the present invention also provides a three-dimensional scene reconstruction system, including: several user terminals, a central node, and distributed nodes; The user terminal is used for near-field wireless communication with the central node and distributed nodes, and for sending rendering tasks to the central node. The central node is used to respond to rendering task execution instructions, obtain the current field of view of several user terminals, and evaluate the rendering task volume within the current field of view of each user terminal; based on the rendering task volume, with the goal of minimizing system energy consumption, a cross-layer optimization problem based on end-to-end latency and rendering quality constraints is modeled; the cross-layer optimization problem is solved to obtain optimization variables, which include the target rendering model selected from the candidate rendering model set; based on the optimization variables, the rendering task is allocated to several distributed nodes; Several distributed nodes are used for collaborative rendering to reconstruct the 3D scene.

[0082] Figure 8 A schematic diagram of the structure of a terminal provided in an embodiment of this application. The terminal may include: The memory 501, the processor 502, and the computer program stored on the memory 501 and capable of running on the processor 502.

[0083] When the processor 502 executes the program, it implements the three-dimensional scene reconstruction method provided in the above embodiments.

[0084] Furthermore, the terminal also includes: Communication interface 503 is used for communication between memory 501 and processor 502.

[0085] The memory 501 is used to store computer programs that can run on the processor 502.

[0086] The memory 501 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.

[0087] If the memory 501, processor 502, and communication interface 503 are implemented independently, they can be interconnected via a bus to communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, only one line is used in the diagram, but this does not imply that there is only one bus or one type of bus.

[0088] Optionally, in a specific implementation, if the memory 501, processor 502, and communication interface 503 are integrated on a single chip, then the memory 501, processor 502, and communication interface 503 can communicate with each other through an internal interface.

[0089] Processor 502 may be a central processing unit (CPU), an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of this application.

[0090] This embodiment also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described three-dimensional scene reconstruction method.

[0091] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0092] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "N" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0093] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or N executable instructions for implementing custom logic functions or processes, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as should be understood by those skilled in the art to which embodiments of this application pertain.

[0094] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can read and execute instructions from or in conjunction with such an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by or in conjunction with an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). In addition, computer-readable media can even be paper or other suitable media on which programs can be printed, because programs can be obtained electronically by optically scanning paper or other media, then editing, interpreting or otherwise processing them as necessary, and then storing them in computer memory.

[0095] It should be understood that the various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, the N steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. If implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0096] Those skilled in the art will understand that all or part of the steps of the methods described in the above embodiments can be implemented by a program instructing related hardware, and the program can be stored in a computer-readable storage medium. When executed, the program includes one or a combination of the steps of the method embodiments.

[0097] Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0098] The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc. Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of this application.

Claims

1. A method for reconstructing a three-dimensional scene, characterized in that, include: Respond to rendering task execution instructions, obtain the current field of view of several user terminals, and evaluate the amount of rendering task within the current field of view of each user terminal; Based on the rendering workload, with the goal of minimizing system energy consumption, a cross-layer optimization problem based on end-to-end latency and rendering quality constraints is modeled. Solving the cross-layer optimization problem yields optimization variables, which include the target rendering model selected from the candidate rendering model set. Based on the optimization variables, the rendering task is distributed to several distributed nodes so that the distributed nodes can perform collaborative rendering to reconstruct the 3D scene.

2. The three-dimensional scene reconstruction method according to claim 1, characterized in that, Based on the aforementioned rendering workload, and with the goal of minimizing system energy consumption, a cross-layer optimization problem is modeled based on end-to-end latency and rendering quality constraints, including: Based on the amount of rendering tasks, determine the rendering energy consumption and transmission energy consumption corresponding to several distributed nodes, and obtain the system energy consumption based on the rendering energy consumption and transmission energy consumption. Determine the constraints for rendering task allocation, transmission power, rendering quality, and latency of real-time rendering; With the goal of minimizing system energy consumption, and with constraints such as rendering task allocation, transmission power, rendering quality, and real-time rendering latency, a cross-layer optimization problem based on end-to-end latency and rendering quality constraints is modeled.

3. The three-dimensional scene reconstruction method according to claim 2, characterized in that, Based on the rendering task volume, determine the rendering energy consumption and transmission energy consumption corresponding to several distributed nodes, including: The rendering energy consumption of the distributed nodes is obtained by acquiring and using the effective switched capacitors, computational efficiency, computational resources, rendering task volume on the user side, data size corresponding to each Gaussian ellipsoid in the rendering task volume, and task allocation variables on the user side. The downlink transmission delay of the user terminal is obtained by acquiring and based on the data size of the image stream when the user terminal transmits the completed rendering task, the channel bandwidth, the transmission rate, the factor of the data size of adjacent Gaussian ellipsoids, the amount of rendering task of the user terminal, and the data size corresponding to each Gaussian ellipsoid. Obtain the beamforming vector at the user end, and obtain the transmission power consumption corresponding to the distributed node based on the downlink transmission delay and the beamforming vector.

4. The three-dimensional scene reconstruction method according to claim 3, characterized in that, Define rendering task allocation constraints, transmission power constraints, rendering quality constraints, and latency constraints for real-time rendering, including: Obtain and obtain rendering task allocation constraints based on the user-side task allocation variables; Obtain a preset transmission power threshold and the beamforming vector of the user terminal, and obtain a transmission power constraint based on the transmission power threshold and the beamforming vector of the user terminal; Obtain the voxel size and preset minimum point cloud density corresponding to the current field of view of the user terminal, and obtain the rendering quality constraints based on the rendering task volume, voxel size and preset minimum point cloud density; Obtain the rendering latency at the user end, and derive the latency constraint for real-time rendering based on the rendering latency at the user end and the downlink transmission latency.

5. The three-dimensional scene reconstruction method according to claim 4, characterized in that, Solving the aforementioned cross-level optimization problem yields the optimization variables, including: The cross-layer optimization problem is decomposed into several sub-problems, including: updating task allocation variables, updating transmission beamforming vectors, and model selection. Solve the subproblems to update the task assignment variables, the signals received by the user, the beamforming vectors, and the target rendering model; The task allocation variables, signals received by the user, beamforming vectors, and target rendering models are continuously updated iteratively until the stopping condition is met. The task allocation variables obtained after stopping updates, the signals received by the user, the beamforming vector, and the target rendering model are used as optimization variables for distributed nodes.

6. The three-dimensional scene reconstruction method according to claim 5, characterized in that, Solving the subproblems to update the task assignment variables, the signals received by the user, the beamforming vectors, and the target rendering model includes: The subproblem of updating the task allocation variable is transformed into a standard integer linear programming problem. An integer programming solver is used to obtain the optimal solution corresponding to the subproblem of updating the task allocation variable, so as to update the task allocation variable. The subproblem of updating the transmission beamforming vector is transformed into an unequal transformation subproblem using AM-GM unequal transformation. The unequal transformation subproblem is solved to obtain a closed-form solution. The signal received by the user terminal is updated based on the closed-form solution. The unequal transformation subproblem is transformed into an approximate convex problem. The approximate convex problem is solved to obtain the stationary point solution. The beamforming vector is updated based on the closed solution and the stationary point solution. Solve the closed-form optimal solution to the subproblem of model selection, and update the target rendering model based on the closed-form optimal solution.

7. A three-dimensional scene reconstruction device, characterized in that, include: The response module is used to respond to rendering task execution instructions, obtain the current field of view of several user terminals, and evaluate the amount of rendering task within the current field of view of each user terminal. The modeling module is used to model a cross-layer optimization problem based on end-to-end latency and rendering quality constraints, with the goal of minimizing system energy consumption, according to the rendering task volume. The solution module is used to solve the cross-layer optimization problem and obtain optimization variables, including the target rendering model selected from the candidate rendering model set. The reconstruction module is used to allocate the rendering task to several distributed nodes based on the optimization variables, so that the several distributed nodes can perform collaborative rendering to reconstruct the 3D scene.

8. A three-dimensional scene reconstruction system, characterized in that, include: Several user terminals, a central node, and distributed nodes; The user terminal is used for near-field wireless communication with the central node and distributed nodes, and for sending rendering tasks to the central node. The central node is used to respond to rendering task execution instructions, obtain the current field of view of several user terminals, and evaluate the amount of rendering tasks within the current field of view of each user terminal; based on the amount of rendering tasks, with the goal of minimizing system energy consumption, a cross-layer optimization problem based on end-to-end latency and rendering quality constraints is modeled. Solving the cross-layer optimization problem yields optimization variables, which include the target rendering model selected from the candidate rendering model set. Based on the optimization variables, the rendering task is distributed to several distributed nodes; Several distributed nodes are used for collaborative rendering to reconstruct the 3D scene.

9. A terminal, characterized in that, include: The device includes a memory, a processor, and a 3D scene reconstruction program stored in the memory and executable on the processor, wherein the 3D scene reconstruction program, when executed by the processor, implements the steps of the 3D scene reconstruction method as described in any one of claims 1 to 6.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that can be executed to implement the steps of the three-dimensional scene reconstruction method as described in any one of claims 1 to 6.

Images