Three-dimensional visual synthesis system based on heterogeneous collaboration and screen alignment fusion

Abstract

The invention relates to a three-dimensional visual synthesis system based on heterogeneous collaboration and screen alignment fusion, which is characterized in that a master control system generates complete viewpoint configuration information of a current scene acquired by an image acquisition device, and the complete viewpoint configuration information is used as a single data source and is simultaneously distributed to a graphic processing unit and a neural rendering coprocessor of the master control system; synchronous starting and parallel execution of the two heterogeneous rendering pipelines are achieved. And in combination with a subsequent screen alignment fusion mechanism, an end-to-end collaboration framework with low delay, low coupling and high coordination is constructed, and the key problem of real-time integration of high-fidelity contents is effectively solved. On the premise of not depending on a traditional deep buffer area or geometric alignment, high-fidelity visual content generated by special hardware acceleration and a dynamic environment of a main control system are efficiently fused, and complex effects such as lightweight asynchronous communication and real-time shadow are supported. Therefore, multiple bottlenecks in the aspects of real-time performance, compatibility and engineering deployment in the prior art are broken through.

Description

Technical Field

[0001] This invention relates to the fields of computer graphics, human-computer interaction and real-time 3D visual synthesis technology, and particularly to a real-time high-fidelity 3D visual synthesis system based on heterogeneous collaboration and screen alignment fusion. Background Technology

[0002] With the rapid development of deep learning-based 3D content generation technology, implicit representation methods such as Neural Radiance Fields (NeRF) have been able to reconstruct highly realistic 3D scenes from a small number of 2D images [1]. However, the high fidelity of such methods depends on performing hundreds of multilayer perceptron (MLP) queries on each pixel, resulting in huge computational overhead and making it difficult to achieve real-time rendering on consumer hardware or edge devices, which seriously restricts their deployment in practical applications [1].

[0003] To improve efficiency, researchers have proposed a variety of software-level optimization schemes, such as using multi-resolution hash coding to accelerate NeRF inference[2], or storing scene representations through explicit structures such as sparse voxel octrees (SVO)[3]. However, these methods usually still require complex training or fine-tuning on GPUs, introducing significant preprocessing overhead[3]. Another type of research attempts to extract implicit NeRF models back into explicit geometry (such as meshes) to be compatible with existing graphics pipelines, but this process often leads to a decrease in model fidelity or requires high retraining costs[4]. More advanced runtime synthesis methods can achieve complex interactions between neural volumes and meshes, but usually require deep modifications to the underlying rendering pipeline of standard graphics engines, which is not only complex to develop but also has a very high computational load, making it difficult to popularize on general-purpose platforms[5]–[7].

[0004] Existing technologies generally rely on complex processes such as precise depth information exchange, coordinate system calibration, or geometric reprojection when integrating high-performance neural rendering results with dynamic environments. These methods not only increase system coupling and communication latency, but also greatly limit their flexibility and deployment efficiency on heterogeneous computing platforms. Although dedicated hardware accelerators have been shown to effectively alleviate computational bottlenecks [8], how to achieve seamless collaboration between them and general computing platforms with low overhead and without the need for depth information interaction remains a key technical problem that needs to be solved.

[0005] The existing technology has the following core problems: Real-time bottleneck: Although the deep learning-based 3D content generation method can provide photorealistic feel, its underlying rendering process involves massive parallel computing, which requires extremely high computing power. It is difficult to achieve real-time rendering on consumer-grade hardware or edge devices, which seriously restricts its practicality in interactive applications such as augmented reality / virtual reality. Complex engineering deployment: When high-quality visual content generated by dedicated hardware accelerators needs to be fused with a dynamic 3D environment managed by the main control system, traditional solutions generally rely on precise geometric alignment and complex exchange of depth and color information. These operations not only increase the coupling and communication latency of the system, but also make the overall architecture complex, difficult to maintain and deploy; Difficulties in heterogeneous system collaboration: Dedicated hardware accelerators and general-purpose graphics processing units (GPUs) typically run in separate pipelines, lacking efficient collaboration mechanisms to handle complex effects across systems, such as real-time shadow simulation, which limits the realism and integrity of the final composite image.

[0006] [1] B. Mildenhall, PP Srinivasan, M. Tancik, JT Barron, R.Ramamoorthi, and R. Ng, “NeRF: Rrepresenting scenes as neural radiance fields for view synthesis,” Commun. ACM, vol. 65, p. 99–106, Dec. 2021.

[0007] [2] T. Mu¨ller, A. Evans, C. Schied, and A. Keller, “Instant neuralgraphics primitives with a multiresolution hash encoding,” ACM Trans. Graph., vol. 41, pp. 1–15, July 2022.

[0008] [3] A. Yu, R. Li, M. Tancik, H. Li, R. Ng, and A. Kanazawa, “PlenOctrees for real-time rendering of neural radiance fields,” in ICCV, 2021.

[0009] [4] J. Gu, M. Jiang, H. Li, X. Lu, G. Zhu, S. A. A. Shah, L. Zhang,and M. Bennamoun, “UE4-NeRF: Neural radiance field for real-time render-ingof large-scale scene,” Advances in Neural Inf ormation Processing Systems,vol. 36, pp. 59124–59136, 2023。

[0010] [5] H. Turki, V. Agrawal, S. R. Bulo´, L. Porzi, P. Kontschieder, D.Ramanan, M. Zollho¨fer, and C. Richardt, “HybridNeRF: Efficient neuralrendering via adaptive volumetric surfaces,” in 2024 IEEE / CVF Conf erence onComputer Vision and Pattern Recognition(CVPR), pp. 19647–19656, 2024。

[0011] [6] Y.-L. Qiao, A. Gao, Y. Xu, Y. Feng, J.-B. Huang, and M. C. Lin,“Dynamic mesh-aware radiance fields,” in 2023 IEEE / CVF International Conference on Computer Vision(ICCV), pp. 385–396, 2023。

[0012] [7] S. Bi, Z. Xu, P. Srinivasan, B. Mildenhall, K. Sunkavalli, M. Haˇsan, Y. Hold-Geoffroy, D. Kriegman, and R. Ramamoorthi, “Neural reflectancefields for appearance acquisition,” arXiv preprint arXiv:2008.03824, 2020.

[0013] [8] B. Yuan, 2025. Summary of the Invention

[0014] To address the performance bottlenecks, high system coupling, and complex engineering implementation issues faced by complex, computationally intensive intelligent 3D content generation models when deployed in real-time interactive systems, a 3D visual synthesis system based on heterogeneous collaboration and screen alignment fusion is proposed.

[0015] The technical solution of this invention is as follows: a three-dimensional visual synthesis system based on heterogeneous collaboration and screen alignment fusion, including an FPGA and a main control system. A neural rendering coprocessor is set in the FPGA. A lightweight multi-threaded server distributes data received from the main control system and integrates the processing results. The main control system acquires images from an image acquisition device with which it communicates in real time. At the beginning of processing each frame of image, the main control system first generates complete viewpoint configuration information of the current scene acquired by the image acquisition device. The complete viewpoint configuration information is distributed as a single data source to the graphics processing unit and the neural rendering coprocessor of the main control system. The graphics processing unit of the main control system performs rasterization rendering of the dynamic three-dimensional scene in its standard rendering pipeline according to the received viewpoint configuration to generate foreground and background images. At the same time, the neural rendering coprocessor starts its hardware-accelerated ray traversal and neural network inference process according to the same viewpoint configuration to generate corresponding high-fidelity RGB color and alpha channel data. The result data is sent back to the main control system and seamlessly merged with the foreground content generated by the graphics processing unit to form the final image. The entire process does not require any depth information exchange or geometric calibration.

[0016] Preferably, in the main control system, the main control program calculates the camera pose transformation relative to the object origin based on the transformation of the object and camera in the world coordinate system in the dynamic scene of the main control program, and uses it as viewpoint configuration information distributed to the neural rendering coprocessor, ensuring that the two rendering pipelines of the graphics processing unit and the neural rendering coprocessor have completely consistent imaging parameters from the beginning.

[0017] Preferably, the graphics processing unit of the main control system uses the same viewpoint parameters generated by the main control program to calculate the precise sampling coordinates of the object results rendered by the neural rendering coprocessor in the screen space, and directly aligns them naturally in the screen space for image fusion.

[0018] Preferably, the shadow projection module in the main control system directly reuses the data structure describing the occupancy of three-dimensional space inside the neural rendering coprocessor. When the main control system performs lighting simulation based on the pose and field of view in the complete viewpoint configuration information, the shadow projection module queries the data structure along the direction of the incident light to determine whether there is a volume on the path, thereby calculating the cumulative transmittance and realizing real-time soft shadow approximation.

[0019] A working method for a 3D visual compositing system based on heterogeneous collaboration and screen alignment fusion is disclosed. In this system, the main control program transmits complete viewpoint data to a relay module at the start of rendering. The relay module calculates the camera's transformation relative to the object's origin based on this information and the pose of the target object in the scene coordinate system using the neural rendering coprocessor (NRF). This information is then transmitted via TCP to the NRF control program as the camera extrinsic data. Upon receiving the viewpoint configuration, the control program initiates rendering of objects from the same viewpoint. Simultaneously, the graphics processing unit (GPU) and the NRF renderer render their required data. The GPU uses the rendering result from the previous frame to compose the current frame. After the NRF renderer completes rendering the current frame, the data is synthesized into bitmap data by the coprocessor control program and transmitted back to the relay module via TCP. The relay module replaces the material data in the GPU with the newly transmitted bitmap data for image compositing in the next frame.

[0020] Furthermore, the neural rendering coprocessor uses an EDR-NR accelerator. After receiving camera parameters, the EDR-NR accelerator divides the output image into four quadrant regions with the center of the screen as the origin. These quadrant regions are then processed in parallel by four independent computing cores. Each core performs ray traversal and neural network inference based on a shared optical center and a field of view aligned with the long side of the display device. This generates RGBA color and Alpha channel data for the corresponding region and writes it into the DDR memory on the FPGA board. Simultaneously, the graphics processing unit of the main control system begins rendering the foreground 3D image it is responsible for. After the EDR-NR accelerator completes rendering a frame, the multi-threaded pipeline of the multi-threaded server asynchronously reads the rendering result from the DDR memory and fills it into a pixel image. Subsequently, the complete RGBA frame is sent back to the main control program as texture data via a TCP connection.

[0021] The beneficial effects of this invention are as follows: This invention is a three-dimensional visual synthesis system based on heterogeneous collaboration and screen alignment fusion. It can achieve seamless integration of high-fidelity content generated by dedicated hardware with the dynamic environment of the main control system in a low-overhead and low-latency manner, and supports efficient asynchronous communication and lighting effect simulation, thereby breaking through the multiple bottlenecks in performance, compatibility and engineering deployment mentioned above. Attached Figure Description

[0022] Figure 1 This is a structural diagram of the three-dimensional vision synthesis system of the present invention; Figure 2 It is a method for constructing data structures that describe the occupancy of three-dimensional space within a neural rendering coprocessor; Figure 3This is a schematic diagram of the rendering result fusion process proposed in this invention; Figure 4 This is a diagram of the hybrid rendering and compositing system according to an embodiment of the present invention; Figure 5 These are composite and soft shadow simulation effect diagrams from embodiments of the present invention. Detailed Implementation

[0023] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. These embodiments are based on the technical solution of the present invention and provide detailed implementation methods and specific operating procedures. However, the scope of protection of the present invention is not limited to the following embodiments.

[0024] This invention designs a novel system-level architecture that achieves synchronous startup and parallel execution of two heterogeneous rendering pipelines by uniformly generating and distributing viewpoint parameters through the main control program. Combined with a subsequent screen alignment and fusion mechanism, it constructs a low-latency, loosely coupled, and highly coordinated end-to-end collaborative framework, effectively solving the key challenge of real-time integration of high-fidelity content. It can efficiently fuse high-fidelity visual content generated by dedicated hardware acceleration with the dynamic environment of the main control system without relying on traditional depth buffers or geometric alignment, and supports lightweight asynchronous communication and complex effects such as real-time shadows, thereby overcoming multiple bottlenecks in current technology regarding real-time performance, compatibility, and engineering deployment.

[0025] The following steps outline how to construct a heterogeneous collaborative rendering process that is uniformly scheduled by the main control program: 1. Unified parameter distribution and rendering: such as Figure 1The diagram shows the structure of a 3D vision synthesis system. The main control system acquires images from image acquisition devices it communicates with in real time. At the beginning of each frame, the main control program of the main control system (Host RenderPC) first generates complete viewpoint configuration information of the current scene acquired by the image acquisition devices, including the position, pose, and field of view of the virtual camera. This complete viewpoint configuration information, as a single data source, is simultaneously distributed to two independent processing units: the graphics processing unit (GPU) of the main control system and a dedicated neural rendering coprocessor (such as the FPGA-based EDR-NR on FPGA used in this embodiment). This coprocessor is designed for object rendering, and the rendering result only contains the corresponding transparent background image data. During distribution, the main control program calculates the camera pose transformation relative to the object origin based on the transformation of the object and camera in the world coordinate system of the dynamic scene in the main control program, and uses this as the viewpoint configuration information distributed to the coprocessor system. This design ensures that the two rendering pipelines have completely consistent imaging parameters from the beginning. At this time, the GPU of the main control system performs rasterization rendering of the dynamic 3D scene in its standard rendering pipeline according to the received viewpoint configuration, generating foreground and background images. At the same time, the neural rendering coprocessor initiates its hardware-accelerated ray traversal and neural network inference processes based on the same viewpoint configuration, generating corresponding high-fidelity RGB color and alpha channel data.

[0026] 2. Screen-aligned, depthless blending: During post-compositing, the GPU of the main control system uses the same viewpoint parameters generated by the main control program to calculate the precise sampling coordinates of the objects rendered by the dedicated hardware in screen space. Since both share the optical center and field of view, they are naturally aligned directly within screen space, resulting in image blending. For example... Figure 2 As shown, when sampling pixel rays within screen space and performing depth test box color mixing, the results generated by the neural rendering coprocessor can participate in this process as a sheet-like element. Therefore, the GPU can directly obtain RGB colors and alpha values ​​from dedicated hardware and seamlessly blend them with its own rendered foreground content through alpha blending operations to form the final image, without any depth information exchange or geometric calibration.

[0027] 3. Cross-system lighting simulation: For complex effects such as shadows, this invention proposes a method that does not rely on the main control system's lighting pipeline. The shadow casting module in the main control system directly reuses the data structure describing the occupancy of 3D space internally within the dedicated neural rendering coprocessor. This data structure is as follows: Figure 2As shown, for a finest-grained voxel mesh, the mesh is divided into 8 blocks along the axis of symmetry. Each block corresponds to one bit, used to indicate whether an object exists at that location. These 8 blocks are divided into two parts: the first half corresponds to the lower 4 bits, and the second half corresponds to the higher 4 bits. Within each part, they are arranged in the order of lower left, lower right, upper left, and upper right. When the main control system performs lighting simulation based on the attitude and field of view in the complete viewpoint configuration information, the shadow transmission module queries along the direction of the incident light ray to determine whether there is a volume along the path, thereby efficiently calculating the cumulative transmittance and achieving real-time soft shadow approximation.

[0028] 4. Result Feedback and Fusion: After the dedicated hardware completes rendering, the result is transmitted via a high-speed interface (such as PCIe) to a lightweight, multi-threaded server utility. This utility distributes the received data and integrates the processing results. The server then organizes the results and sends them back to the main control system via a network protocol (such as TCP). The workflow is as follows: Figure 3 As shown, when the main control program starts rendering, it transmits complete viewpoint data to the viewpoint data relay module. The relay module calculates the camera's transformation relative to the object's origin based on this information and the pose of the target object in the scene coordinate system. This information is then transmitted via TCP to the control program of the neural rendering coprocessor as the camera's extrinsic parameters. Upon receiving the viewpoint configuration, the control program initiates rendering of objects from the same viewpoint. At this time, the GPU and the neural rendering coprocessor simultaneously render their required data. The GPU uses the coprocessor's rendering results from the previous frame for compositing the current frame. After the neural rendering coprocessor finishes rendering the current frame, the data is synthesized into bitmap data by the coprocessor control program and transmitted back to the relay module via TCP. The relay module replaces the material data in the GPU with the newly transmitted bitmap data for image compositing in the next frame.

[0029] In summary, this invention achieves synchronous startup and parallel execution of two heterogeneous rendering pipelines by uniformly generating and distributing viewpoint parameters through a main control program. Combined with a subsequent screen alignment and fusion mechanism, a low-latency, loosely coupled, and highly coordinated end-to-end collaborative framework is constructed, effectively solving the key challenge of real-time integration of high-fidelity content.

[0030] like Figure 4As shown, the hybrid rendering and compositing system provided by this invention has a hardware core of a quad-core EDR-NR neural rendering coprocessor implemented on the Inspur F37x FPGA platform. This FPGA platform integrates a Xilinx XCVU37P device. The main control program runs on a host equipped with an AMD Ryzen Threadripper 3960X processor and is connected to the FPGA board via a PCIe XDMA interface. Simultaneously, a lightweight multi-threaded server tool also runs on the host, communicating with the main control program via TCP sockets.

[0031] In this embodiment, the main control program first generates the camera pose information for the current frame, including the camera's position in the world coordinate system, rotation attitude, and field of view (FoV). These parameters are simultaneously sent to two targets: one is the graphics processing unit (GPU) on the host computer, used to initiate rasterization rendering of the dynamic 3D mesh scene; the other is sent to the multi-threaded server tool via a network protocol. After receiving the parameters, the server immediately distributes them to the EDR-NR accelerator on the FPGA via the PCIe interface.

[0032] After receiving the camera parameters, the EDR-NR accelerator divides the output image into four quadrant regions with the center of the screen as the origin, and processes them in parallel by four independent computing cores. Each core performs ray traversal and neural network inference based on the shared optical center and the field of view aligned with the long side of the display device, generates the RGBA color and Alpha channel data for the corresponding region, and writes it to the DDR memory on the FPGA board.

[0033] Meanwhile, the main control system's GPU begins rendering the foreground 3D scene it is responsible for. After EDR-NR completes rendering a frame, the server's multi-threaded pipeline asynchronously reads the rendering result from DDR memory, fills it into a pixel map, and then sends the complete RGBA frame back as texture data to the main control program via a TCP connection. Compositing and soft shadow simulation effects are as follows... Figure 5 As shown.

[0034] Tests showed that when processing the "Chair" NeRF dataset, the system took approximately 4.5 milliseconds to render a single frame of EDR-NR, while the entire system, on a host equipped with an entry-level NVIDIA GTX 1050 Ti GPU, had a post-fusion processing time of approximately 11 milliseconds per frame, verifying the efficiency and feasibility of the proposed solution.

[0035] The embodiments described above merely illustrate specific implementations of the present invention, and while the descriptions are detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these all fall within the protection scope of the present invention. Therefore, the protection scope of this invention patent should be determined by the appended claims.

Claims

1. A 3D visual synthesis system based on heterogeneous collaboration and screen alignment fusion, characterized in that, The system comprises an FPGA and a main control system. A neural rendering coprocessor is configured within the FPGA. A lightweight, multi-threaded server distributes data received from the main control system and integrates the processing results. The main control system interconnects with an image acquisition device for real-time image acquisition. At the start of processing each frame, the main control system first generates complete viewpoint configuration information for the current scene acquired by the image acquisition device. This complete viewpoint configuration information, as a single data source, is simultaneously distributed to both the main control system's graphics processing unit (GPU) and the neural rendering coprocessor. Based on the received viewpoint configuration, the GPU performs rasterization rendering of the dynamic 3D scene in its standard rendering pipeline, generating foreground and background images. Simultaneously, the neural rendering coprocessor, based on the same viewpoint configuration, initiates its hardware-accelerated ray traversal and neural network inference processes, generating corresponding high-fidelity RGB color and alpha channel data. The resulting data is transmitted back to the main control system, seamlessly merging with the foreground content generated by the GPU to form the final image. The entire process requires no depth information exchange or geometric calibration.

2. The 3D visual synthesis system based on heterogeneous collaboration and screen alignment fusion according to claim 1, characterized in that, The main control system calculates the camera pose transformation relative to the object's origin based on the transformation of the object and camera in the world coordinate system in the dynamic scene of the main control system. This transformation is then distributed as viewpoint configuration information to the neural rendering coprocessor, ensuring that the two rendering pipelines, the graphics processing unit and the neural rendering coprocessor, have completely consistent imaging parameters from the beginning.

3. The 3D visual synthesis system based on heterogeneous collaboration and screen alignment fusion according to claim 2, characterized in that, The graphics processing unit of the main control system uses the same viewpoint parameters generated by the main control program to calculate the precise sampling coordinates of the object results rendered by the neural rendering coprocessor in the screen space, and directly aligns them naturally in the screen space for image fusion.

4. The 3D visual synthesis system based on heterogeneous collaboration and screen alignment fusion according to claim 2 or 3, characterized in that, The shadow projection module in the main control system directly reuses the data structure describing the occupancy of three-dimensional space inside the neural rendering coprocessor. When the main control system performs lighting simulation based on the pose and field of view in the complete viewpoint configuration information, the shadow projection module queries the data structure along the direction of the incident light to determine whether there is a volume on the path, thereby calculating the cumulative transmittance and realizing real-time soft shadow approximation.

5. A working method for a 3D visual synthesis system based on heterogeneous collaboration and screen alignment fusion, characterized in that, In the 3D visual synthesis system based on heterogeneous collaboration and screen alignment fusion as described in claim 4, the main control program transmits complete viewpoint data to the viewpoint data relay module when rendering begins. The relay module calculates the transformation of the camera relative to the object's origin based on this information and the pose of the target object in the scene coordinate system of the neural rendering coprocessor. This information is then transmitted to the control program of the neural rendering coprocessor via TCP as the camera extrinsic data. The control program starts rendering objects from the same viewpoint after receiving the viewpoint configuration. At this time, the graphics processing unit and the neural rendering coprocessor simultaneously render the data they need. The graphics processing unit uses the rendering result of the coprocessor in the previous frame to synthesize the current frame. After the neural rendering coprocessor finishes rendering the current frame, the data is synthesized into bitmap data by the coprocessor control program and transmitted back to the relay module via TCP. The relay module replaces the material data in the graphics processing unit with the newly transmitted bitmap data for image synthesis in the next frame.

6. The working method of the 3D visual synthesis system based on heterogeneous collaboration and screen alignment fusion according to claim 5, characterized in that, The neural rendering coprocessor uses an EDR-NR accelerator. After receiving camera parameters, the EDR-NR accelerator divides the output image into four quadrant regions with the center of the screen as the origin. These quadrant regions are then processed in parallel by four independent computing cores. Each core performs ray traversal and neural network inference based on a shared optical center and a field of view aligned with the long side of the display device. This generates RGBA color and Alpha channel data for the corresponding region and writes it to the DDR memory on the FPGA board. Simultaneously, the graphics processing unit of the main control system begins rendering the foreground 3D image it is responsible for. After the EDR-NR accelerator completes rendering a frame, the multi-threaded pipeline of the multi-threaded server asynchronously reads the rendering result from the DDR memory and fills it into a pixel image. Subsequently, the complete RGBA frame is sent back to the main control program as texture data via a TCP connection.

Images