Real-time 3D display system and method based on synchronous acquisition from multiple cameras

Abstract

This invention discloses a real-time 3D display system and method based on multi-camera synchronous acquisition, relating to the fields of computer vision and augmented reality. Addressing the problems of poor real-time performance in dynamic scene reconstruction and difficulties in aligning virtual and real data on the edge, this application utilizes a multi-camera synchronous acquisition module in the cloud to acquire high-precision temporally aligned multi-view images. Features are extracted using an SP-GS neural network model, and a high-dimensional Gaussian parameter matrix is ​​generated through regression to construct a dynamic 3D point cloud. Subsequently, a 3D reconstruction engine performs temporal noise reduction filtering, high-frequency pose estimation, and multi-level rendering optimization on the point cloud. A multi-threaded parallel streaming module pushes the video, Gaussian point cloud, and prior pose displacement information to the terminal with low latency. On the AR edge, high-precision pose constraints are used to correct spatial positioning and achieve real-time overlay splash rendering and accurate fusion display of the virtual Gaussian model.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and augmented reality, and more specifically, to a real-time 3D display system and method based on synchronous acquisition from multiple cameras. Background Technology

[0002] With the continuous deepening and evolution of augmented reality, virtual reality, and the metaverse concept across industries, the high-precision digitization of real-world physical scenes through 3D reconstruction technology and its fusion display on terminal devices has become a core foundation for building next-generation human-computer interaction and spatial computing platforms. In numerous cutting-edge application scenarios such as precision industrial online inspection, highly interactive virtual reality live streaming, remote assisted medical care in complex environments, and digital twins for intelligent manufacturing, high-fidelity real-time 3D modeling and spatial holographic display of highly dynamic scenes containing complex dynamic objects are often required. Existing 3D reconstruction technologies are gradually revealing insurmountable technical limitations and architectural bottlenecks when facing these stringent business requirements of high dynamism and low latency.

[0003] Existing 3D reconstruction architectures typically rely on a single computing node or highly integrated edge devices to simultaneously handle the parallel acquisition of multi-source visual data and the heavy workload of geometric spatial computation. Since 3D modeling of dynamic scenes involves extremely high-dimensional data tensor operations and complex spatial mapping optimization, the computational resources consumed by a single node are extremely high when faced with a concurrent influx of massive amounts of multi-view high-definition image data. This easily triggers system computational bottlenecks and memory bandwidth exhaustion, resulting in persistently high end-to-end processing latency. Consequently, it becomes impossible to output a coherent, smooth, and frame-free high-frame-rate 3D video stream on a physical time scale. Furthermore, dynamic objects in real-world physical scenes often exhibit unpredictable, high-speed, and irregular motion characteristics. If traditional multi-view visual acquisition devices employ loose software clock synchronization or asynchronous triggering mechanisms, it is highly likely that tiny temporal phase differences will occur between the image sensors at the moment of exposure for each viewpoint. This millisecond-level temporal asynchrony, when mapped to a high-speed moving 3D space, directly translates into severe object motion ghosting, spatial geometric tearing, and spatiotemporal topological misalignment, rendering any subsequent high-precision 3D feature extraction and model matching algorithms without a reliable data foundation.

[0004] More critically, in the terminal presentation stage of augmented reality, when highly complex dynamic 3D model data is pushed to lightweight AR headsets or mobile smart terminals with strict limitations in computing power, power consumption, and size, the terminal devices mainly rely on built-in pure vision or inertial odometry for spatial positioning due to the lack of absolute coordinate references in the global dimension and strong prior support for high-frequency and high-precision pose estimation. This purely terminal-side SLAM system is prone to trajectory drift and continuously amplified cumulative errors in pose estimation when dealing with large-scale movement or missing scene textures. Such spatial positioning errors directly lead to serious distortions when the rendered virtual 3D model is superimposed on the real physical world video stream, such as obvious model jitter, disordered depth occlusion relationships, and detached virtual-real fusion boundaries. This makes it impossible to establish and maintain accurate, stable, and low-latency pixel-level alignment between the virtual model and the real physical scene, greatly undermining the immersiveness and reliability of spatial interaction in the augmented reality system. Summary of the Invention

[0005] In view of the above-mentioned technical defects in the existing technology, the purpose of this invention is to provide a real-time 3D display system and method based on synchronous acquisition by multiple cameras, so as to solve the core technical problems of traditional 3D reconstruction technology when dealing with complex dynamic scenes, such as poor real-time performance due to highly limited computing resources, disordered dynamic modeling structure due to asynchronous acquisition from multiple perspectives, and difficulty in virtual-real alignment and large cumulative positioning errors due to insufficient computing power of end devices.

[0006] To achieve the above-mentioned objectives, the present invention provides a real-time 3D display system based on multi-camera synchronous acquisition, including a multi-camera synchronous acquisition module, a point cloud generation module, a real-time 3D reconstruction engine, a multi-threaded parallel streaming module, and an AR display module deployed on an AR terminal device, all deployed on a cloud server.

[0007] The multi-camera synchronous acquisition module is used to achieve time synchronization acquisition of multiple cameras through a triggering method to obtain multi-view images under the same time reference; the multi-camera synchronous acquisition module adopts a precise time protocol for time synchronization control to constrain the acquisition time synchronization error of the multiple cameras to within 1 millisecond.

[0008] The point cloud generation module is used to perform feature extraction and parameter regression processing on the multi-view image using an SP-GS neural network model to generate a dynamic 3D point cloud. The SP-GS neural network model includes a feature extraction layer, a superpoint detection layer, and a Gaussian parameter generation layer connected in sequence. The feature extraction layer is used to extract and fuse multi-scale feature maps based on a feature pyramid network constructed with a ResNet-18 backbone network.

[0009] The superpoint detection layer is used to calculate the confidence score of a superpoint at each spatial location in the multi-scale feature map based on the keypoint response function, and to select several independent points with the highest built-in confidence in the local area as the finally determined superpoints and to directly extract high-dimensional feature vectors from the corresponding positions in the multi-scale feature map.

[0010] The Gaussian parameter generation layer is used to predict the mean, color, transparency and covariance matrix of the three-dimensional spatial position of each superpoint through the parametric regression sub-network, using the position of the superpoint and its corresponding high-dimensional feature vector as the core input matrix, and outputs a dynamic three-dimensional point cloud represented by a set of Gaussian parameters.

[0011] The real-time 3D reconstruction engine is used to sequentially perform point cloud filtering, pose estimation and rendering optimization processes on the dynamic 3D point cloud, and output the optimized ready-state point cloud and 6-DOF pose.

[0012] The point cloud filtering process is used to sequentially apply statistical outlier removal and temporal consistency filtering to the initially generated dynamic 3D point cloud to obtain a purified point cloud.

[0013] The pose estimation process is used to register the cleaned point cloud of the current frame with the historical key frame using an algorithm based on iterative nearest point, and to calculate the 6-DOF pose in the global coordinate system in real time.

[0014] The rendering optimization process is used to take the purified point cloud and the 6-DOF pose as joint inputs, and sequentially perform frustum boundary culling, hierarchical detail dynamic adjustment based on distance, and illumination consistency color fine-tuning to output the ready point cloud; the multi-threaded parallel streaming module is used to allocate the video stream, the ready point cloud, and the 6-DOF pose to independent threads for network transmission and send them to the AR terminal device.

[0015] The AR display module is used to receive data transmitted through the network, use the received 6-DOF pose as a priori keyframe constraints to provide positioning initialization constraints for the SLAM system built into the AR terminal device, and assist in completing the spatial positioning of the AR terminal device in the current physical environment; and render the ready-state point cloud into a virtual 3D model in real time, and overlay it with the real physical scene collected by the AR terminal device.

[0016] As a preferred technical solution of the present invention, the multi-threaded parallel streaming module transmits the video stream, the ready-state point cloud, and the 6-DOF pose through heterogeneous networks via Real-Time Streaming Protocol (RTSP), Transmission Control Protocol (TCP), and User Datagram Protocol (UDP), respectively.

[0017] This invention also provides a real-time 3D display method based on synchronous acquisition by multiple cameras, comprising:

[0018] A multi-camera synchronous acquisition module achieves time-synchronized acquisition from multiple cameras via a trigger mechanism to obtain multi-view images under the same time reference. This module employs a precise time protocol for time synchronization control, constraining the time synchronization error of the multiple cameras to within 1 millisecond. The multi-view images are processed by an SP-GS neural network model built into a point cloud generation module to generate a dynamic 3D point cloud. The SP-GS neural network model includes a feature extraction layer, a superpoint detection layer, and a Gaussian parameter generation layer connected in sequence. The feature extraction layer is based on ResNet-18... A feature pyramid network constructed from the backbone network extracts and fuses multi-scale feature maps. The superpoint detection layer calculates the confidence score of a superpoint at each spatial location in the multi-scale feature map based on the keypoint response function, and uses a non-maximum suppression algorithm to select several independent points with the highest built-in confidence in the local region as the final superpoints, and directly extracts high-dimensional feature vectors from the corresponding positions in the multi-scale feature map. The Gaussian parameter generation layer uses the position of the superpoint and its corresponding high-dimensional feature vector as the core input matrix, and predicts the mean, color, transparency, and covariance matrix of the three-dimensional spatial position of each superpoint through a parametric regression sub-network, outputting a dynamic three-dimensional point cloud represented by a set of Gaussian parameters. The real-time three-dimensional reconstruction engine sequentially executes the point cloud filtering process, the pose estimation process, and the rendering optimization process, outputting an optimized ready-state point cloud and a 6-DOF pose. The point cloud filtering process sequentially applies statistical outlier removal and temporal consistency filtering to the initially generated dynamic three-dimensional point cloud to obtain a cleaned point cloud. The pose estimation process uses an iterative nearest point algorithm to extract the cleaned point cloud of the current frame. The system is registered with historical keyframes to calculate the 6-DOF pose in the global coordinate system in real time. The rendering optimization process uses the purified point cloud and the 6-DOF pose as joint inputs, sequentially performing frustum culling, distance-based hierarchical detail dynamic adjustment, and illumination consistency color fine-tuning to output the ready-state point cloud. The multi-threaded parallel streaming module is used to allocate the video stream, the ready-state point cloud, and the 6-DOF pose to independent threads for network transmission to the AR terminal device. The AR display module receives the data transmitted over the network and uses the received 6-DOF pose as a priori keyframe constraint to provide positioning initialization constraints for the SLAM system built into the AR terminal device, assisting in the spatial positioning of the AR terminal device in the current physical environment. The ready-state point cloud is rendered into a virtual 3D model in real time and overlaid on the real physical scene captured by the AR terminal device.

[0019] As a preferred technical solution of the present invention, in the step of time synchronization acquisition of images through the multi-camera synchronous acquisition module, the IEEE1588 protocol is used to perform clock alignment compensation on each camera node in the local area network, so that the phase difference of the acquisition of multiple video streams is less than 1 millisecond.

[0020] As a preferred embodiment of the present invention, the process by which the AR display module renders the ready-state point cloud into a virtual 3D model in real time and overlays it with the real physical scene includes:

[0021] Based on the 6-DOF pose aligned with the SLAM system, the ready-state point cloud coordinate system is transformed into the global coordinate system of the AR terminal device; the graphics processor is invoked and the virtual 3D model is rendered in real time based on the transparency and covariance parameters of the point cloud, and finally pixel-level alignment is achieved with the real physical scene video stream acquired by the AR terminal device.

[0022] The beneficial technical effects of this invention are as follows: This invention creatively proposes and successfully implements a deeply complementary cloud-edge collaborative system computing architecture. It cleverly offloads all computationally complex high-dimensional dynamic scene feature extraction, real-time reconstruction of differentiable Gaussian point clouds based on neural networks, and multi-level sequential resource-intensive optimization tasks, including filtering and denoising, high-frequency pose estimation, and view frustum pruning, to cloud or edge computing server clusters with massive concurrent computing power for centralized high-speed processing. Then, the extremely lightweight ready-to-use point cloud combination, after deep information stripping, compression, and spatial alignment, along with core pose data, is streamed with low latency to user terminals with limited computing power via an intelligently scheduled multi-threaded heterogeneous network protocol. Finally, on the mobile terminal, a dedicated graphics interface completes the final low-power viewpoint-related rendering and virtual-real image overlay fusion. This "cloud-edge collaborative" division of labor and its end-to-end optimization mechanism fundamentally overcomes the industry's core technical bottleneck—the long-standing difficulty in achieving real-time, smooth, and high-quality 3D modeling and reconstruction of complex, large-scale dynamic scenes on resource-constrained portable devices—from its underlying logic. Furthermore, by introducing a physical clock synchronization protocol with sub-millisecond precision at the front-end acquisition side, the system fundamentally eliminates motion blur and multi-view spatial geometric topological misalignment. Simultaneously, at the terminal display side, a high-frequency, high-precision pose data stream delivered in real-time from the cloud serves as a strong prior constraint loop for terminal SLAM perception, effectively correcting common issues in pure visual positioning of mobile devices, such as trajectory drift, cumulative error amplification, and computational latency. Combining these innovations, the system ultimately achieves extremely low latency in end-to-end transmission and processing, maintains an extremely high refresh rate, and achieves a high-quality augmented reality display effect where the virtual digital reconstruction model and the real physical environment scene maintain extremely high spatial precision pixel-level alignment and overlay. This invention's solution balances high algorithm generalization ability with excellent system lateral scalability, possessing extremely broad prospects for industrial application transformation. It is particularly well-suited for many high-precision and cutting-edge technology application scenarios, such as online visual inspection of precision components in industrial automation, virtual reality digital human e-commerce live streaming that pursues an ultimate immersive interactive experience, and complex on-site remote cross-regional assisted medical surgery where the fidelity of the spatial model and the system feedback latency have critically stringent requirements. Attached Figure Description

[0023] Figure 1 A schematic diagram of the overall architecture of a real-time 3D display system based on synchronous acquisition by multiple cameras provided in an embodiment of the present invention;

[0024] Figure 2 This is a diagram illustrating the internal structure of the SP-GS neural network model provided in an embodiment of the present invention.

[0025] Figure 3 This is a flowchart of the spatiotemporal serialization progressive optimization process of the real-time 3D reconstruction engine provided in this embodiment of the invention;

[0026] Figure 4 This is a schematic diagram of cloud-side and edge-side collaborative processing and data flow interaction provided in an embodiment of the present invention. Detailed Implementation

[0027] To make the technical solution and effects of the present invention clearer, the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only for explaining the present invention and do not constitute a limitation on the scope of protection of the present invention. Various modifications and improvements made to the present invention by those skilled in the art without departing from the design concept of the present invention should fall within the scope of protection of the present invention.

[0028] This invention discloses a real-time 3D display system based on synchronous acquisition from multiple cameras. The overall architecture of this system aims to solve the technical bottlenecks of traditional 3D reconstruction technology when dealing with complex dynamic scenes, such as poor real-time performance, excessive consumption of computing resources, and difficulty in dynamic modeling. Figure 1 The system architecture diagram shown depicts a system primarily composed of a multi-camera synchronous acquisition module, a point cloud generation module, a real-time 3D reconstruction engine, an AR display module, and a multi-threaded parallel streaming module. Within this overall architecture, the system achieves an organic combination of high-precision dynamic reconstruction and lightweight real-time presentation through a reasonable division of labor between the cloud and the edge. The system's input supports simultaneous access and processing of no fewer than eight high-definition cameras, thereby enabling comprehensive multi-view coverage of the target scene in physical space.

[0029] At the source of data acquisition, the multi-camera synchronous acquisition module undertakes the crucial task of unifying the system's time base and acquiring high-quality raw image sequences. To ensure that each perspective captures the scene state within the same extremely short time window, this module not only supports traditional hardware or software triggering methods but also, more preferably, employs a time synchronization mechanism based on the PTP protocol, i.e., the IEEE 1588 standard, for strict clock alignment. By electing a master clock node within the local area network and using hardware timestamps to record the transmission and reception times of network packets, the system can accurately estimate and compensate for network transmission delays and clock skew between devices. Actual testing and verification have shown that this module can strictly control the synchronization error of multiple cameras to within 1 millisecond. This extremely low-error, sub-millisecond synchronization completely eliminates motion ghosting and spatiotemporal misalignment caused by inconsistent exposure times of different cameras when dynamic objects are moving at high speeds, laying a crucial data foundation for subsequent high-precision 3D feature extraction and model reconstruction.

[0030] Multi-view RGB image data, typically represented as H×W×3, is acquired and spatiotemporally aligned, and then input into the system's point cloud generation module. This module is the core hub for real-time 3D digitization of dynamic scenes. Internally, it employs a deep neural network model called SP-GS to efficiently process multi-view images end-to-end to generate dynamic 3D point clouds. Combined with... Figure 2 The SP-GS neural network model structure diagram shown clearly demonstrates a top-down streamlined architecture, comprising an input layer, feature extraction layer, superpoint detection layer, Gaussian parameter generation layer, and rendering layer. In practical deployments, this model not only possesses powerful forward inference capabilities but also particularly supports online fine-tuning for specific dynamic scenes, continuously adapting to subtle changes in ambient lighting or object appearance.

[0031] As the front end of visual understanding, the feature extraction layer employs a Feature Pyramid Network (FPN) structure built on a ResNet-18 backbone to extract and fuse visual features across different scales from the input multi-view images. Choosing ResNet-18 as the backbone network ensures sufficient feature representation while minimizing computational complexity to meet the system's real-time requirements. During operation, this layer first utilizes shallow convolutional networks to capture low-level detailed features, such as geometric edges, sharp corners, and fine surface textures in the scene. This is crucial for subsequently reconstructing high-fidelity geometric details of objects. As the network's receptive field expands, mid-level networks begin extracting mid-level structural features encompassing local shape and component-level semantic information, enabling the system to initially understand the physical components of objects. Further, deep networks are responsible for extracting high-level semantic features, which include explicit object category attributes and global scene contextual relationships, ensuring the semantic consistency and physical plausibility of the final 3D reconstruction. Through the top-down path and lateral connections of the FPN architecture, the system effectively fuses the features of the above-mentioned levels. The output multi-scale fused feature map constitutes the basic visual representation system for all subsequent superpoint detection and Gaussian parameter generation steps.

[0032] After acquiring the multi-scale fused feature map, the superpoint detection layer intervenes in the processing flow. Its main responsibility is to locate the salient regions that contribute the most to 3D reconstruction in the vast image space based on the keypoint response function. This layer incorporates a lightweight keypoint response network to calculate the confidence score of a superpoint at each spatial location in the feature map. To prevent excessively dense distribution of superpoints and waste of computational resources, the system then applies a non-maximum suppression algorithm, namely the NMS algorithm, to simultaneously perform spatial filtering in the 2D image domain and the high-dimensional feature domain, thereby selecting several independent points with the highest built-in confidence in local regions as the final determined superpoints. This layer then outputs the 2D or 3D prior coordinates of each selected superpoint and its high-dimensional feature vector directly extracted from the corresponding position in the multi-scale feature map. The superpoint detection layer plays a pivotal role in the entire network, transforming continuous and dense image features into discrete salient geometric points. It directly relies on the rich representation matrix provided by the feature extraction layer to accurately locate the key regions for reconstruction and seamlessly pass this highly condensed spatial and semantic information to the next processing stage.

[0033] The Gaussian parameter generation layer is the core component of the SP-GS model, transforming two-dimensional features into a three-dimensional renderable representation. This layer uses the superpoint locations and their corresponding high-dimensional feature vectors output by the superpoint detection layer as its core input matrix. Internally, it integrates a complex parametric regression subnetwork to predict a set of mathematical parameters for each detected superpoint, fully defining the properties of the three-dimensional Gaussian ellipsoid. These parameters specifically include the mean parameter of the three-dimensional spatial location, i.e., the Gaussian distribution. Color information that reflects appearance characteristics, namely RGB value parameters , Transparency parameter for controlling the blending effect And the covariance matrix parameters that determine the shape and orientation of the ellipsoid. The prediction process for these high-dimensional parameters deeply integrates visual features from the corresponding superpoint locations, thus ensuring, mathematically, that the generated Gaussian properties are highly consistent with the objective appearance exhibited by the multi-view images. In the specific neural network implementation, the system constructs a learnable nonlinear mapping relationship to achieve parameter regression calculation. Specifically, the... The color parameters of each superpoint can be obtained through activation function regression, i.e. Its transparency parameter is limited by a S-shaped function, i.e. To ensure the positive definiteness of the covariance matrix, it is usually regressed on a logarithmic scale, i.e. In the above mapping relationship, Indicates the first The feature vectors of each superpoint , , as well as , , These represent the learnable weight matrix and bias vector of the corresponding parameter regression network layer, respectively. This represents a nonlinear activation function. This series of complex regression processes tightly couples the input two-dimensional image features with geometric location information and semantic appearance information, directly generating a three-dimensional representation that can be used for subsequent efficient rendering. The final Gaussian parameter set output by this layer essentially constitutes the mathematical entity of the dynamic three-dimensional point cloud defined by this system. Further referring to the above activation function, sigmoid function, and covariance matrix regression formula, it theoretically clarifies the rigorous mathematical principle of the differentiable generation of point cloud physical properties from abstract image features.

[0034] As the final step in the point cloud generation module, the rendering layer is responsible for real-time visualization of the dynamic point cloud data output by the Gaussian parameter generation layer. This layer innovatively employs AlphaSplatting, a Gaussian splash rendering technique, to achieve rapid rasterization rendering of massive discrete points. During processing, the system uses a GPU-optimized tile renderer based on parameters such as the 3D position, spatial covariance, color, and transparency of each Gaussian point. It directly projects the 3D Gaussian point cloud in a back-to-forward depth order and blends it onto any user-specified 2D image viewing plane according to the transparency formula, thereby generating a highly realistic image with rich details. For dynamically changing scene sequences, the rendering layer also possesses temporal processing capabilities. It can perform temporal interpolation and smooth fusion of point cloud data acquired between consecutive video frames based on the timestamps of the input data, thus outputting a naturally transitioning and smooth 3D reconstructed video stream. As the final output interface of the entire neural network pipeline, the rendering layer directly receives the set of preceding Gaussian parameters. The rendering quality of its output image and the frame rate performance of the system depend directly on the computational performance of the preceding superpoint detection and Gaussian parameter generation network layers in terms of point cloud spatial precision distribution and structured optimization.

[0035] The initial dynamic point cloud data output from the SP-GS model, while possessing a complete 3D representation, still needs to be fed into a real-time 3D reconstruction engine for further serialization optimization and rendering enhancement to meet the stringent display requirements of AR devices. This engine bears the heavy responsibility of post-processing optimization of the point cloud data and scheduling rendering resources. Combined with... Figure 3The flowchart shown illustrates the optimization process of the real-time 3D reconstruction engine. This optimization process is meticulously designed as a progressive pipeline comprising three core sub-processes: point cloud filtering, pose estimation, and rendering optimization. The first sub-process is the point cloud filtering sub-process, which aims to deeply cleanse the initial Gaussian point cloud generated in each frame. At this stage, the system prioritizes an outlier removal algorithm based on statistical principles, calculating the distance distribution characteristics of each point within its local neighborhood, thereby forcibly eliminating isolated noise points caused by image sensor noise or neural network feature matching errors. Subsequently, the system introduces a temporal consistency filter. By comparing and tracking the motion trajectory of the same semantic point cloud features across multiple consecutive frames on the time axis, it sensitively captures and filters out invalid feature points exhibiting abnormally high-frequency jitter or violating physical kinematics laws. Finally, it outputs a cleaned point cloud with a compact spatial structure, stable temporal series, and extremely low overall noise to the next stage, serving as crucial pre-input data for pose estimation.

[0036] Next, the system enters the pose estimation subprocess, which uses the aforementioned purified point cloud sequence as the core spatial reference input. The system employs an improved variant of the Iterative Closest Point (ICP) algorithm. By constructing an error metric function between points and surfaces or between points and models, it performs high-precision spatial registration calculations between the point cloud data of the latest frame and the local scene map or historical keyframes established during system initialization. Through this continuous registration iteration, the system can calculate the absolute spatial position of the user wearing the AR device in the global world coordinate system in real time, or track the 6-DOF high-frequency pose data of independent dynamic objects in the scene. This crucial 6-DoF pose data has a dual purpose: firstly, it is passed back to the front-end point cloud generation module through the feedback channel, helping the neural network achieve closed-loop alignment and error correction of spatial pose; secondly, it is packaged as a system-level key parameter and directly passed to subsequent rendering optimization steps and the final terminal display device.

[0037] Rendering optimization is the final step in the real-time 3D reconstruction engine pipeline. It requires the purified point cloud data obtained from the previous two steps and high-precision real-time pose information as joint inputs to execute a multi-dimensional, multi-level rendering optimization strategy. First, based on the current viewing angle calculated from the latest pose data, the system performs strict frustum culling in 3D space, completely discarding all Gaussian point cloud data outside the user's field of view to significantly reduce invalid rendering load. Next, the system performs hierarchical detail optimization, also known as LOD optimization. The algorithm calculates the physical distance of point cloud groups from the user's viewing position in real time and dynamically adjusts the rendering granularity and Gaussian sphere sampling density of each region accordingly. After geometry and density optimization, the system finally makes subtle adjustments to the transparency and color parameters of Gaussian points within the field of view based on global illumination consistency physical constraints. This eliminates abrupt changes in lighting and shadows that may occur when switching between different viewpoints, significantly improving the visual realism of the final virtual-real fusion scene. After the above-mentioned rigorous optimization process, the system finally outputs a high-quality point cloud combination and its associated rendering parameters that are in a "ready state" and can be directly delivered to the AR end and used by the rendering layer. This marks the completion of all processing procedures of the real-time 3D reconstruction engine in the cloud.

[0038] Because the system employs a collaborative architecture where the cloud handles the heavy reconstruction computations and the edge handles the lightweight display, efficient data transmission becomes particularly important. The system's multi-threaded parallel streaming module is specifically designed to intelligently distribute and transmit different types of data streams according to their varying bandwidth and latency requirements, minimizing end-to-end system latency from data acquisition to final display. In practice, the system operates multiple independent transmission threads. Two-dimensional reference video streams, which require time continuity, are allocated to the RTSP protocol channel for continuous streaming. Larger, high-precision, ready-to-use point cloud data with stringent integrity requirements are allocated to the TCP protocol channel to ensure absolute reliability and zero packet loss during transmission. For small amounts of spatial attitude information with extremely demanding real-time requirements, the system uses a low-overhead UDP protocol channel for high-frequency broadcast transmission. This refined protocol segmentation and parallel streaming design enables the efficient and smooth delivery of video streams, point cloud data, and attitude information to the user's AR device.

[0039] The final display stage of the system relies on a highly integrated AR display module. The core task of this module is to overlay the received reconstruction results onto the real physical world in real time using high-transmittance AR glasses or a portable display screen. For example... Figure 4The diagram illustrating the cloud-edge collaborative data flow visually compares and demonstrates the division of labor and collaborative interaction between the cloud server pipeline and the AR device's edge pipeline. After the AR device receives the video stream pushed from multiple threads, the optimized ready-state Gaussian point cloud data, and its corresponding real-time pose sequence, the device's built-in processor immediately initiates a series of operations to achieve perfect virtual-real fusion. The first step is SLAM localization enhancement. While the AR device itself has a built-in SLAM system for spatial awareness, pure edge-side operation is prone to accumulating errors. At this point, the device system utilizes high-precision real-time pose data transmitted via the network as strong prior information for localization initialization or global keyframe constraints, thereby assisting the edge-side SLAM system in quickly achieving accurate spatial localization in the current physical environment. This mechanism not only significantly reduces trajectory drift and accumulated errors in complex dynamic scenes with pure edge-side SLAM but also greatly reduces the computational load and latency of the edge chip, achieving deep support and enhancement of SLAM spatial localization by the AR display module.

[0040] After achieving precise positioning, the AR device enters the coordinate system and model rendering phase. The system's computing unit receives 3D Gaussian point cloud data with complete rendering parameters from the network and, strictly adhering to the precise pose information obtained in the previous step and aligned with the device's own SLAM system, accurately projects and transforms it into the AR device's global world coordinate system through matrix multiplication. After the transformation, the system's underlying engine directly calls the device's built-in high-performance mobile GPU, fully utilizing the Gaussian parameter set of each point cloud—namely, spatial position, spatial covariance, color RGB values, and transparency alpha values—again employing efficient AlphaSplatting technology to generate realistic 3D model pixel grids in real time within an extremely short rendering cycle.

[0041] Finally, in the real-time overlay and display stage, the system rigorously aligns the high-quality, low-latency 3D virtual model image just rendered by the GPU with the real-time video stream of the physical scene captured by the AR device's front-facing camera, down to the pixel level. Technically, this alignment and image fusion operation is typically achieved by calling standard rendering interfaces provided by the system-level AR runtime library, such as using Apple's ARKit or Android's ARCore underlying interfaces to complete the final compositing of the mixed images. Through these steps, the system ultimately outputs stable, naturally lit, and seamless augmented reality fusion images on the user's AR glasses lenses or terminal display, achieving precise and seamless physical alignment between the virtual model and the real scene.

[0042] The method and system provided in this invention possess outstanding innovation in their overall technical approach. Their core advantage lies in the creative proposal and implementation of a deep collaborative design at the system architecture level: it fully utilizes cloud or edge computing server clusters with abundant computing power for high-precision dynamic scene feature extraction, point cloud reconstruction, and multi-level optimization processing. Subsequently, the Gaussian point cloud clusters, which have undergone deep stripping and lightweighting and have completed initial alignment, along with high-frequency pose data, are efficiently and stream-transmitted to user terminals with limited computing power. Finally, viewpoint-related rendering and virtual-real image fusion are completed on the device side. This unique "cloud-device collaborative" division of labor fundamentally breaks through the long-standing industry technical bottleneck of achieving real-time, high-quality 3D reconstruction of complex dynamic scenes on resource-constrained portable devices. It not only ensures the geometric accuracy and visual fidelity of the reconstructed model but also significantly enhances the overall augmented reality experience for users, providing extremely low interaction latency and a highly immersive experience. This solution excels in system scalability and technology generalization, and has great potential for industrial application. It is widely applicable and highly suitable for many cutting-edge application scenarios, such as online inspection of industrial precision parts, highly interactive virtual reality e-commerce live streaming, and remote assisted medical care that requires extremely high visual accuracy and zero-latency feedback. This invention encompasses a complete and logically rigorous real-time 3D reconstruction and augmented reality display method and system. The process includes: Step 1, system hardware initialization; Step 2, PTP microsecond-level multi-view image synchronous acquisition and transmission; Step 3, feature extraction, superpoint detection, and regression generation of a high-dimensional dynamic 3D point cloud containing mean color transparency and covariance based on a complex and highly optimized SP-GS neural network model; Step 4, rigorous point cloud filtering and purification, ICP high-frequency pose estimation, and multi-level rendering optimization including frustum culling and LOD in the cloud engine to generate ready-to-use data; Step 5, seamless diversion of complex data to network channels to reduce latency using multi-threading and heterogeneous network transmission protocols; and finally, Step 6, where the AR device uses prior pose to strongly constrain its own SLAM system to complete Gaussian point cloud reprojection and the pixel-level fusion display of the virtual and real data at the AR runtime library level. This comprehensive system fully embodies the core technological contributions and outstanding technical effects of this invention.

[0043] It should be noted that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A real-time 3D display system based on multi-camera synchronous acquisition, including a multi-camera synchronous acquisition module, a point cloud generation module, a real-time 3D reconstruction engine, a multi-threaded parallel streaming module, and an AR display module deployed on an AR terminal device, all deployed on a cloud server. The multi-camera synchronous acquisition module is used to achieve time synchronization acquisition of multiple cameras through a triggering method to obtain multi-view images under the same time reference; the multi-camera synchronous acquisition module adopts a precise time protocol for time synchronization control to constrain the acquisition time synchronization error of the multiple cameras to within 1 millisecond. The point cloud generation module is used to perform feature extraction and parameter regression processing on the multi-view image using an SP-GS neural network model to generate a dynamic 3D point cloud. The SP-GS neural network model includes a feature extraction layer, a superpoint detection layer, and a Gaussian parameter generation layer connected in sequence. The feature extraction layer is used to extract and fuse multi-scale feature maps based on a feature pyramid network constructed with a ResNet-18 backbone network. The superpoint detection layer is used to calculate the confidence score of a superpoint at each spatial location in the multi-scale feature map based on the keypoint response function, and to select several independent points with the highest built-in confidence in the local area as the finally determined superpoints and to directly extract high-dimensional feature vectors from the corresponding positions in the multi-scale feature map. The Gaussian parameter generation layer is used to predict the mean, color, transparency and covariance matrix of the three-dimensional spatial position of each superpoint through the parametric regression sub-network, using the position of the superpoint and its corresponding high-dimensional feature vector as the core input matrix, and outputs a dynamic three-dimensional point cloud represented by a set of Gaussian parameters. The real-time 3D reconstruction engine is used to sequentially perform point cloud filtering, pose estimation and rendering optimization processes on the dynamic 3D point cloud, and output the optimized ready-state point cloud and 6-DOF pose. The point cloud filtering process is used to sequentially apply statistical outlier removal and temporal consistency filtering to the initially generated dynamic 3D point cloud to obtain a purified point cloud. The pose estimation process is used to register the cleaned point cloud of the current frame with the historical key frame using an algorithm based on iterative nearest point, and to calculate the 6-DOF pose in the global coordinate system in real time. The rendering optimization process is used to take the purified point cloud and the 6-DOF pose as joint inputs, and sequentially perform view frustum boundary culling, distance-based hierarchical detail dynamic adjustment and illumination consistency color fine-tuning to output the ready-state point cloud. The multi-threaded parallel streaming module is used to allocate the video stream, the ready-state point cloud, and the 6-DOF pose to independent threads for network transmission and send them to the AR terminal device. The AR display module is used to receive data transmitted through the network, use the received 6-DOF pose as a priori keyframe constraints to provide positioning initialization constraints for the SLAM system built into the AR terminal device, and assist in completing the spatial positioning of the AR terminal device in the current physical environment; and render the ready-state point cloud into a virtual 3D model in real time, and overlay it with the real physical scene collected by the AR terminal device.

2. The real-time three-dimensional display system based on synchronous acquisition by multiple cameras according to claim 1, characterized in that, The multi-threaded parallel streaming module transmits the video stream, the ready-state point cloud, and the 6-DOF pose through heterogeneous networks via Real-Time Streaming Protocol (RTSP), Transmission Control Protocol (TCP), and User Datagram Protocol (UDP), respectively.

3. A real-time 3D display method based on multi-camera synchronous acquisition, applied to the real-time 3D display system based on multi-camera synchronous acquisition as described in claim 1 or 2, comprising: A multi-camera synchronous acquisition module achieves time-synchronized acquisition from multiple cameras via a trigger mechanism to obtain multi-view images under the same time reference. This module employs a precise time protocol for time synchronization control, constraining the time synchronization error of the multiple cameras to within 1 millisecond. The multi-view images are processed by an SP-GS neural network model built into a point cloud generation module to generate a dynamic 3D point cloud. The SP-GS neural network model includes a feature extraction layer, a superpoint detection layer, and a Gaussian parameter generation layer connected in sequence. The feature extraction layer is based on ResNet-18... A feature pyramid network constructed from the backbone network extracts and fuses multi-scale feature maps; the superpoint detection layer is used to calculate the confidence score of a superpoint at each spatial location in the multi-scale feature map based on the keypoint response function, and applies a non-maximum suppression algorithm to select several independent points with the highest built-in confidence in the local region as the final determined superpoints and directly extract high-dimensional feature vectors from the corresponding positions in the multi-scale feature map; the Gaussian parameter generation layer is used to predict the mean, color, transparency, and covariance matrix of the three-dimensional spatial location of each superpoint through a parametric regression sub-network, using the position of the superpoint and its corresponding high-dimensional feature vector as the core input matrix, and outputs a dynamic three-dimensional point cloud represented by a set of Gaussian parameters; the real-time three-dimensional reconstruction engine sequentially executes the point cloud filtering process, pose estimation process, and rendering optimization process, outputting an optimized ready-state point cloud and a 6-DOF pose; the point cloud filtering process is used to sequentially apply statistical outlier removal and temporal consistency filtering to the initially generated dynamic three-dimensional point cloud to obtain a cleaned point cloud; The pose estimation process uses an iterative nearest-point algorithm to register the cleaned point cloud of the current frame with historical keyframes, and calculates the 6-DOF pose in the global coordinate system in real time. The rendering optimization process uses the cleaned point cloud and the 6-DOF pose as joint inputs, and sequentially performs frustum culling, hierarchical detail dynamic adjustment based on distance, and illumination consistency color fine-tuning to output the ready-state point cloud. The video stream, the ready-state point cloud, and the 6-DOF pose are allocated to independent threads for network transmission to the AR terminal device through a multi-threaded parallel streaming module. The AR display module receives the data transmitted through the network, uses the received 6-DOF pose as a priori keyframe constraints to provide positioning initialization constraints for the SLAM system built into the AR terminal device, and assists in completing the spatial positioning of the AR terminal device in the current physical environment. The ready-state point cloud is rendered into a virtual 3D model in real time and overlaid on the real physical scene captured by the AR terminal device.

4. The real-time three-dimensional display method based on synchronous acquisition by multiple cameras according to claim 3, characterized in that, In the step of time-synchronized image acquisition via a multi-camera synchronous acquisition module, the IEEE 1588 protocol is used to perform clock alignment compensation on each camera node in the local area network, so that the phase difference between the acquisition of multiple video streams is less than 1 millisecond.

5. The real-time three-dimensional display method based on synchronous acquisition by multiple cameras according to claim 3, characterized in that, The process by which the AR display module renders the ready-state point cloud into a virtual 3D model in real time and overlays it with the real physical scene includes: Based on the 6-DOF pose aligned with the SLAM system, the ready-state point cloud coordinate system is transformed into the global coordinate system of the AR terminal device; the graphics processor is invoked and the virtual 3D model is rendered in real time based on the transparency and covariance parameters of the point cloud, and finally pixel-level alignment is achieved with the real physical scene video stream acquired by the AR terminal device.

Images