An augmented reality method, system and computer readable storage medium for real-time atomized video

Abstract

The application discloses a kind of real-time atomization video's augmented reality method, system and computer readable storage medium, method includes:1, augmented reality equipment gathers camera picture and coding to generate uplink video stream, and push stream to server;2, server is decoded to uplink video stream, shaping, and encapsulation is frame batch;3, call single view three-dimensional reconstruction model to frame batch, generate three-dimensional main scene and pixel level depth, according to pixel level depth calculation three-dimensional world coordinate;4, generate a atomization Gaussian particle for each pixel;5, 3DGS volume rendering algorithm will Gaussian scatter point of three-dimensional scene with atomization Gaussian particle be volume rendering and forward synthesis, obtain enhancement frame;6, to enhancement frame coding to generate downlink enhancement video stream and push stream to server;7, augmented reality equipment is pulled from server enhancement video stream and real-time decoding and display.The application has the advantages that augmented reality equipment end side burden is low, end-to-end time delay is small and atomization level is controllable.

Description

Technical Field

[0001] This invention belongs to the field of augmented reality technology, specifically an augmented reality method, system, and computer-readable storage medium for real-time fogging video. Background Technology

[0002] Augmented reality has been widely used in scenarios such as driver training and security drills. It emphasizes the completion of three-dimensional understanding and visual synthesis of images from the camera of augmented reality devices within a short time scale that users can perceive, and then transmits them back to the terminal display in a low-latency closed loop. Especially when simulating environments with limited visibility such as heavy fog and smoke, it is expected to present a layered effect where the foreground remains clear and the background gradually becomes foggy with depth, so as to obtain a training experience that is closer to real perception.

[0003] To achieve this goal, existing approaches include edge-side integrated reconstruction and rendering, multi-view 3D perception, 2D post-processing fogging, cloud-based remote compositing, and multi-sensor direct depth driving. Among these: 1) Edge-side integrated reconstruction has the advantage of a short link and closed-loop operation without uplink. However, on mobile SoCs or glasses-level platforms, simultaneously handling single-frame depth estimation, 3D representation construction, and volumetric rendering can easily trigger thermal design and battery life bottlenecks. As resolution, frame rate, or algorithm complexity increases, frame rate and latency become difficult to stabilize within the target range. 2) While multi-view 3D perception can achieve relatively stable imaging geometry, it is sensitive to the quality of external trajectories, particularly in fast motion, weak textures, or occlusion. Obstruction can lead to drift and degradation. For multiple cameras, cross-device calibration and clock alignment are also required, resulting in high system integration and maintenance costs. 3) Although 2D post-processing fogging is simple to deploy, it lacks pixel-level reliable depth constraints, making it difficult to accurately achieve the gradual relationship between near-clear and far-fog. It is prone to problems such as improper masking of foreground details and inconsistent transparency in backlight or nighttime environments. 4) If a multi-sensor solution using ToF, structured light, or LiDAR is used directly, a more stable depth can be obtained in a controlled environment. However, the hardware cost, power consumption, and size are high. Strong outdoor light and sparse sampling at long distances will affect data quality. Moreover, multi-sensor systems also face calibration and timing synchronization challenges, making large-scale deployment difficult.

[0004] In summary, the current key challenge in augmented reality (AR) research lies in how to coordinate video encoding / decoding, transmission protocols, GPU parallelism, 3D representation, and volumetric rendering within a single AR device-server-AR device link. This involves two key aspects: firstly, reliably obtaining 3D representation and depth for fogging from a single frame without relying on extrinsic parameter estimation or multi-view accumulation, while maintaining geometric consistency with resolution-driven intrinsic parameter settings; secondly, maintaining real-time output through a multi-dimensional collaborative strategy involving threads, queues, batch processing, and code control under bandwidth fluctuations and load variations, avoiding sacrificing latency for image quality through long buffering. From an application perspective, if single-frame-driven 3D representation and depth-consistent fogging can be achieved under these two constraints, and end-to-end latency can be controlled through low-latency encoding / decoding and parallel pipelines, then the spatial distribution of scene visibility can be accurately reproduced in driver training, improving trainees' perception and decision-making abilities in adverse weather conditions. Furthermore, in scenarios such as fire drills, special vehicle operations, and educational simulations, parameterized fogging templates can be used to quickly switch environments, reducing costs and increasing training coverage. Therefore, a systematic method is proposed, which is based on single-view to 3DGS, drives the atomized Gaussian particles and transmits them back through the low-latency H.264 RTSP link, and is combined with a server-side three-thread dual-queue and full-load discarding strategy. This method has clear engineering necessity and practical feasibility. Summary of the Invention

[0005] The present invention aims to provide an augmented reality method, system, and computer-readable storage medium for real-time fogging video, which has the advantages of low device-side burden, low end-to-end latency, and controllable fogging levels, and can be applied to application scenarios that require clear foreground and progressive fogging display of background.

[0006] This invention is achieved through the following technical solution: An augmented reality method for real-time fogging video includes the following steps: Step 1: The augmented reality device captures camera footage. The augmented reality device encodes the captured camera footage according to the H.264 standard using a low-latency preset video hardware encoding unit to generate an uplink video stream. The uplink video stream is pushed to the server's stream1 address via an RTSP session. Step 2: The server's decoding thread pulls the stream from the stream1 address, calls the video hardware decoding unit integrated in the GPU to decode the uplink video stream, obtains video frames, and then uses a filter chain in the GPU memory to uniformly shape the video frames to obtain shaped video frames. Then, the shaped video frames are packaged into frame batches and written to the inference queue in units of frame batches. Step 3: The server's inference thread retrieves a batch of frames from the inference queue and processes them in the external parameters. It is an identity matrix and its intrinsic parameters Based on resolution estimation, single-view calls are performed on a batch of frames. Figure 3 The model is reconstructed to generate a 3D main scene represented by 3DGS and obtain pixel-level depth for each frame. Then, based on the inverse intrinsic parameter matrix, the direction vector of each pixel in the 3D main scene is calculated, and the pixel-level depth of each frame is calculated. Multiplying by the direction vector yields the 3D coordinates in the camera coordinate system. Finally, through external parameters and three-dimensional coordinates Calculate the three-dimensional world coordinates of the pixel in three-dimensional space. ; Step 4: Based on the 3D world coordinates obtained in Step 3 Generate a fogged Gaussian particle for each pixel, represented by a quintuple of parameters; Step 5: The server's inference thread inputs the Gaussian scatter points in the 3D scene represented by 3DGS and the fogged Gaussian particles generated in Step 4 into the 3DGS volumetric rendering algorithm, and performs volumetric rendering and forward compositing according to the pixel ray direction to obtain the enhanced frame. Step 6: The server writes the enhanced frame into the encoding queue. The server's encoding thread performs low-latency encoding on the enhanced frame on the GPU side according to the H.264 standard through the video hardware encoding unit to generate a downlink enhanced video stream. The downlink enhanced video stream is pushed to the server's stream2 address through an RTSP session. Step 7: The augmented reality device pulls the augmented video stream from the stream2 address, decodes it in real time, and displays it for the user to observe.

[0007] Furthermore, the low latency parameters of the video hardware encoding units in steps 1 and 6 are: GOP=10~30, B=0, target bitrate=2~8 Mbps, fps=5~60, and pixel format is yuv420p.

[0008] Furthermore, the GPU filter chain in step 2 includes GPU scaling, optional cropping, frame rate shaping, and pixel format conversion, wherein: GPU scaling is used to scale the aspect ratio of the video frame to a value not less than the target resolution; optional cropping is used to crop the center of the GPU-scaled video frame to the target resolution; frame rate shaping is used to shape the video frame cropped to the target resolution to the target frame rate; and pixel format conversion is used to convert the video frame to the required color format.

[0009] Furthermore, the target resolution and target frame rate are 1280×720 and 15 fps, respectively, and every three consecutive video frames are packaged into a frame batch.

[0010] Furthermore, the internal parameters of step 3 for A matrix, represented as: In the formula: and They represent direction and Pixels representing the direction; and These represent the coordinates of the principal point in the pixel coordinate system; and These represent the width and height of the target resolution, respectively. These are empirical coefficients used to adjust the field of view. .

[0011] Furthermore, the three-dimensional world coordinates of step 3 The calculation process is as follows: First, based on the inverse internal parameter... of The matrix is ​​used to calculate the orientation vector of each pixel in the 3D main scene, and then the pixel-level depth of each frame is calculated. Multiplying by the direction vector yields the 3D coordinates in the camera coordinate system. , is represented as: In the formula: Indicates internal reference, through Pixels The direction of the unit ray mapped to the camera coordinate system; ( , () represents the pixel-level depth of each frame; and They represent direction and Pixels representing the direction; and These represent the coordinates of the principal point in the pixel coordinate system; Then, pixel ( , In three-dimensional world coordinates , is represented as: In the formula: Indicates transpose; The identity matrix represents the extrinsic parameters. .

[0012] Furthermore, the specific process of step 5 is as follows: Step 5.1: Combine the Gaussian kernel and opacity to calculate the... Gaussian particles in a pixel Effective Opacity , is represented as: In the formula: For the first The basic opacity parameter of a Gaussian particle; For the first The mean vector of Gaussian particles; express The covariance matrix of a Gaussian particle is used to define its shape and orientation; The square of the Mahalanobis distance is used to measure the pixel position to the nth... The distance between the centers of Gaussian particles; The Gaussian projection kernel value; Opacity mapping for forward alpha composition; Step 5.2: For each pixel The visible Gaussian pixels are weighted and accumulated in a forward-looking order to obtain the pixel values. Rendering colors , is represented as: In the formula: For pixels The rendered color is either an RGB three-channel vector or a scalar; For projection onto pixels All visible Gaussian quantities, including Gaussian particles in the main 3D scene and fog Gaussian particles; and To iterate through the indices, sort them in forward order; For the first The color of a Gaussian particle; To reach the The cumulative transmittance before a Gaussian particle; For the first Gaussian particles in a pixel Effective opacity.

[0013] Furthermore, both the inference queue in step 2 and the encoding queue in step 6 are set with a capacity limit and a single waiting time limit. When the inference queue or encoding queue reaches the capacity threshold, the first frame batch that enters the queue is discarded. During the process of writing the frame batch into the inference queue or encoding queue, if the single waiting time exceeds the threshold, the frame batch is discarded.

[0014] An augmented reality system for real-time fogging video, comprising augmented reality devices and a server; The augmented reality device includes a camera, an application processor with an integrated video hardware encoding unit, a device-side network interface, a display module, and a memory, wherein: the memory stores a computer program that can be loaded and executed by the application processor to implement the methods described in steps 1 and 7; the camera is used to capture camera images; the video hardware encoding unit is used to encode the captured camera images into an uplink video stream, and then push the uplink video stream to the server by calling the device-side network interface through the RTSP protocol stack; The server includes a processor, an image processing unit integrating a video hardware decoding unit and a video hardware encoding unit, a server-side network interface, and a memory, and the image processing unit is deployed with a single-view... Figure 3 A 3D reconstruction model and a 3DGS volumetric rendering algorithm are provided, wherein: the memory stores a computer program that can be loaded and executed by the processor to implement the methods described in steps 2 to 6; the video hardware decoding unit is used to decode the uplink video stream to generate video frames, and to shape the video frames on the GPU through a filter chain, and the shaped video frames are packaged into frame batches; single-view Figure 3 The 3D reconstruction model performs single-view to 3D Gaussian scatter reconstruction on a batch of frames to generate a 3D main scene represented by 3DGS and pixel-level depth for each frame, and generates foggy Gaussian particles based on the pixel-level depth; the 3DGS volumetric rendering algorithm performs volumetric rendering synthesis of Gaussian scatter and foggy Gaussian particles to generate enhanced frames; the video hardware encoding unit performs low-latency encoding on the enhanced frames to generate downlink enhanced video streams, and then calls the server-side network interface through the RTSP protocol stack to return the downlink enhanced video streams to the memory; the downlink enhanced video streams are decoded in real time and displayed through the display module.

[0015] A computer-readable storage medium storing a computer program that can be loaded and executed by a processor to implement an augmented reality method for real-time fogging video.

[0016] The present invention has the following beneficial technical effects: This invention offers advantages such as low device-side load, low end-to-end latency, and controllable fog level in augmented reality (AR) devices. Specifically: First, the AR device only performs camera image acquisition, encoding, and display; complex single-view reconstruction and volumetric rendering are handled by the server, reducing device power consumption and thermal load, and adapting to a wider range of AR hardware. Second, the server employs a parallel pipeline structure with decoding threads, decoding queues, inference threads, encoding queues, and encoding threads, coupled with an internal link strategy for GPU memory decoding, shaping, reconstruction, rendering, and encoding, reducing copying and serial waiting between the CPU and GPU, thus lowering end-to-end latency. Third, the extrinsic unit matrix and intrinsic parameters are calculated at fixed resolutions, eliminating the need for pose estimation from adjacent frames and multi-camera fusion. This keeps the reconstructed image geometry and imaging parameters within stable and reproducible constraints, facilitating variance and latency control in real-time systems. This allows for precise realization of the near-clear, far-foggy progressive relationship, reduces engineering integration complexity, and facilitates expansion and maintenance. Fourth, single-view... Figure 3 The dimensional reconstruction adopts a frame-by-frame independent strategy, does not track the camera motion trajectory of adjacent frames, reduces state dependence, improves parallelism and robustness, and is naturally decoupled from subsequent fog rendering, which can reduce historical accumulated errors and control the total time of single-batch reconstruction inference within 200 ms, so as to achieve stable output of the target frame rate throughout the entire link.

[0017] This invention sets the low-latency parameters of the video hardware encoding unit to GOP=10~30, B=0, target bitrate=2~8Mbps, and pixel format to YUV420p. This effectively reduces encoding queuing latency and adaptively maintains image quality, exhibiting good bandwidth and congestion robustness. Simultaneously, by setting capacity limits and single-wait time limits for both the inference queue and the encoding queue, under the strategy of discarding the oldest data when fully loaded, latency does not accumulate over time, and the output can be stabilized at the target frame rate of 15 fps. Attached Figure Description

[0018] Figure 1 This is a flowchart of the augmented reality method for real-time fogging video according to the present invention; Figure 2 This is a schematic diagram of the augmented reality system for real-time fogging video according to the present invention; Figure 3 This is a flowchart of the three-dimensional reconstruction and 3DGS volume rendering of the present invention; Figure 4 This is the first set of augmented reality effect images and the corresponding original images of this invention in an extreme weather driving training scenario; Figure 5 This is the second set of augmented reality effect diagrams and the corresponding original images for the present invention in extreme weather driving training scenarios. Detailed Implementation

[0019] The present invention will be further described in detail below with reference to specific embodiments. These descriptions are for explanation purposes only and are not intended to limit the scope of the invention.

[0020] like Figure 1 and Figure 3 As shown, an augmented reality method for real-time fogging video includes the following steps: Step 1, Uplink Acquisition and Streaming: The augmented reality device (AR device) acquires camera footage. The AR device encodes the acquired camera footage according to the H.264 standard using a low-latency preset video hardware encoding unit to generate an uplink video stream. The uplink video stream is then pushed to the server's stream1 address via an RTSP session. Apart from acquisition, encoding, and streaming, the AR device does not perform 3D reconstruction or fogging calculations, thereby minimizing the operational burden and heat generation of the AR device. The augmented reality device presets the acquisition resolution and frame rate according to the server's target shaping requirements, so as to uniformly shape the data into the target input specifications on the server side. The resolution is 640×360 to 1920×1080 and the frame rate is 5~60fps. Preferably, the resolution is 1280×720 and the frame rate is 30 fps; The low-latency preset parameters of the video hardware encoding unit include: GOP=10~30, B=0, CBR or VBR code control, target bitrate=2~8 Mbps, and pixel format yuv420p. Preferably, GOP=15, target bitrate=4 Mbps; Step 2: Decoding, Shaping, and Batch Enqueuing: The server's decoding thread pulls the stream from the stream1 address and calls the video hardware decoding unit integrated in the GPU to decode the uplink video stream, obtaining video frames. Then, a filter chain is used in the GPU memory to uniformly shape the video frames. This filter chain includes GPU scaling, optional cropping, frame rate shaping, and pixel format conversion. First, GPU scaling scales the aspect ratio of the video frames to a value no less than the target resolution. Then, optional cropping crops the center of the video frames to the target resolution. Finally, frame rate shaping... The process involves shaping video frames to the target frame rate, then converting them to the required color format via pixel format conversion to obtain shaped video frames. These shaped video frames are then packaged into frame batches and written to the inference queue in batches. The inference queue has a capacity limit and a single wait time limit. When the inference queue reaches the capacity threshold, the first batch of frames that entered the queue is discarded to reduce latency accumulation and suppress cumulative latency, thus prioritizing the newest frames. During the process of writing frame batches to the inference queue, if the single wait time exceeds the threshold, the batch of frames is discarded, thereby effectively reducing latency. The process of uniformly shaping video frames by the GPU filter chain is completed entirely within the GPU memory, which can prevent copying between the CPU and GPU. If the input protocol or target resolution changes, only the parameters of the filter chain need to be adjusted to adapt. The inference thread always executes with the unified input of the target resolution and target frame rate, thereby maintaining the stability of processing latency. The target resolution is between 640×360 and 1920×1080, and the target frame rate is 5~60 fps. In this embodiment, the target resolution is 1280×720, and the target frame rate is 15 fps, to match... The batch processing rhythm, of which: For frames, that is, for each consecutive A frame is encapsulated into a frame batch; Step 3, Single View Figure 3 Dimensional reconstruction: The server's inference thread retrieves a batch of frames from the inference queue and performs dimensional reconstruction on the external parameters. It is an identity matrix and its intrinsic parameters Based on resolution estimation, call single-view for a batch of frames. Figure 3 The model is reconstructed to generate a 3D main scene represented by 3DGS (3D Gaussian Splatting) and the pixel-level depth of each frame is obtained. Wherein: the internal parameters for A matrix, represented as: In the formula: and They represent direction and Pixels representing the direction; and These represent the coordinates of the principal point in the pixel coordinate system; and These represent the width and height of the target resolution, respectively. These are empirical coefficients used to adjust the field of view. ,when At that time, corresponding to a field of view of 120°, through This ensures that the internal references are stable and reproducible during real-time reconstruction. Then based on the inverse internal reference of The matrix calculates the direction vector of each pixel in the 3D main scene, and the pixel-level depth of each frame is calculated. Multiplying by the direction vector yields the 3D coordinates in the camera coordinate system. , is represented as: In the formula: Indicates internal reference, through Pixels The direction of the unit ray mapped to the camera coordinate system; ( , () represents the pixel-level depth of each frame; and They represent direction and Pixels representing the direction; and These represent the coordinates of the principal point in the pixel coordinate system; Then, pixel ( , In three-dimensional world coordinates , is represented as: In the formula: Indicates transpose; The identity matrix represents the extrinsic parameters. ; Through external parameters Internal Reference and pixel-level depth Solving for 3D world coordinates Without the need to introduce poses from adjacent frames and multi-camera fusion, the inference thread processes each batch. Frame-by-frame reconstruction inference fixes the geometric and imaging parameters of the reconstructed image within stable and reproducible constraints, which is beneficial for controlling variance and latency in real-time systems. Reconstruction adopts a frame-by-frame independent strategy, does not track the camera motion trajectory of adjacent frames, reduces state dependence, and improves parallelism and robustness. At the same time, it is naturally decoupled from subsequent fog rendering, which can reduce historical accumulated errors and control the total time of reconstruction inference in a single batch to within 200 ms, so as to achieve stable output of the target frame rate throughout the entire link. Step 4: Depth-Driven Fog Gaussian Particle Parameter Calculation: Based on the 3D world coordinates obtained in Step 3, generate a fog Gaussian particle for each pixel, represented by a quintuple of parameters. The quintuple parameters include position, scale, opacity, rotation, and color DC, where: 1) Position, for pixels at the target resolution... Construct homogeneous coordinates Seeking , will pixels The unit ray direction is mapped to the camera coordinate system, and then pixels are mapped according to pixel depth. Scale by point to obtain 3D coordinates ,Right now ,Will 1) As the 3D center of the atomized Gaussian particles, thus aligning the 3D center of the atomized Gaussian particles with the camera coordinates; 2) Scale, along , and Three-axis equal scale, varying with pixel-level depth Increasing and truncated within [0.001, 0.2], near-field particles are extremely small to avoid excessive interference with details, while far-field particles approach the upper limit to create a noticeable fog; 3) Opacity increases with pixel-level depth. Incrementing and mapped to [0,1], with near objects approaching 0 and far objects approaching 1, a starting threshold can be set. 0 and saturation depth sat 4) Rotation, represented by a unit quaternion, defaults to (1,0,0,0). Since the particle scale is uniform, rotation's visual contribution is negligible, so the default is no additional rotation; 5) Color DC, represented by spherical harmonic zero-order DC coefficients for uniform haze (e.g., grayish-white, yellowish, bluish). For example, neutral grayish-white haze is represented by spherical harmonic zero-order DC coefficients as (0.5,0.5,0.5) to match... Figure 4 and Figure 5 The image shows a daytime training scenario with dense fog. The depth mapping of scale and opacity can be linear, piecewise linear, or an exponential family of functions. To achieve a near-clear and rapidly emphasizing effect in driver training, the depth mapping of scale and opacity uses a piecewise linear approach: when... < 0 (for example, When 0 = 20 m, the scale and opacity remain extremely small, close to zero; when 0≤ < sat (For example sat When =50m), it increases linearly with a larger slope; when ≥ sat At this point, the parameters remain at their upper limit, resulting in particle scale saturation and opacity close to 1. This ensures clear visibility for the driver within 20 meters, transitions from light fog to dense fog in the 20-50 meter range, and beyond 50 meters, distant views are essentially obscured by the fog curtain. Figure 4 and Figure 5 The daytime dense fog is consistent with the illustration; Step 5: 3DGS Volumetric Rendering and Enhanced Frame Composition: The server's inference thread inputs the Gaussian scatter points in the 3D scene represented by 3DGS and the fogged Gaussian particles generated in Step 4 into the 3DGS volumetric rendering algorithm. Volumetric rendering and forward compositing are performed according to the pixel ray direction to obtain the rendered color. This refers to enhanced frames, and the specific process is as follows: Step 5.1: Combine the Gaussian kernel and opacity to calculate the... Gaussian particles in a pixel Effective Opacity That is, pixels modulated by a projected Gaussian kernel Transparency of contributions Represented as: In the formula: For the first The basic opacity parameter of a Gaussian particle; For the first The mean vector of a Gaussian particle, i.e., the center position; express The covariance matrix of a Gaussian particle is used to define its shape and orientation; The square of the Mahalanobis distance is used to measure the pixel position to the nth... The distance between the centers of Gaussian particles takes into account the shape and orientation of the Gaussian particles; It is a Gaussian projection kernel value with attenuated kernel amplitude; Forward alpha composition, the opacity map is used to combine the Gaussian kernel and opacity; Step 5.2: During rendering, for each pixel... The visible Gaussian pixels are weighted and accumulated in a forward-looking order to obtain the pixel values. Rendering colors That is, pixel pixel Rendering colors It is a weighted superposition of the colors of all Gaussian particles at that pixel location, with the weights of the superposition determined by... The cumulative transmittance is obtained by multiplying by the preceding Gaussian, where: pixel Rendering colors Represented as: In the formula: For pixels The rendered color is either an RGB three-channel vector or a scalar; For projection onto pixels All visible Gaussian quantities, including Gaussian particles in the main 3D scene and fog Gaussian particles; and To traverse the index, sort it in forward order, that is, sort it from closest to furthest from the camera; For the first The color (RGB or grayscale) of each Gaussian particle; To reach the The cumulative transmittance before the nth Gaussian particle, i.e., the light from the observer to the current nth Gaussian particle. The proportion of unabsorbed Gaussian particles among the individual particles; For the first Gaussian particles in a pixel Effective opacity; The contribution of fogged Gaussian particles is greater in the distance: on the one hand, the opacity of fog particles themselves increases with depth; on the other hand, the energy distribution (kernel amplitude) of the Gaussian particles in the distant main scene on the projection plane is weaker and more easily masked by the superposition of fog particles. Therefore, the enhancement result presents a progressive visual hierarchy of near clarity and far fog. In order to avoid numerical instability, the contribution of each pixel is truncated by threshold (ignoring excessively low kernel values) and ordered (only the first few significant Gaussian particles are synthesized). Step 6, Encoding and Downlink Push: The server's inference thread completes each batch. After the frames are synthesized, the enhanced frames are written to the encoding queue. The encoding queue has a capacity limit and a single wait time limit. When the encoding queue reaches the capacity threshold, the first batch of frames that entered the queue is discarded to reduce latency accumulation. Alternatively, during the process of writing enhanced frames to the encoding queue, if the single wait time exceeds the threshold, the batch of frames is discarded. The server's encoding thread is on the GPU side. According to the H.264 standard, the enhanced frames are encoded with low latency by the video hardware encoding unit to generate a downlink enhanced video stream. The downlink enhanced video stream is pushed to the server's stream2 address through an RTSP session. The low latency parameters include GOP=10~30, B=0, target bitrate BR=2~8 Mbps, pixel format yuv420p and frame rate 20~60 fps. Preferably, the low latency parameters in this embodiment include GOP=15, B=0, target bitrate BR=4 Mbps, pixel format yuv420p and frame rate of 15 fps; Step 7: Augmented Reality Stream Display: The augmented reality device pulls the augmented video stream from the stream2 address, decodes it in real time, and displays it in the field of view for the user to observe. Taking a driving training scenario as an example: the user can see that the road markings, vehicle edges, and traffic signs in front are clearly readable; beyond... 0's field of vision was quickly covered by fog, and satAlmost invisible beyond the field of view, this visual experience does not alter the geometry and lighting of real-world images; it achieves the same effect simply by applying depth-consistent fog enhancement to the video image. Figure 4 and Figure 5 The above shows the requirements for driving in dense fog during the day.

[0021] like Figure 2 As shown, an augmented reality system for real-time fogging video includes an augmented reality device and a server. The augmented reality device includes a camera, an application processor with an integrated video hardware encoding unit, a device-side network interface, a display module, and a memory. The memory stores a computer program that can be loaded and executed by the application processor to implement the uplink acquisition and streaming method in step 1. The camera is used to acquire camera images. The video hardware encoding unit encodes the acquired camera images into an uplink video stream. The RTSP protocol stack uploads the uplink video stream to the server by calling the device-side network interface. The server includes a processor (CPU), an image processing unit (GPU) integrating a video hardware decoding unit and a video hardware encoding unit, a server-side network interface, and a memory. The memory stores a computer program that can be loaded and executed by the CPU to implement the augmented reality method in steps 2 to 6. Specifically: the video hardware decoding unit decodes the uplink video stream to generate video frames; a filter chain is used to shape the video frames on the GPU; the shaped video frames are packaged into frame batches and written to the inference queue; and under the condition that external parameters participate in the fixed computation, the single-view array deployed on the GPU... Figure 3 The 3D reconstruction model performs single-view to 3D Gaussian scatter reconstruction on a batch of frames to generate a 3D main scene represented by 3DGS and obtains the pixel-level depth of each frame. Based on the pixel-level depth, foggy Gaussian particles are generated. The 3DGS volumetric rendering algorithm deployed on the GPU performs volumetric rendering and synthesis of the Gaussian scatter and foggy Gaussian particles to generate enhanced frames. The video hardware encoding unit performs low-latency encoding on the enhanced frames to generate downlink enhanced video streams. The RTSP protocol stack returns the downlink enhanced video streams to memory by calling the server-side network interface. The downlink enhanced video streams are decoded in real time and displayed through the display module.

[0022] A computer-readable storage medium storing a computer program that can be loaded and executed by a processor to implement a real-time fogging video augmented reality method for steps 1 to 7.

[0023] The key technical features of the real-time fogging video augmented reality method proposed in this embodiment are as follows: First, single-view to 3DGS reconstruction constraints: the extrinsic parameters are fixed as the identity matrix, and the intrinsic parameters are estimated only by resolution, avoiding the latency and uncertainty introduced by camera pose estimation; moreover, each frame is reconstructed independently, and frames within a batch only share computing resources, not states, thereby reducing stuttering and drift caused by cross-frame dependencies; Second, pixel ray extension from depth to three-dimensional coordinates: for each pixel Construct homogeneous pixel vectors The pixels are converted using the inverse matrix of the intrinsic parameters. The unit ray direction mapped to the camera coordinate system is then scaled by the pixel depth Z to obtain the corresponding 3D world coordinates; third, 3DGS volume rendering: a 3D Gaussian kernel is formed by projecting it onto the pixel plane to create a 2D Gaussian kernel. The effective opacity is calculated based on the kernel's contribution to the pixel and the particle's basic opacity parameter, and the pixel color is accumulated using the forward alpha compounding rule; fourth, fog mapping family: a piecewise linear approach is used to achieve a strong sense of layering, from near-no fog to rapid increase in the middle and far distances to saturation in the far distance. In other scenes, it can also be switched to linear or exponential family functions, but regardless of the function, it must adhere to the following: scale ≤ 0.2, opacity [0,1], and settings. 0 and sat The parameter constraints improve rendering stability and visual continuity; fifth, the trade-off between batch size N and latency: Increasing N can improve GPU throughput and reduce the amortized overhead per unit frame within a batch, but it increases the waiting time per batch; if N is too small, the pipeline parallelism is insufficient. Therefore, the target resolution and target frame rate are set to 1280×720 and 15fps, respectively. =3, on most general-purpose GPUs, the single-batch turnaround time of "decoding-reconstruction-rendering-encoding" can be controlled to around 0.20 s, thereby achieving a steady output of 15fps.

[0024] Without changing the single-path link, external and internal parameter settings, and single-view configuration Figure 3 By incorporating the core technologies of DGS reconstruction, depth-driven fogging, three-thread dual-queue parallelism, and the principle of discarding the oldest data at full load, this approach can adapt to other resolution and frame rate combinations within the same framework. It can also replace H.264 with an equivalent hardware acceleration standard at the codec layer, thereby reducing the bitrate and improving image quality under the same latency budget. As can be seen, the augmented reality technology in this embodiment provides an engineering-reproducible end-cloud collaborative solution under strict real-time constraints. It can stably output near-clear to far-foggy enhanced video in tasks that emphasize visibility control, including driver training, and has good robustness to network fluctuations.

Claims

1. An augmented reality method for real-time fogging video, characterized in that, Includes the following steps: Step 1: The augmented reality device captures camera footage. The augmented reality device encodes the captured camera footage according to the H.264 standard using a low-latency preset video hardware encoding unit to generate an uplink video stream. The uplink video stream is pushed to the server's stream1 address via an RTSP session. Step 2: The server's decoding thread pulls the stream from the stream1 address, calls the video hardware decoding unit integrated in the GPU to decode the uplink video stream, obtains video frames, and then uses a filter chain in the GPU memory to uniformly shape the video frames to obtain shaped video frames. Then, the shaped video frames are packaged into frame batches and written to the inference queue in units of frame batches. Step 3: The server's inference thread retrieves a batch of frames from the inference queue and processes them in the external parameters. It is an identity matrix and its intrinsic parameters Based on resolution estimation, the single-view 3D reconstruction model is called in batches of frames to generate a 3D main scene represented by 3DGS and obtain the pixel-level depth of each frame. Then, based on the inverse intrinsic parameter matrix, the direction vector of each pixel in the 3D main scene is calculated, and the pixel-level depth of each frame is calculated. Multiplying by the direction vector yields the 3D coordinates in the camera coordinate system. Finally, through external parameters and three-dimensional coordinates Calculate the three-dimensional world coordinates of the pixel in three-dimensional space. ; Step 4: Based on the 3D world coordinates obtained in Step 3 Generate a fogged Gaussian particle for each pixel, represented by a quintuple of parameters; Step 5: The server's inference thread inputs the Gaussian scatter points in the 3D scene represented by 3DGS and the fogged Gaussian particles generated in Step 4 into the 3DGS volumetric rendering algorithm, and performs volumetric rendering and forward compositing according to the pixel ray direction to obtain the enhanced frame. Step 6: The server writes the enhanced frame into the encoding queue. The server's encoding thread performs low-latency encoding on the enhanced frame on the GPU side according to the H.264 standard through the video hardware encoding unit to generate a downlink enhanced video stream. The downlink enhanced video stream is pushed to the server's stream2 address through an RTSP session. Step 7: The augmented reality device pulls the augmented video stream from the stream2 address, decodes it in real time, and displays it for the user to observe.

2. The augmented reality method for real-time fogging video according to claim 1, characterized in that, The low latency parameters of the video hardware encoding units in steps 1 and 6 are: GOP=10~30, B=0, target bitrate=2~8 Mbps, fps=5~60, and pixel format is yuv420p.

3. The augmented reality method for real-time fogging video according to claim 1, characterized in that, The GPU filter chain in step 2 includes GPU scaling, optional cropping, frame rate shaping, and pixel format conversion, wherein: GPU scaling is used to scale the aspect ratio of the video frame to a value not less than the target resolution; optional cropping is used to crop the center of the GPU-scaled video frame to the target resolution; frame rate shaping is used to shape the video frame cropped to the target resolution to the target frame rate; and pixel format conversion is used to convert the video frame to the required color format.

4. The augmented reality method for real-time fogging video according to claim 3, characterized in that, The target resolution and target frame rate are 1280×720 and 15 fps, respectively, and every three consecutive video frames are encapsulated into a frame batch.

5. The augmented reality method for real-time fogging video according to claim 1, characterized in that, The internal parameters of step 3 for A matrix, represented as: In the formula: and They represent direction and Pixels representing the direction; and These represent the coordinates of the principal point in the pixel coordinate system; and These represent the width and height of the target resolution, respectively. These are empirical coefficients used to adjust the field of view. .

6. The augmented reality method for real-time fogging video according to claim 5, characterized in that, The three-dimensional world coordinates in step 3 The calculation process is as follows: First, based on the inverse internal parameter... of The matrix is ​​used to calculate the orientation vector of each pixel in the 3D main scene, and then the pixel-level depth of each frame is calculated. Multiplying by the direction vector yields the 3D coordinates in the camera coordinate system. , is represented as: In the formula: Indicates internal reference, through Pixels The direction of the unit ray mapped to the camera coordinate system; ( , () represents the pixel-level depth of each frame; and They represent direction and Pixels representing the direction; and These represent the coordinates of the principal point in the pixel coordinate system; Then, pixel ( , In three-dimensional world coordinates , is represented as: In the formula: Indicates transpose; The identity matrix represents the extrinsic parameters. .

7. The augmented reality method for real-time fogging video according to claim 1, characterized in that, The specific process of step 5 is as follows: Step 5.1: Combine the Gaussian kernel and opacity to calculate the... Gaussian particles in a pixel Effective Opacity , is represented as: In the formula: For the first The basic opacity parameter of a Gaussian particle; For the first The mean vector of Gaussian particles; express The covariance matrix of a Gaussian particle is used to define its shape and orientation; The square of the Mahalanobis distance is used to measure the pixel position to the nth... The distance between the centers of Gaussian particles; The Gaussian projection kernel value; Opacity mapping for forward alpha composition; Step 5.2, for each pixel The visible Gaussian pixels are weighted and accumulated in a forward-looking order to obtain the pixel values. Rendering colors , is represented as: In the formula: For pixels The rendered color is either an RGB three-channel vector or a scalar; For projection onto pixels All visible Gaussian quantities, including Gaussian particles in the main 3D scene and fog Gaussian particles; and To iterate through the indices, sort them in forward order; For the first The color of a Gaussian particle; To reach the The cumulative transmittance before a Gaussian particle; For the first Gaussian particles in a pixel Effective opacity.

8. The augmented reality method for real-time fogging video according to claim 1, characterized in that, Both the inference queue in step 2 and the encoding queue in step 6 are set with a capacity limit and a single waiting time limit. When the inference queue or encoding queue reaches the capacity threshold, the first frame batch that enters the queue is discarded. During the process of writing the frame batch into the inference queue or encoding queue, if the single waiting time exceeds the threshold, the frame batch is discarded.

9. An augmented reality system for implementing the method of any one of claims 1 to 8 for real-time fogged video, characterized in that, Including augmented reality devices and servers; The augmented reality device includes a camera, an application processor with an integrated video hardware encoding unit, a device-side network interface, a display module, and a memory, wherein: the memory stores a computer program that can be loaded and executed by the application processor to implement the methods described in steps 1 and 7 of claims 1 to 8; the camera is used to capture camera images; the video hardware encoding unit is used to encode the captured camera images into an uplink video stream, and then push the uplink video stream to the server by calling the device-side network interface through the RTSP protocol stack; The server includes a processor, an image processing unit integrating a video hardware decoding unit and a video hardware encoding unit, a server-side network interface, and a memory. The image processing unit is equipped with a single-view 3D reconstruction model and a 3DGS volumetric rendering algorithm. The memory stores a computer program that can be loaded and executed by the processor to implement the methods described in steps 2 to 6 of claims 1 to 8. The video hardware decoding unit decodes the uplink video stream to generate video frames and shapes the video frames on the GPU using a filter chain. The shaped video frames are then packaged into a batch. The single-view 3D reconstruction model performs single-view to 3D Gaussian scatter point reconstruction on the batch of frames to generate a 3D main scene represented by 3DGS and pixel-level depth for each frame, and generates foggy Gaussian particles based on the pixel-level depth. The 3DGS volumetric rendering algorithm performs volumetric rendering synthesis of the Gaussian scatter points and foggy Gaussian particles to generate enhanced frames. The video hardware encoding unit performs low-latency encoding on the enhanced frames to generate a downlink enhanced video stream. The downlink enhanced video stream is then returned to the memory via the server-side network interface through the RTSP protocol stack. The downlink enhanced video stream is decoded in real time and displayed through a display module.

10. A computer-readable storage medium, characterized in that, A computer program is stored that can be loaded and executed by a processor to implement an augmented reality method for real-time fogging video as described in any one of claims 1 to 8.

Images