AI large model virtual-real fusion video stream real-time rendering method for online exhibition

Abstract

The present application relates to the field of online exhibition and augmented reality fusion technology, and more particularly to an AI large model virtual-real fusion video stream real-time rendering method for online exhibition, which comprises: pre-rendering an exhibition three-dimensional model on the server side, generating a pre-rendered video stream containing a depth map and an occlusion map, and extracting key frame metadata containing camera extrinsic parameters and rendering quality scores; the client collects real scene video stream and detects markers to obtain pose estimation information, and monitors network state parameters; three judgment indexes are calculated according to the rendering quality score, the pose accuracy index and the network adaptability index, and a comprehensive judgment index is calculated based on the three to determine whether the rendering mode is qualified, and when it is not qualified, the reason category is determined according to the deviation direction between each judgment index and the corresponding adjustment strategy is executed. Through the cooperation of pre-rendering and pose estimation and multi-dimensional layered adjustment, dynamic closed-loop control of virtual-real fusion rendering quality is realized.

Description

Technical Field

[0001] This invention relates to the field of online exhibition and augmented reality fusion technology, and in particular to a real-time rendering method for AI large-scale model virtual-real fusion video streams used in online exhibitions. Background Technology

[0002] Traditional online exhibitions primarily rely on 2D images, text descriptions, or pre-recorded videos for display, resulting in poor immersion, weak interactivity, and low integration of virtual exhibits with real-world scenes. With the development of augmented reality (AR) technology, AR-based online exhibitions are gradually becoming a trend, but existing technologies still have the following shortcomings: First, rendering quality is limited; traditional methods rely on local rendering on the client side, and differences in terminal performance make it difficult to smoothly present high-quality 3D virtual exhibits on ordinary devices. Second, network adaptability is poor; fixed bitrate transmission cannot adapt to complex and changing network environments, and bandwidth fluctuations easily lead to screen stuttering or blurring. Third, the integration of virtual and real elements is inconsistent; virtual exhibits lack accurate occlusion relationships with real-world scenes, resulting in visual effects that appear floating or embedded. Fourth, pose estimation accuracy is low; traditional visual methods lack robustness to changes in lighting and marker occlusion, causing virtual content to fail to accurately anchor to real-world booths. Fifth, latency is too high; end-to-end rendering latency exceeds 100ms, easily causing visual dizziness and operational lag. Sixth, resource consumption is high; high-quality rendering requires high-end GPU equipment, which ordinary mobile phones or browsers cannot handle.

[0003] Chinese Patent Publication No. CN121888056A discloses a system and method for producing live-streaming content for cultural tourism that supports real-time virtual-real fusion and interaction, belonging to the field of virtual-real fusion interactive live-streaming technology. The system includes: a video acquisition and spatial positioning module that calculates the pose data of the acquisition device in physical space in real time; an interaction processing module that receives and identifies interaction commands from at least one end; a virtual asset storage module whose virtual assets include 3D models, animation elements, and UI interaction elements; a real-time rendering and compositing module that, based on the pose data and interaction commands, calls the corresponding virtual assets and performs adaptive rendering, then accurately generates a fused video stream by combining the rendered virtual assets with the real-scene video stream; and a live-streaming output and interaction feedback module that outputs the fused video stream to the viewer's end and sends the interaction commands from the viewer's end back to the interaction processing module. This system solves the problems of inaccurate virtual-real fusion, superficial interaction methods, and high barriers to high-quality content production in existing outdoor cultural tourism live-streaming.

[0004] Therefore, the existing technology has the following problems: Lacking the ability to dynamically perceive and hierarchically adjust to network state fluctuations and changes in client pose estimation accuracy, it is difficult to adaptively maintain the stability and smoothness of virtual-real fusion when bandwidth changes or pose drift occurs. Summary of the Invention

[0005] To address this, the present invention provides a real-time rendering method for virtual-real fusion video streams of large AI models used in online exhibitions. This method overcomes the problem in existing technologies that lack dynamic perception and hierarchical adjustment capabilities for network state fluctuations and changes in client pose estimation accuracy, making it difficult to adaptively maintain the stability and smoothness of virtual-real fusion when bandwidth changes or pose drift occurs.

[0006] To achieve the above objectives, this invention provides a real-time rendering method for AI large-scale model virtual-real fusion video streams used in online exhibitions, comprising: Step S1: The server pre-renders the 3D model of the exhibition, generates a pre-rendered video stream containing a depth map and an occlusion map, extracts key frame metadata from the pre-rendered video stream, and generates pre-rendered analysis results based on the key frame metadata. The key frame metadata includes camera extrinsic parameters and rendering quality scores. Step S2: The client receives the pre-rendered video stream and the pre-rendered analysis result, collects the real-world scene video stream and detects markers to obtain the client's pose estimation information relative to the markers. At the same time, the client monitors network state parameters, calculates the pose accuracy index based on the pose estimation information, and calculates the network adaptability index based on the network state parameters. Step S3: Extract the rendering quality score from the pre-rendering analysis results as the rendering quality judgment index; calculate the pose accuracy judgment index based on the pose accuracy index; calculate the network adaptability judgment index based on the network adaptability index; calculate the comprehensive judgment index based on the rendering quality judgment index, the pose accuracy judgment index, and the network adaptability judgment index; determine whether the current rendering mode is qualified based on the comprehensive judgment index; and determine the category of the reason for the failure based on the deviation direction between each judgment index if it is unqualified. Step S4: Execute the corresponding adjustment strategy according to the category of non-compliance reasons, and output the result after completing the virtual-real fusion based on the depth map after executing the adjustment strategy.

[0007] Further, in step S1, a pre-rendered video stream containing a depth map and an occlusion map is generated, and keyframe metadata is extracted, including: Step S11: The server loads the 3D model of the exhibition, sets multiple preset viewpoints and multiple preset lighting conditions, renders the 3D model under each preset viewpoint and each preset lighting condition, generates a pre-rendered color image corresponding to each viewpoint, extracts the depth value of each pixel in the pre-rendered color image during the rendering process, generates a depth map based on the extracted depth value, and determines the front and back occlusion relationship of each pixel based on the depth value of each pixel in the depth map, and generates an occlusion map. Step S12: Encode the pre-rendered color image, the depth map, and the occlusion map from the same viewpoint into a pre-rendered video stream in chronological order; Step S13: Extract metadata from each keyframe of the pre-rendered video stream. The metadata includes at least camera extrinsic parameters and rendering quality scores. Associate the metadata with the corresponding keyframes and store them to form a pre-rendered analysis result.

[0008] Further, in step S2, obtaining the pose estimation information of the client relative to the marker and calculating the pose accuracy index includes: Step S211: The client calls the camera to capture a video stream of the real scene, extracts the current frame image from the video stream, detects whether there is a preset marker in the current frame image, and if there is, extracts the corner coordinates and encoding information of the marker. The marker is a QR code or positioning mark that is pre-arranged at the exhibition site. Step S212: Obtain the three-dimensional coordinates of the marker in the world coordinate system. Based on the corner coordinates and the three-dimensional coordinates of the marker, solve for the initial rotation matrix and initial translation vector of the client camera relative to the marker, as initial pose estimation information. Step S213: Compare the initial pose estimation information with the camera extrinsic parameters in the keyframe metadata, calculate the rotation deviation between the initial rotation matrix and the rotation matrix in the camera extrinsic parameters, and the translation deviation between the initial translation vector and the translation vector in the camera extrinsic parameters, and use the rotation deviation and the translation deviation as the pose accuracy index.

[0009] Further, in step S2, network state parameters are monitored and network adaptability indices are calculated, including: Step S221: The client periodically measures the available bandwidth of the current network and records the bandwidth fluctuation amplitude during the measurement process. Step S222: The client sends probe data packets to the server and receives a response, measuring the current network round-trip time and packet loss rate; Step S223: Normalize the available bandwidth, the bandwidth fluctuation range, the round-trip time and the packet loss rate respectively, and take the weighted average value as the network adaptability index.

[0010] Further, in step S3, calculating the pose accuracy judgment index based on the pose accuracy index and calculating the network adaptability judgment index based on the network adaptability index includes: Obtain a preset allowable value for rotation deviation and allowable value for translation deviation. Divide the rotation deviation by the allowable value for rotation deviation to obtain a normalized rotation deviation. Divide the translation deviation by the allowable value for translation deviation to obtain a normalized translation deviation. Take the arithmetic mean of the normalized rotation deviation and the normalized translation deviation as the pose accuracy judgment index. Obtain the pre-set lower limit of available bandwidth, allowable bandwidth fluctuation range, upper limit of round-trip latency, and upper limit of packet loss rate. Divide the lower limit of available bandwidth by the available bandwidth to obtain a bandwidth adequacy index; divide the bandwidth fluctuation range by the allowable bandwidth fluctuation range to obtain a normalized bandwidth fluctuation index; divide the round-trip latency by the upper limit of round-trip latency to obtain a normalized latency index; divide the packet loss rate by the upper limit of packet loss rate to obtain a normalized packet loss index; and take the arithmetic mean of the bandwidth adequacy index, the normalized bandwidth fluctuation index, the normalized latency index, and the normalized packet loss index as the network adaptability judgment index.

[0011] Further, in step S3, a comprehensive judgment index is calculated based on the rendering quality judgment index, the pose accuracy judgment index, and the network adaptability judgment index, and the current rendering mode is determined to be qualified based on the comprehensive judgment index, including: Obtain a preset judgment threshold. When the rendering quality judgment index is less than or equal to the judgment threshold, the pose accuracy judgment index is less than or equal to the judgment threshold, and the network adaptability judgment index is less than or equal to the judgment threshold, the current rendering mode is determined to be qualified. When the rendering quality judgment index is greater than the judgment threshold, or the pose accuracy judgment index is greater than the judgment threshold, or the network adaptability judgment index is greater than the judgment threshold, the current rendering mode is determined to be unqualified.

[0012] Furthermore, in step S3, when the rendering mode is unqualified, the category of the unqualified reason is determined based on the deviation direction between each judgment index, including: When the current rendering mode is determined to be unqualified, calculate the first ratio of the network adaptability determination index to the pose accuracy determination index, and the second ratio of the pose accuracy determination index to the rendering quality determination index. When the first ratio is greater than or equal to the first preset ratio, the failure reason category is determined to be a network problem; when the first ratio is less than or equal to the second preset ratio, and the second ratio is greater than or equal to the third preset ratio, the failure reason category is determined to be a pose estimation problem; when the first ratio is between the second preset ratio and the first preset ratio, and the second ratio is between the fourth preset ratio and the third preset ratio, the failure reason category is determined to be a comprehensive problem.

[0013] Further, in step S4, the corresponding adjustment strategy is executed according to the category of non-compliance reasons, including: When the failure reason category is determined to be a network problem, the transmission bit rate is adjusted according to the available bandwidth, and the resolution or frame rate of the video stream is reduced so that the network adaptability judgment index returns to less than or equal to the judgment threshold. When the cause of non-compliance is determined to be a pose estimation problem, Kalman filtering is activated to predict and correct the initial pose estimation information. The filtering parameters are dynamically adjusted according to the rotation deviation and the translation deviation so that the pose accuracy judgment index returns to less than or equal to the judgment threshold. When the failure reason category is determined to be a comprehensive problem, bit rate adjustment and pose filtering correction are performed simultaneously, and asynchronous time distortion compensation is initiated so that each component of the comprehensive judgment index returns to less than or equal to the judgment threshold.

[0014] Furthermore, after executing the adjustment strategy and performing virtual-real fusion based on the depth map, the output includes: The client obtains the current frame pre-rendered color image, the depth map, and the occlusion map from the pre-rendered video stream, and projects the three-dimensional model of the virtual exhibit onto the current frame image coordinate system according to the current pose to obtain the two-dimensional projection area of ​​the virtual exhibit and the virtual depth value corresponding to each pixel; For each pixel within the two-dimensional projection area, the virtual depth value is compared with the real depth value at the corresponding position in the depth map. When the virtual depth value is less than the real depth value, the virtual exhibit is determined to be visible, and the color of the virtual exhibit is used as the fusion output color of the pixel. When the virtual depth value is greater than or equal to the real depth value, the virtual exhibit is determined to be occluded, and the color of the real scene is used as the fusion output color of the pixel. The merged output image is accelerated and rendered, then encoded into a low-latency video stream and pushed to the client's display screen.

[0015] Furthermore, the activation condition for the asynchronous time distortion compensation is as follows: when the rendering quality judgment index is greater than the judgment threshold, the time difference between the rendering completion time of the current frame and the planned display time is calculated. Based on the time difference and the angle change of the client camera between the current time and the rendering time, the pixel displacement compensation amount is calculated. The pixel displacement compensation amount is applied to the blended image that has been rendered but not yet displayed, and a compensated image with a smooth view transition is generated and then output.

[0016] Compared with existing technologies, the advantages of this invention lie in that it transfers the rendering calculation of high-quality 3D virtual exhibits from the client to the server by pre-rendering 3D models under multi-view and multi-lighting conditions on the server side, combined with the synchronous extraction and encoding of depth maps and occlusion maps. This overcomes the technical problem of traditional methods relying on local rendering on the client side, which leads to differences in terminal performance affecting the presentation quality. This allows ordinary terminal devices to obtain high-quality visual presentations of 3D exhibits. Simultaneously, the generation of depth maps and occlusion maps during the pre-rendering process provides an accurate data foundation for determining occlusion relationships in subsequent virtual-real fusion, providing technical assurance for the correct spatial fusion of virtual exhibits and real-world scenes.

[0017] Furthermore, this invention achieves two key input parameters required for virtual-real fusion by parallel processing of real-time pose estimation on the client side and network status monitoring. On the one hand, by solving the six-degree-of-freedom pose of the client's camera and comparing the estimation results with the pre-rendered camera extrinsic parameters on the server side, rotational deviation and translational deviation are used as pose accuracy indicators. This enables the system to quantitatively evaluate the degree of deviation of the current pose estimation from the pre-rendered standard pose, providing a clear trigger basis and data benchmark for subsequent pose filtering correction. On the other hand, by periodically measuring available bandwidth, bandwidth fluctuation amplitude, round-trip latency, and packet loss rate, multi-dimensional network status parameters are normalized and weighted into a single network adaptability indicator. This enables the system to comprehensively evaluate the current network's support capability for real-time video stream transmission, providing a quantitative decision basis for subsequent adaptive bitrate adjustment.

[0018] Furthermore, this invention constructs three independent judgment indicators—rendering quality, pose accuracy, and network adaptability—and uses a comprehensive judgment indicator based on these three indicators to determine the pass / fail status of the current rendering mode, achieving a multi-dimensional comprehensive evaluation of the virtual-real fusion rendering quality. When any judgment indicator exceeds its limit, a failure judgment is triggered. Then, a first ratio and a second ratio are calculated based on the deviation direction between the judgment indicators. By comparing the ratio with a preset threshold, the cause of failure is accurately attributed to network problems, pose estimation problems, or a combination of problems. This hierarchical judgment mechanism overcomes the technical problems of traditional methods, such as the difficulty of comprehensively evaluating virtual-real fusion quality with a single indicator and the reliance on human experience and strong subjectivity in cause judgment. It enables the system to automatically identify the dominant dimension of quality degradation, providing a precise attribution basis for subsequent differentiated adjustment strategies, and improving the accuracy of problem diagnosis and the pertinence of adjustment strategies.

[0019] Furthermore, this invention achieves precise closed-loop control of virtual-real fusion rendering quality by implementing differentiated adjustment strategies for three types of non-compliance causes: network problems, pose estimation problems, and comprehensive problems. When a network problem is identified, the resolution and frame rate are dynamically adjusted based on the ratio of available bandwidth to the current transmission bitrate, bringing the network adaptability assessment index back to the normal range and overcoming the technical problem that fixed bitrate transmission cannot adapt to bandwidth fluctuations. When a pose estimation problem is identified, the initial rotation matrix and translation vector are used as observations, and the pose state variables are predicted and corrected using Kalman filtering. The filter noise covariance matrix is ​​then adjusted inversely based on the magnitude of the rotation and translation deviations, allowing the filter parameters to adaptively match the current pose drift level. This achieves dynamic and precise correction of pose estimation, overcoming the problem of insufficient adaptability of traditional fixed-parameter filtering when the drift level changes. Attached Figure Description

[0020] Figure 1 This is a flowchart of the real-time rendering method for AI large-scale model virtual-real fusion video stream used in this embodiment for online exhibitions; Figure 2 This is a flowchart illustrating the process of generating a pre-rendered video stream in the real-time rendering method of AI large-scale model virtual-real fusion video stream for online exhibitions in this embodiment. Detailed Implementation

[0021] To make the objectives and advantages of the present invention clearer, the present invention will be further described below with reference to embodiments; it should be understood that the specific embodiments described herein are merely for explaining the present invention and are not intended to limit the present invention.

[0022] Preferred embodiments of the present invention will now be described with reference to the accompanying drawings. Those skilled in the art should understand that these embodiments are merely illustrative of the technical principles of the present invention and are not intended to limit the scope of protection of the present invention.

[0023] Please see Figures 1-2 As shown, Figure 1 This is a flowchart of the real-time rendering method for AI large-scale model virtual-real fusion video stream used in this embodiment for online exhibitions; Figure 2 This is a flowchart illustrating the process of generating a pre-rendered video stream in the real-time rendering method of AI large-scale model virtual-real fusion video stream for online exhibitions in this embodiment.

[0024] This embodiment provides a method for real-time rendering of a virtual-real fusion video stream using a large AI model for online exhibitions, including: Step S1: The server pre-renders the 3D model of the exhibition, generates a pre-rendered video stream containing a depth map and an occlusion map, extracts key frame metadata from the pre-rendered video stream, and generates pre-rendered analysis results based on the key frame metadata. The key frame metadata includes camera extrinsic parameters and rendering quality scores. Step S2: The client receives the pre-rendered video stream and the pre-rendered analysis result, collects the real-world scene video stream and detects markers to obtain the client's pose estimation information relative to the markers. At the same time, the client monitors network state parameters, calculates the pose accuracy index based on the pose estimation information, and calculates the network adaptability index based on the network state parameters. Step S3: Extract the rendering quality score from the pre-rendering analysis results as the rendering quality judgment index; calculate the pose accuracy judgment index based on the pose accuracy index; calculate the network adaptability judgment index based on the network adaptability index; calculate the comprehensive judgment index based on the rendering quality judgment index, the pose accuracy judgment index, and the network adaptability judgment index; determine whether the current rendering mode is qualified based on the comprehensive judgment index; and determine the category of the reason for the failure based on the deviation direction between each judgment index if it is unqualified. Step S4: Execute the corresponding adjustment strategy according to the category of non-compliance reasons, and output the result after completing the virtual-real fusion based on the depth map after executing the adjustment strategy.

[0025] Specifically, in step S1, a pre-rendered video stream containing a depth map and an occlusion map is generated, and keyframe metadata is extracted, including: Step S11: The server loads the 3D model of the exhibition, sets multiple preset viewpoints and multiple preset lighting conditions, renders the 3D model under each preset viewpoint and each preset lighting condition, generates a pre-rendered color image corresponding to each viewpoint, extracts the depth value of each pixel in the pre-rendered color image during the rendering process, generates a depth map based on the extracted depth value, and determines the front and back occlusion relationship of each pixel based on the depth value of each pixel in the depth map, and generates an occlusion map. Step S12: Encode the pre-rendered color image, the depth map, and the occlusion map from the same viewpoint into a pre-rendered video stream in chronological order; Step S13: Extract metadata from each keyframe of the pre-rendered video stream. The metadata includes at least camera extrinsic parameters and rendering quality scores. Associate the metadata with the corresponding keyframes and store them to form a pre-rendered analysis result.

[0026] In this embodiment of the invention, the preset viewing angle is determined based on the distribution location of the booths in the exhibition hall and the preset audience viewing path, covering the frontal view, side view, and top view of each booth; the preset lighting conditions are set according to the lighting environment of a standard exhibition hall, including uniform light environment, focused light environment for key exhibits, and simulated natural light environment, to ensure that the pre-rendered color image can cover the visual presentation requirements of different scenes. The depth map has the same pixel resolution as the pre-rendered color image, and the depth value of each pixel represents the distance of the corresponding 3D model surface point from the virtual camera. The occlusion map is generated as follows: for each pixel in the depth map, the depth value of the pixel is compared with the depth values ​​of its neighboring pixels. If the depth value of the pixel is less than the depth value of its neighboring pixels, the area corresponding to the pixel is determined to be in the foreground; otherwise, it is determined to be in the background. The determination results of all pixels are summarized to form the occlusion map. The pre-rendered color image, the depth map, and the occlusion map together constitute a keyframe data unit. Multiple consecutive keyframe data units under the same viewing angle are arranged in the order of rendering time and encoded as a pre-rendered video stream in H.265 format. The camera extrinsic parameters are the position and orientation parameters of the virtual camera in the 3D model space corresponding to each keyframe, including rotation matrices and translation vectors. The rendering quality score is comprehensively evaluated based on the light intensity value, color temperature value, and resolution and texture detail richness of the pre-rendered color image under the preset lighting conditions, and is used to characterize the pre-rendering quality level of the keyframe. The pre-rendering analysis results contain a set of metadata for all keyframes, which can be called by the client in subsequent steps.

[0027] This invention utilizes server-side pre-rendering of 3D models under multi-view, multi-lighting conditions, combined with the simultaneous extraction and encoding of depth maps and occlusion maps, to transfer the rendering calculation of high-quality 3D virtual exhibits from the client to the server. This overcomes the technical problem of traditional methods relying on local client-side rendering, which leads to differences in terminal performance affecting presentation quality. This allows ordinary terminal devices to obtain high-quality visual presentations of 3D exhibits. Simultaneously, the generation of depth maps and occlusion maps during pre-rendering provides an accurate data foundation for determining occlusion relationships in subsequent virtual-real fusion, offering technical assurance for the correct spatial integration of virtual exhibits with real-world scenes.

[0028] Specifically, in step S2, obtaining the pose estimation information of the client relative to the marker and calculating the pose accuracy index includes: Step S211: The client calls the camera to capture a video stream of the real scene, extracts the current frame image from the video stream, detects whether there is a preset marker in the current frame image, and if there is, extracts the corner coordinates and encoding information of the marker. The marker is a QR code or positioning mark that is pre-arranged at the exhibition site. Step S212: Obtain the three-dimensional coordinates of the marker in the world coordinate system. Based on the corner coordinates and the three-dimensional coordinates of the marker, solve for the initial rotation matrix and initial translation vector of the client camera relative to the marker, as initial pose estimation information. Step S213: Compare the initial pose estimation information with the camera extrinsic parameters in the keyframe metadata, calculate the rotation deviation between the initial rotation matrix and the rotation matrix in the camera extrinsic parameters, and the translation deviation between the initial translation vector and the translation vector in the camera extrinsic parameters, and use the rotation deviation and the translation deviation as the pose accuracy index.

[0029] In this embodiment of the invention, there are multiple markers, each with unique encoded information. These markers are pre-positioned at fixed locations within each booth at the exhibition venue. The three-dimensional coordinates of each marker in the world coordinate system are pre-calibrated and stored in the client's local database through construction surveying or 3D scanning. After the client calls the camera to capture a video stream of the real-world scene, it performs marker detection on each frame. The detection uses a MobileNetV3 lightweight convolutional neural network for real-time inference. This network is pre-trained using a training dataset containing marker images under multiple angles and lighting conditions to output the corner coordinates and corresponding encoded information of each marker in the image. Simultaneously, it outputs the confidence score for the detected corner, which is the maximum probability output by the softmax layer of the network. When multiple markers are detected, the marker with the highest confidence score is selected as the localization reference for the current frame.

[0030] The process of determining the initial rotation matrix and initial translation vector of the client camera relative to the marker involves: establishing a correspondence between the two-dimensional image coordinates and the three-dimensional world coordinates using a perspective n-point algorithm based on the corner coordinates of the marker and its three-dimensional coordinates in the world coordinate system. Here, n represents the number of detected corner points of the marker. The solution is executed when n is greater than or equal to 4. The initial rotation matrix and initial translation vector of the client camera are obtained by minimizing the reprojection error. The minimization of the reprojection error is iteratively optimized using the Levenberg-Marquardt algorithm. The iteration terminates when the change in reprojection error between two consecutive iterations is less than a preset error change threshold. The initial rotation matrix and initial translation vector together constitute a six-degree-of-freedom pose estimation of the client camera in the local coordinate system where the marker is located.

[0031] The initial pose estimation information is compared with the camera extrinsic parameters in the keyframe metadata. Specifically, the client searches for the keyframe metadata corresponding to the currently detected marker encoding information from the pre-rendering analysis results, and extracts the camera extrinsic parameters from this metadata. The camera extrinsic parameters are the virtual camera pose from the viewpoint corresponding to the marker during server-side pre-rendering. The initial rotation matrix is ​​subtracted from the rotation matrix in the camera extrinsic parameters to obtain the rotation deviation, which is in degrees and specifically the difference in rotation angle between the initial rotation matrix and the rotation matrix in the camera extrinsic parameters. The initial translation vector is subtracted from the translation vector in the camera extrinsic parameters to obtain the translation deviation, which is in centimeters and specifically the Euclidean distance between the initial translation vector and the translation vector in the camera extrinsic parameters. The rotation deviation and the translation deviation together serve as the pose accuracy index, used to characterize the degree of deviation of the current client pose estimation from the pre-rendered standard pose. The smaller the deviation, the more accurate the pose estimation. When the rotation deviation exceeds a preset allowable value or the translation deviation exceeds a preset allowable value, a subsequent pose filtering correction strategy is triggered. The allowable value of rotation deviation is determined based on the critical angle at which the virtual exhibit in the virtual-real fusion produces a perceptible offset in the viewpoint rotation direction, and the allowable value of translation deviation is determined based on the critical distance at which the virtual exhibit produces a perceptible drift in the spatial position.

[0032] Specifically, in step S2, monitoring network state parameters and calculating network adaptability indicators includes: Step S221: The client periodically measures the available bandwidth of the current network and records the bandwidth fluctuation amplitude during the measurement process. Step S222: The client sends probe data packets to the server and receives a response, measuring the current network round-trip time and packet loss rate; Step S223: Normalize the available bandwidth, the bandwidth fluctuation range, the round-trip time and the packet loss rate respectively, and take the weighted average value as the network adaptability index.

[0033] In this embodiment of the invention, the available bandwidth is measured as follows: the client sends probe data packets to the server at fixed time intervals. The available bandwidth of the current network link is calculated by measuring the transmission rate of the probe data packets, and the difference between the maximum and minimum values ​​in multiple consecutive measurements is recorded as the bandwidth fluctuation amplitude. Simultaneously, the client sends echo probe data packets to the server, and the time difference between the sending time and the receiving time of the probe data packet is recorded as the round-trip delay. The ratio of the total number of data packets sent within a predetermined time period to the number of data packets for which no response was received is calculated as the packet loss rate.

[0034] The method for determining the preset parameters involved in the normalization process is as follows: the preset lower limit of bandwidth is determined based on the bitrate requirement corresponding to the lowest acceptable resolution of the pre-rendered video stream, and is specifically set as the minimum bitrate value required to ensure smooth playback of the video stream; the preset allowable fluctuation value is determined based on experiments on the impact of network jitter on the smoothness of video playback, and is specifically set as the maximum bandwidth fluctuation range that can be tolerated without significant stuttering; the preset upper limit of latency is determined based on the latency threshold that can be perceived by the user in virtual-real fusion interaction, and is specifically set as the end-to-end latency threshold that can be perceived by the human eye; the preset upper limit of packet loss rate is determined based on the video decoding fault tolerance capability, and is specifically set as the maximum packet loss ratio that the decoder can normally recover the image from.

[0035] The normalization process involves: comparing the available bandwidth with a preset lower bandwidth limit to obtain a bandwidth adequacy index; comparing the bandwidth fluctuation amplitude with a preset fluctuation allowable value to obtain a normalized bandwidth fluctuation index; comparing the round-trip delay with a preset delay upper limit to obtain a normalized delay index; and comparing the packet loss rate with a preset packet loss rate upper limit to obtain a normalized packet loss index. The bandwidth adequacy index is calculated by dividing the preset lower bandwidth limit by the available bandwidth. A larger available bandwidth results in a smaller index value, indicating a more abundant network.

[0036] When calculating the weighted average, the bandwidth adequacy index, the normalized bandwidth fluctuation index, the normalized latency index, and the normalized packet loss index are each assigned a preset weight coefficient, and the sum of each weight coefficient is 1. The weight coefficients are determined by testing the impact of individual changes in each parameter on the stuttering rate of the virtual-real fusion video in a simulated network environment, and using the proportion of the impact of each parameter as the corresponding weight coefficient. The network adaptability index is used to characterize the ability of the current network state to support the real-time transmission of the virtual-real fusion video stream. The larger the network adaptability index, the worse the network condition. When the network adaptability index is greater than the judgment threshold, it indicates that the current network cannot support the current transmission bitrate and resolution, and subsequent bitrate adjustment strategies need to be triggered.

[0037] This invention achieves two key input parameters required for virtual-real fusion by parallel processing of real-time pose estimation on the client side and network status monitoring. On the one hand, by solving the six-DOF pose of the client's camera and comparing the estimation results with the pre-rendered camera extrinsic parameters on the server side, rotational deviation and translational deviation are used as pose accuracy indicators. This allows the system to quantitatively evaluate the deviation of the current pose estimation from the pre-rendered standard pose, providing a clear trigger basis and data benchmark for subsequent pose filtering correction. On the other hand, by periodically measuring available bandwidth, bandwidth fluctuation amplitude, round-trip latency, and packet loss rate, multi-dimensional network status parameters are normalized and weighted into a single network adaptability indicator. This allows the system to comprehensively evaluate the current network's support capability for real-time video stream transmission, providing a quantitative decision basis for subsequent adaptive bitrate adjustment.

[0038] Specifically, in step S3, calculating the pose accuracy judgment index based on the pose accuracy index and calculating the network adaptability judgment index based on the network adaptability index includes: Obtain a preset allowable value for rotation deviation and allowable value for translation deviation. Divide the rotation deviation by the allowable value for rotation deviation to obtain a normalized rotation deviation. Divide the translation deviation by the allowable value for translation deviation to obtain a normalized translation deviation. Take the arithmetic mean of the normalized rotation deviation and the normalized translation deviation as the pose accuracy judgment index. Obtain the pre-set lower limit of available bandwidth, allowable bandwidth fluctuation range, upper limit of round-trip latency, and upper limit of packet loss rate. Divide the lower limit of available bandwidth by the available bandwidth to obtain a bandwidth adequacy index; divide the bandwidth fluctuation range by the allowable bandwidth fluctuation range to obtain a normalized bandwidth fluctuation index; divide the round-trip latency by the upper limit of round-trip latency to obtain a normalized latency index; divide the packet loss rate by the upper limit of packet loss rate to obtain a normalized packet loss index; and take the arithmetic mean of the bandwidth adequacy index, the normalized bandwidth fluctuation index, the normalized latency index, and the normalized packet loss index as the network adaptability judgment index.

[0039] In this embodiment of the invention, the method for determining the allowable value of rotational deviation is as follows: in a simulated virtual-real fusion scenario, the rotational deviation value is gradually increased by a preset step size, and the tester observes the degree of visual drift of the virtual exhibit under different rotational deviations. The minimum rotational deviation value when the virtual exhibit produces a perceptible angular offset relative to the real scene is taken as the allowable value of rotational deviation. The method for determining the allowable value of translational deviation is as follows: in a simulated virtual-real fusion scenario, the translational deviation value is gradually increased by a preset step size, and the tester observes the degree of visual drift of the virtual exhibit under different translational deviations. The minimum translational deviation value when the virtual exhibit produces a perceptible positional offset relative to the real scene is taken as the allowable value of translational deviation.

[0040] The method for determining the lower limit of available bandwidth is as follows: obtain the minimum bitrate requirement of the pre-rendered video stream at different resolutions, and take the bitrate value corresponding to the highest resolution plus a preset percentage as the lower limit of available bandwidth to ensure that the bandwidth meets the minimum requirements for normal transmission of the video stream; the method for determining the allowable value of bandwidth fluctuation is as follows: conduct video playback tests under different bandwidth fluctuation conditions, and take the maximum fluctuation amplitude when there is no stuttering or blurring during playback as the allowable value of bandwidth fluctuation; the method for determining the upper limit of round-trip latency is as follows: conduct virtual-real fusion interaction tests under different latency conditions, and take the maximum latency value when there is no perceptible delay between the user's start operation and the screen response as the upper limit of round-trip latency; the method for determining the upper limit of packet loss rate is as follows: conduct video decoding tests under different packet loss rate conditions, and take the maximum packet loss ratio when the decoder can normally recover the image without screen tearing or frame loss as the upper limit of packet loss rate.

[0041] The pose accuracy metric characterizes the deviation of the current client pose estimation accuracy from the allowable accuracy benchmark. A value greater than 1 indicates that the pose estimation accuracy has exceeded the limit, requiring pose filtering correction. The network adaptability metric characterizes the deviation of the current network state from the allowable network state. A value greater than 1 indicates that the network state can no longer support the current video stream transmission, requiring a bitrate adjustment strategy. Larger values ​​for both the pose accuracy metric and the network adaptability metric indicate a worse state in the corresponding dimension. Both use a value of 1 as the boundary benchmark for determining whether the condition is acceptable.

[0042] Specifically, in step S3, a comprehensive judgment index is calculated based on the rendering quality judgment index, the pose accuracy judgment index, and the network adaptability judgment index, and the current rendering mode is determined to be qualified based on the comprehensive judgment index, including: Obtain a preset judgment threshold. When the rendering quality judgment index is less than or equal to the judgment threshold, the pose accuracy judgment index is less than or equal to the judgment threshold, and the network adaptability judgment index is less than or equal to the judgment threshold, the current rendering mode is determined to be qualified. When the rendering quality judgment index is greater than the judgment threshold, or the pose accuracy judgment index is greater than the judgment threshold, or the network adaptability judgment index is greater than the judgment threshold, the current rendering mode is determined to be unqualified.

[0043] In this embodiment of the invention, the comprehensive judgment index is any one of the arithmetic mean, weighted average, or square root of the sum of squares of the rendering quality judgment index, the pose accuracy judgment index, and the network adaptability judgment index. Its specific calculation method is preset by the system and is used to comprehensively characterize the overall quality level of the current virtual-real fusion rendering process. The method for determining the judgment threshold is as follows: under standard network environment and standard pose conditions, running with the system's preset optimal rendering parameters, measuring the rendering quality judgment index, pose accuracy judgment index, and network adaptability judgment index under stable conditions, and taking the maximum value among the three as the judgment threshold. The judgment threshold is used to distinguish whether the current rendering mode meets the normal operation requirements.

[0044] When all three judgment indicators are less than or equal to the judgment threshold, it indicates that the rendering quality, pose accuracy, and network adaptability are all within the allowable range, and the coordination among these dimensions is normal. The system can provide users with a smooth and accurate virtual-real fusion viewing experience, so the current rendering mode is deemed qualified, and the current rendering parameters are maintained. When any one judgment indicator exceeds the judgment threshold, it indicates that at least one dimension has exceeded the allowable range. The abnormality in this dimension will lead to an overall decrease in the virtual-real fusion effect, specifically manifested as screen stuttering or blurring, virtual exhibit position drift, or visual delay. In this case, the current rendering mode is deemed unqualified, and further analysis of the reasons for the unqualification is required to trigger corresponding adjustment strategies to bring each judgment indicator back to the normal state of being less than or equal to the judgment threshold.

[0045] Specifically, in step S3, when the rendering mode is unqualified, the category of the unqualified reason is determined based on the deviation direction between various judgment indicators, including: When the current rendering mode is determined to be unqualified, calculate the first ratio of the network adaptability determination index to the pose accuracy determination index, and the second ratio of the pose accuracy determination index to the rendering quality determination index. When the first ratio is greater than or equal to the first preset ratio, the failure reason category is determined to be a network problem; when the first ratio is less than or equal to the second preset ratio, and the second ratio is greater than or equal to the third preset ratio, the failure reason category is determined to be a pose estimation problem; when the first ratio is between the second preset ratio and the first preset ratio, and the second ratio is between the fourth preset ratio and the third preset ratio, the failure reason category is determined to be a comprehensive problem.

[0046] In this embodiment of the invention, when the current rendering mode is determined to be unqualified, it indicates that at least one of the rendering quality judgment index, pose accuracy judgment index, and network adaptability judgment index is greater than the judgment threshold. At this time, by calculating a first ratio of the network adaptability judgment index to the pose accuracy judgment index, and a second ratio of the pose accuracy judgment index to the rendering quality judgment index, the relative deviation between the three judgment indices is quantified to identify the main source direction of the unqualification.

[0047] The first preset ratio and the second preset ratio are determined as follows: the first preset ratio is determined based on the critical ratio between the network adaptability judgment index and the pose accuracy judgment index when the network adaptability judgment index is dominantly unqualified. Specifically, the value is the critical multiple when the network adaptability judgment index is several times the judgment threshold and the pose accuracy judgment index is close to the judgment threshold. The second preset ratio is the reciprocal of the first preset ratio and is used to determine whether the pose accuracy judgment index dominates the relationship between the network adaptability judgment index and the pose accuracy judgment index. The third preset ratio and the fourth preset ratio are determined as follows: the third preset ratio is determined based on the critical ratio between the pose accuracy judgment index and the rendering quality judgment index when the pose accuracy judgment index is dominantly unqualified. Specifically, the value is the critical multiple when the pose accuracy judgment index is several times the judgment threshold and the rendering quality judgment index is close to the judgment threshold. The fourth preset ratio is the reciprocal of the third preset ratio and is used to determine whether the rendering quality judgment index dominates the relationship between the pose accuracy judgment index and the rendering quality judgment index.

[0048] When the first ratio is greater than or equal to the first preset ratio, it indicates that the network adaptability assessment index is significantly larger than the pose accuracy assessment index, meaning the network state deterioration is much greater than the decrease in pose estimation accuracy. Therefore, the failure reason is classified as a network problem. When the first ratio is less than or equal to the second preset ratio, and the second ratio is greater than or equal to the third preset ratio, it indicates that the network adaptability assessment index is significantly smaller than the pose accuracy assessment index, meaning the pose accuracy assessment index dominates the relationship between the two indices. Simultaneously, the pose accuracy assessment index is significantly larger than the rendering quality assessment index, meaning the decrease in pose estimation accuracy is much greater than the decrease in rendering quality. Therefore, the failure reason is classified as a pose estimation problem. When the first ratio is between the second preset ratio and the first preset ratio, and the second ratio is between the fourth preset ratio and the third preset ratio, it indicates that no single dimension significantly dominates the deviation among the assessment indices. The failure is caused by anomalies in multiple dimensions, therefore, the failure reason is classified as a comprehensive problem.

[0049] This invention constructs three independent evaluation indicators—rendering quality, pose accuracy, and network adaptability—and uses a comprehensive evaluation of these three indicators to determine the pass / fail status of the current rendering mode, achieving a multi-dimensional comprehensive evaluation of virtual-real fusion rendering quality. When any evaluation indicator exceeds its limit, a failure judgment is triggered. Then, a first ratio and a second ratio are calculated based on the deviation directions between the evaluation indicators. By comparing these ratios with preset thresholds, the cause of failure is precisely attributed to network problems, pose estimation problems, or a combination of issues. This hierarchical judgment mechanism overcomes the technical problems of traditional methods, such as the difficulty in comprehensively evaluating virtual-real fusion quality with a single indicator and the reliance on human experience and strong subjectivity in cause judgment. It enables the system to automatically identify the dominant dimension of quality degradation, providing accurate attribution basis for subsequent differentiated adjustment strategies, improving the accuracy of problem diagnosis and the targeting of adjustment strategies.

[0050] Specifically, in step S4, the corresponding adjustment strategy is executed according to the category of non-compliance reasons, including: When the failure reason category is determined to be a network problem, the transmission bit rate is adjusted according to the available bandwidth, and the resolution or frame rate of the video stream is reduced so that the network adaptability judgment index returns to less than or equal to the judgment threshold. When the cause of non-compliance is determined to be a pose estimation problem, Kalman filtering is activated to predict and correct the initial pose estimation information. The filtering parameters are dynamically adjusted according to the rotation deviation and the translation deviation so that the pose accuracy judgment index returns to less than or equal to the judgment threshold. When the failure reason category is determined to be a comprehensive problem, bit rate adjustment and pose filtering correction are performed simultaneously, and asynchronous time distortion compensation is initiated so that each component of the comprehensive judgment index returns to less than or equal to the judgment threshold.

[0051] In this embodiment of the invention, when the failure reason is determined to be a network problem, the specific execution method of bitrate adjustment is as follows: The client calculates a bitrate adjustment coefficient based on the ratio of the available bandwidth to the current transmission bitrate, multiplies the current transmission bitrate by the bitrate adjustment coefficient to obtain the target bitrate, matches the corresponding resolution and frame rate combination from a preset bitrate-resolution-frame rate mapping table according to the target bitrate, and sends a bitrate switching request to the server. After responding to the request, the server re-encodes the pre-rendered video stream according to the new resolution and frame rate and sends it to the client. If the network adaptability judgment index is still greater than the judgment threshold after adjustment, the bitrate adjustment is performed cyclically until the network adaptability judgment index is less than or equal to the judgment threshold, or until the resolution drops to a preset minimum resolution limit.

[0052] When the failure reason is determined to be a pose estimation problem, the specific execution method of Kalman filtering is as follows: using the initial rotation matrix and the initial translation vector as observations, and the pose state variables of the client camera as filter state variables, a state transition matrix and an observation matrix are set. The process noise covariance matrix and the observation noise covariance matrix are dynamically adjusted according to the rotation deviation and the translation deviation. The corrected pose estimation value is obtained through iterative calculation of the prediction step and update step of the Kalman filter. The larger the rotation deviation, the smaller the value of the observation noise covariance matrix is ​​adjusted to enhance the filter's confidence in the current observation value; the larger the translation deviation, the smaller the value of the process noise covariance matrix is ​​adjusted to enhance the filter's confidence in the predicted value. After filtering correction, the pose accuracy judgment index is recalculated. If it is still greater than the judgment threshold, the filtering correction is performed iteratively until the pose accuracy judgment index is less than or equal to the judgment threshold.

[0053] When the failure reason is determined to be a comprehensive problem, bitrate adjustment and pose filtering correction are executed simultaneously. The execution frequency of bitrate adjustment and pose filtering correction is preset by the system. After each cycle of bitrate adjustment, the network adaptability judgment index is recalculated, and after each cycle of pose filtering correction, the pose accuracy judgment index is recalculated. If either the network adaptability judgment index or the pose accuracy judgment index is still greater than the judgment threshold after adjustment, the corresponding dimension adjustment continues until both indicators are simultaneously less than or equal to the judgment threshold. Simultaneously with bitrate adjustment and pose filtering correction, asynchronous time distortion compensation is initiated. This asynchronous time distortion compensation operates independently of the rendering pipeline, applying viewpoint transformation compensation based on the latest pose to the rendered frames before each display refresh to smooth out screen jitter caused by the adjustment process. Once both the network adaptability judgment index and the pose accuracy judgment index return to less than or equal to the judgment threshold, the current adjustment parameters continue to run, and each judgment index is continuously monitored until none of the judgment indices exceed the limit again within multiple consecutive monitoring cycles. At this point, the judgment adjustment strategy is complete.

[0054] This invention achieves precise closed-loop control of virtual-real fusion rendering quality by implementing differentiated adjustment strategies for three types of non-compliance causes: network problems, pose estimation problems, and comprehensive problems. When a network problem is identified, the resolution and frame rate are dynamically adjusted based on the ratio of available bandwidth to the current transmission bitrate, bringing the network adaptability assessment index back to the normal range and overcoming the technical problem that fixed bitrate transmission cannot adapt to bandwidth fluctuations. When a pose estimation problem is identified, the initial rotation matrix and translation vector are used as observations, and the pose state variables are predicted and corrected using Kalman filtering. The filter noise covariance matrix is ​​then adjusted inversely based on the magnitude of the rotation and translation deviations, allowing the filter parameters to adaptively match the current pose drift level. This achieves dynamic and precise correction of pose estimation, overcoming the problem of insufficient adaptability of traditional fixed-parameter filtering when the drift level changes.

[0055] Specifically, after executing the adjustment strategy and performing virtual-real fusion based on the depth map, the output includes: The client obtains the current frame pre-rendered color image, the depth map, and the occlusion map from the pre-rendered video stream, and projects the three-dimensional model of the virtual exhibit onto the current frame image coordinate system according to the current pose to obtain the two-dimensional projection area of ​​the virtual exhibit and the virtual depth value corresponding to each pixel. For each pixel within the two-dimensional projection area, the virtual depth value is compared with the real depth value at the corresponding position in the depth map. When the virtual depth value is less than the real depth value, the virtual exhibit is determined to be visible, and the color of the virtual exhibit is used as the fusion output color of the pixel. When the virtual depth value is greater than or equal to the real depth value, the virtual exhibit is determined to be occluded, and the color of the real scene is used as the fusion output color of the pixel. The merged output image is accelerated and rendered, then encoded into a low-latency video stream and pushed to the client's display screen.

[0056] Specifically, the activation condition for the asynchronous time distortion compensation is as follows: when the rendering quality judgment index is greater than the judgment threshold, the time difference between the rendering completion time of the current frame and the planned display time is calculated. Based on the time difference and the angle change of the client camera between the current time and the rendering time, the pixel displacement compensation amount is calculated. The pixel displacement compensation amount is applied to the blended image that has been rendered but not yet displayed, and a compensated image with a smooth view transition is generated and output.

[0057] In this embodiment of the invention, the client obtains the current frame pre-rendered color image, the depth map, and the occlusion map from the pre-rendered video stream. Specifically, the client decodes the pre-rendered color image of the current frame to be displayed from the received pre-rendered video stream, and extracts the depth map and occlusion map corresponding to the frame from the auxiliary data channel transmitted synchronously with the pre-rendered video stream. The current pose is the latest pose estimation result of the client camera after Kalman filtering correction at the current moment. The projection of the three-dimensional model of the virtual exhibit onto the current frame image coordinate system according to the current pose specifically involves: based on the intrinsic parameter matrix of the client camera and the current pose, transforming each vertex of the three-dimensional model of the virtual exhibit from the world coordinate system to the camera coordinate system, and then projecting it onto the image plane coordinate system to obtain the two-dimensional projection area of ​​the virtual exhibit in the image, and recording the distance of each pixel's corresponding three-dimensional model surface point from the virtual camera as the virtual depth value.

[0058] For each pixel within the two-dimensional projection area, the specific method for comparing the virtual depth value with the corresponding real depth value in the depth map is as follows: Obtain the depth value of the pixel in the depth map that is at the same position as each pixel in the two-dimensional projection area, and use this as the corresponding real depth value; when the virtual depth value is less than the real depth value, it indicates that the virtual exhibit is in front of the real scene at that pixel position, and the virtual exhibit is determined to be visible. The fusion output color of this pixel is then set to the virtual exhibit color of the corresponding pixel in the pre-rendered color image; when the virtual depth value is greater than or equal to the real depth value, it indicates that the virtual exhibit is occluded by an object in the real scene at that pixel position, and the virtual exhibit is determined to be occluded. The fusion output color of this pixel is then set to the real scene color of the corresponding pixel in the real scene video stream captured by the client.

[0059] After accelerated rendering, the merged output screen is encoded into a low-latency video stream and pushed to the client's display screen. Specifically, the client calls the GPU graphics interface to perform hardware-accelerated rendering of the merged output screen, inputs the rendered screen frames into the encoder, encodes them into a low-latency video stream using the H.265 encoding format, and pushes them to the client's display screen for display via a real-time streaming protocol.

[0060] The asynchronous time warp compensation is activated when the rendering quality judgment index exceeds the judgment threshold, indicating insufficient rendering frame rate or excessive rendering latency. The compensation process is as follows: The rendering time and the corresponding client camera pose are recorded as the rendering pose when the current frame is completed. When the planned display time arrives, the latest pose of the client camera is obtained as the display pose. The time difference between the planned display time and the rendering time is calculated, and the angular change of the display pose relative to the rendering pose is calculated. Based on the time difference and the angular change, the pixel displacement compensation amount on the image plane caused by the viewpoint change is calculated. The pixel displacement compensation amount is applied to each pixel in the blended image that has been rendered but not yet displayed, moving the pixel to the corresponding position after the viewpoint change. The resulting voids are filled using interpolation, generating a smoothly transitioned compensated image before output. The pixel displacement compensation amount is calculated as follows: Based on the angular change and the client camera's focal length parameters, the pixel movement distance along the horizontal and vertical directions on the image plane is calculated. The pixel movement distance is proportional to the angular change. Asynchronous time warp compensation is performed before each display refresh, without blocking the main rendering pipeline, so that the viewpoint movement can still be smooth and continuous even when the rendering frame rate is insufficient or the latency fluctuates.

[0061] The technical solution of the present invention has been described above with reference to the preferred embodiments shown in the accompanying drawings. However, it will be readily understood by those skilled in the art that the scope of protection of the present invention is obviously not limited to these specific embodiments. Without departing from the principles of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will all fall within the scope of protection of the present invention.

Claims

1. A real-time rendering method for AI large-scale model virtual-real fusion video streams used in online exhibitions, characterized in that, include: Step S1: The server pre-renders the 3D model of the exhibition, generates a pre-rendered video stream containing a depth map and an occlusion map, extracts key frame metadata from the pre-rendered video stream, and generates pre-rendered analysis results based on the key frame metadata. The key frame metadata includes camera extrinsic parameters and rendering quality scores. Step S2: The client receives the pre-rendered video stream and the pre-rendered analysis result, collects the real-world scene video stream and detects markers to obtain the client's pose estimation information relative to the markers. At the same time, the client monitors network state parameters, calculates the pose accuracy index based on the pose estimation information, and calculates the network adaptability index based on the network state parameters. Step S3: Extract the rendering quality score from the pre-rendering analysis results as the rendering quality judgment index; calculate the pose accuracy judgment index based on the pose accuracy index; calculate the network adaptability judgment index based on the network adaptability index; calculate the comprehensive judgment index based on the rendering quality judgment index, the pose accuracy judgment index, and the network adaptability judgment index; determine whether the current rendering mode is qualified based on the comprehensive judgment index; and determine the category of the reason for the failure based on the deviation direction between each judgment index if it is unqualified. Step S4: Execute the corresponding adjustment strategy according to the category of non-compliance reasons, and output the result after completing the virtual-real fusion based on the depth map after executing the adjustment strategy.

2. The real-time rendering method for AI large-scale model virtual-real fusion video stream for online exhibitions according to claim 1, characterized in that, In step S1, a pre-rendered video stream containing a depth map and an occlusion map is generated, and keyframe metadata is extracted, including: Step S11: The server loads the 3D model of the exhibition, sets multiple preset viewpoints and multiple preset lighting conditions, renders the 3D model under each preset viewpoint and each preset lighting condition, generates a pre-rendered color image corresponding to each viewpoint, extracts the depth value of each pixel in the pre-rendered color image during the rendering process, generates a depth map based on the extracted depth value, and determines the front and back occlusion relationship of each pixel based on the depth value of each pixel in the depth map, and generates an occlusion map. Step S12: Encode the pre-rendered color image, the depth map, and the occlusion map from the same viewpoint into a pre-rendered video stream in chronological order; Step S13: Extract metadata from each keyframe of the pre-rendered video stream. The metadata includes at least camera extrinsic parameters and rendering quality scores. Associate the metadata with the corresponding keyframes and store them to form a pre-rendered analysis result.

3. The real-time rendering method for AI large-scale model virtual-real fusion video stream for online exhibitions according to claim 1, characterized in that, In step S2, the pose estimation information of the client relative to the marker is obtained and the pose accuracy index is calculated, including: Step S211: The client calls the camera to capture a video stream of the real scene, extracts the current frame image from the video stream, detects whether there is a preset marker in the current frame image, and if there is, extracts the corner coordinates and encoding information of the marker. The marker is a QR code or positioning mark that is pre-arranged at the exhibition site. Step S212: Obtain the three-dimensional coordinates of the marker in the world coordinate system. Based on the corner coordinates and the three-dimensional coordinates of the marker, solve for the initial rotation matrix and initial translation vector of the client camera relative to the marker, as initial pose estimation information. Step S213: Compare the initial pose estimation information with the camera extrinsic parameters in the keyframe metadata, calculate the rotation deviation between the initial rotation matrix and the rotation matrix in the camera extrinsic parameters, and the translation deviation between the initial translation vector and the translation vector in the camera extrinsic parameters, and use the rotation deviation and the translation deviation as the pose accuracy index.

4. The real-time rendering method for AI large-scale model virtual-real fusion video stream for online exhibitions according to claim 3, characterized in that, In step S2, network state parameters are monitored and network adaptability indices are calculated, including: Step S221: The client periodically measures the available bandwidth of the current network and records the bandwidth fluctuation amplitude during the measurement process. Step S222: The client sends probe data packets to the server and receives a response, measuring the current network round-trip time and packet loss rate; Step S223: Normalize the available bandwidth, the bandwidth fluctuation range, the round-trip time and the packet loss rate respectively, and take the weighted average value as the network adaptability index.

5. The real-time rendering method for AI large-scale model virtual-real fusion video stream for online exhibitions according to claim 1, characterized in that, In step S3, the pose accuracy determination index is calculated based on the pose accuracy index, and the network adaptability determination index is calculated based on the network adaptability index, including: Obtain a preset allowable value for rotation deviation and allowable value for translation deviation. Divide the rotation deviation by the allowable value for rotation deviation to obtain a normalized rotation deviation. Divide the translation deviation by the allowable value for translation deviation to obtain a normalized translation deviation. Take the arithmetic mean of the normalized rotation deviation and the normalized translation deviation as the pose accuracy judgment index. Obtain the pre-set lower limit of available bandwidth, allowable bandwidth fluctuation range, upper limit of round-trip latency, and upper limit of packet loss rate. Divide the lower limit of available bandwidth by the available bandwidth to obtain a bandwidth adequacy index; divide the bandwidth fluctuation range by the allowable bandwidth fluctuation range to obtain a normalized bandwidth fluctuation index; divide the round-trip latency by the upper limit of round-trip latency to obtain a normalized latency index; divide the packet loss rate by the upper limit of packet loss rate to obtain a normalized packet loss index; and take the arithmetic mean of the bandwidth adequacy index, the normalized bandwidth fluctuation index, the normalized latency index, and the normalized packet loss index as the network adaptability judgment index.

6. The real-time rendering method for AI large-scale model virtual-real fusion video stream for online exhibitions according to claim 4, characterized in that, In step S3, a comprehensive judgment index is calculated based on the rendering quality judgment index, the pose accuracy judgment index, and the network adaptability judgment index, and the current rendering mode is determined to be qualified based on the comprehensive judgment index, including: Obtain a preset judgment threshold. When the rendering quality judgment index is less than or equal to the judgment threshold, the pose accuracy judgment index is less than or equal to the judgment threshold, and the network adaptability judgment index is less than or equal to the judgment threshold, the current rendering mode is determined to be qualified. When the rendering quality judgment index is greater than the judgment threshold, or the pose accuracy judgment index is greater than the judgment threshold, or the network adaptability judgment index is greater than the judgment threshold, the current rendering mode is determined to be unqualified.

7. The real-time rendering method for AI large-scale model virtual-real fusion video stream for online exhibitions according to claim 6, characterized in that, In step S3, when the rendering mode is unqualified, the category of the unqualified reason is determined based on the deviation direction between each judgment index, including: When the current rendering mode is determined to be unqualified, calculate the first ratio of the network adaptability determination index to the pose accuracy determination index, and the second ratio of the pose accuracy determination index to the rendering quality determination index. When the first ratio is greater than or equal to the first preset ratio, the failure reason category is determined to be a network problem; when the first ratio is less than or equal to the second preset ratio, and the second ratio is greater than or equal to the third preset ratio, the failure reason category is determined to be a pose estimation problem; when the first ratio is between the second preset ratio and the first preset ratio, and the second ratio is between the fourth preset ratio and the third preset ratio, the failure reason category is determined to be a comprehensive problem.

8. The real-time rendering method for AI large-scale model virtual-real fusion video stream for online exhibitions according to claim 7, characterized in that, In step S4, the corresponding adjustment strategy is executed according to the category of non-compliance reasons, including: When the failure reason category is determined to be a network problem, the transmission bit rate is adjusted according to the available bandwidth, and the resolution or frame rate of the video stream is reduced so that the network adaptability judgment index returns to less than or equal to the judgment threshold. When the cause of non-compliance is determined to be a pose estimation problem, Kalman filtering is activated to predict and correct the initial pose estimation information. The filtering parameters are dynamically adjusted according to the rotation deviation and the translation deviation so that the pose accuracy judgment index returns to less than or equal to the judgment threshold. When the failure reason category is determined to be a comprehensive problem, bit rate adjustment and pose filtering correction are performed simultaneously, and asynchronous time distortion compensation is initiated so that each component of the comprehensive judgment index returns to less than or equal to the judgment threshold.

9. The real-time rendering method for AI large-scale model virtual-real fusion video stream for online exhibitions according to claim 8, characterized in that, After executing the adjustment strategy, the output is completed based on the depth map after virtual-real fusion, including: The client obtains the current frame pre-rendered color image, the depth map, and the occlusion map from the pre-rendered video stream, and projects the three-dimensional model of the virtual exhibit onto the current frame image coordinate system according to the current pose to obtain the two-dimensional projection area of ​​the virtual exhibit and the virtual depth value corresponding to each pixel; For each pixel within the two-dimensional projection area, the virtual depth value is compared with the real depth value at the corresponding position in the depth map. When the virtual depth value is less than the real depth value, the virtual exhibit is determined to be visible, and the color of the virtual exhibit is used as the fusion output color of the pixel. When the virtual depth value is greater than or equal to the real depth value, the virtual exhibit is determined to be occluded, and the color of the real scene is used as the fusion output color of the pixel. The merged output image is accelerated and rendered, then encoded into a low-latency video stream and pushed to the client's display screen.

10. The real-time rendering method for AI large-scale model virtual-real fusion video stream for online exhibitions according to claim 9, characterized in that, The activation condition for the asynchronous time distortion compensation is as follows: when the rendering quality judgment index is greater than the judgment threshold, the time difference between the rendering completion time of the current frame and the planned display time is calculated. Based on the time difference and the angle change of the client camera between the current time and the rendering time, the pixel displacement compensation amount is calculated. The pixel displacement compensation amount is applied to the blended image that has been rendered but not yet displayed, and the compensated image with smooth view transition is generated and then output.

Images