Method and system for video transmission based on mmwave, 5g and wi-fi hybrid networking

Abstract

The application relates to a method and system for video transmission based on millimeter wave, 5G and Wi-Fi hybrid networking. A unified resource pool of millimeter wave, 5G and Wi-Fi is constructed through a network interface abstraction layer, and a multi-path transmission protocol is used to dynamically allocate paths for transmission units in combination with video stream segmentation, thereby realizing aggregation and parallel transmission of bandwidth resources of a heterogeneous network. Compared with a single network or a simple primary-backup scheme, the method breaks through the physical bandwidth limitation of a single frequency band, effectively solves the problems of spectrum congestion and signal shielding in a large-space multi-user scene, greatly improves the total throughput and transmission stability of the system, and thereby supports concurrent transmission of higher-resolution XR video streams.

Description

Technical Field

[0001] This application relates to the field of wireless communication and extended reality technologies, and in particular to a method and system for video transmission based on a hybrid network of millimeter wave, 5G and Wi-Fi. Background Technology

[0002] With the rapid development of extended reality (XR), metaverse, and digital twin technologies, large-space, multi-user XR applications, such as VR e-sports arenas, large-scale virtual exhibitions, and industrial simulation training, are gradually becoming industry hotspots. These applications require providing immersive video streams with resolutions up to 8K and refresh rates exceeding 90Hz to dozens or even hundreds of users simultaneously within a physical space of several hundred square meters. This poses extremely high challenges to the bandwidth capacity, transmission latency, and stability of wireless communication networks.

[0003] However, existing solutions have the following limitations:

[0004] The bottlenecks of a single network are obvious: Wi-Fi is prone to congestion and has large latency jitter under high concurrency; 5G Sub-6GHz bandwidth capacity is limited; although millimeter wave has high bandwidth, its penetration is poor and it is very easy to cause signal interruption due to human body blockage.

[0005] Insufficient heterogeneous network collaboration: Existing Wi-Fi+5G solutions mostly adopt a simple master-slave switching mode, which cannot aggregate the bandwidth of multiple networks, resulting in resource waste and failing to meet the requirements of high bit rate transmission.

[0006] Lagging handover mechanism: Traditional handover relies on physical indicators such as signal strength index (RSSI), which is a passive response. In sudden situations such as millimeter wave signal blockage, the system often loses connection before it can establish a new link, causing XR screen to stutter or go black, which seriously affects the immersion and causes dizziness. Summary of the Invention

[0007] To address the aforementioned issues, this application provides a method and system for video transmission based on a hybrid network of millimeter wave, 5G, and Wi-Fi to effectively improve transmission capacity and stability in large-space XR scenarios.

[0008] To achieve the above objectives, in a first aspect, the method for video transmission based on a hybrid network of millimeter wave, 5G, and Wi-Fi provided in this application includes the following steps:

[0009] S1. Register the millimeter-wave network interface, 5G network interface and Wi-Fi network interface in the current environment through the network interface abstraction layer to establish a unified network resource pool;

[0010] S2. Monitor the status parameters of each network interface in the network resource pool in real time, and aggregate and calculate the global capacity index that characterizes the system's transmission capability;

[0011] S3. Divide the XR video stream to be transmitted into multiple independent transmission units;

[0012] S4. Based on the status parameters of each network interface and the attributes of the transmission unit, a transmission path is dynamically allocated to each transmission unit using a multipath transmission protocol, and the transmission unit is sent through at least one selected network interface.

[0013] S5. At the receiving end, the transmission units arriving through different network interfaces are reassembled to restore a continuous video stream.

[0014] Preferably, the method further includes a seamless switching mechanism based on the user's mobility state, specifically including:

[0015] The system collects measurement data from the inertial measurement unit of the XR terminal worn by the user in real time, and performs low-pass filtering and attitude calculation on the data to estimate the short-term displacement and orientation of the XR terminal.

[0016] Based on the short-term displacement and orientation, combined with the preset environmental model or historical signal change characteristics, the prediction model is used to calculate the link blockage probability and prediction confidence within the future preset time window.

[0017] The link occlusion probability is compared with a preset occlusion threshold, or the decrease in the current link quality coefficient is monitored.

[0018] When the prediction result of the prediction model within the future preset time window satisfies the link occlusion probability being higher than the occlusion threshold and the prediction confidence level being met, or when the instantaneous drop in the current link quality coefficient exceeds the preset safety tolerance, the backup link establishment process is immediately triggered before the current link is disconnected, and a handshake is performed in advance to reserve a switching margin.

[0019] Preferably, after triggering and establishing the backup link, the system enters dual-link parallel transmission. The specific steps of the dual-link parallel transmission include:

[0020] The same transmission unit is copied into two copies, which are then transmitted in parallel through the current link and the backup link, respectively.

[0021] The receiving end performs integrity verification on the data packets; if the data packets from both links pass the verification, it further compares the arrival delay, signal-to-noise ratio snapshot, or bit error rate soft information, and selects the data packet copy with better link quality as valid data.

[0022] The communication quality of the backup link is monitored in real time. The verification is deemed successful only when the effective bandwidth, packet loss rate and round-trip delay of the backup link simultaneously meet the preset quality thresholds and the satisfaction status continues for a preset number of monitoring periods.

[0023] After the verification is passed, the data transmission weight of the current link is gradually reduced until transmission stops completely, and the switchover is completed; if a performance regression is detected during the monitoring period after the switchover, the system is immediately rolled back to the current link.

[0024] Preferably, step S2, the step of aggregating and calculating the global capacity index characterizing the system's transmission capability, specifically includes:

[0025] S21. Periodically collect raw network status data for each network interface, wherein the raw data includes at least physical negotiation rate, received signal strength, signal-to-noise ratio, short-time packet loss rate, and round-trip time.

[0026] S22. Normalize the received signal strength, signal-to-noise ratio, short-time packet loss rate and round-trip delay respectively, and perform time-domain smoothing filtering on the normalized values ​​to obtain the normalized index values ​​of each dimension.

[0027] S23. According to the preset weighting strategy, the normalized index values ​​of each dimension are weighted and summed to calculate the link quality coefficient of each network interface;

[0028] S24. Calculate the effective bandwidth of each network interface based on the physical negotiation rate, link quality coefficient, and preset protocol overhead coefficient;

[0029] S25. Select network interfaces with a link quality coefficient greater than a preset quality threshold as available interfaces, accumulate the effective bandwidth of all available interfaces, and apply a safety margin coefficient to the accumulation result to obtain the global capacity index.

[0030] Preferably, step S3, which involves dividing the XR video stream into multiple independent transmission units, specifically includes:

[0031] S31. Perform deep packet inspection on the XR video stream and parse the header information of the network abstraction layer unit to identify video frame boundaries and frame types, wherein the frame types include at least keyframes and reference frames;

[0032] S32. According to the video coding boundary or the preset maximum transmission unit limit, several network abstraction layer units are aggregated and encapsulated into a transmission unit;

[0033] S33. For transmission units containing key frames, mark them as high priority and calculate and generate forward error correction redundancy information to encapsulate into the transmission unit; for transmission units containing reference frames, mark them as dynamic scheduling priority.

[0034] S34. Add an encapsulation header to each of the transmission units, wherein the encapsulation header includes at least a global sequence number, a priority identifier, a timestamp, and a forward error correction index for receiver reassembly and error recovery.

[0035] Preferably, in step S4, the dynamic allocation of transmission paths follows the following priority strategy:

[0036] S41. Set the millimeter wave network path as the highest priority, the 5G network path as the second priority, and the Wi-Fi network path as the basic guarantee path;

[0037] S42. Monitor the packet loss rate, signal-to-noise ratio change, round-trip delay increment, and link blockage probability of the current working path in real time; determine that the current working path has been degraded only when any of the above indicators exceeds the preset threshold and continues to meet the preset number of consecutive monitoring periods.

[0038] S43. When the millimeter wave network path is downgraded, the candidate 5G network path and Wi-Fi network path are comprehensively ranked using a weighted scoring method, and the network path with the highest score is switched to.

[0039] S44. When making a path switching decision, the switching action is only allowed if the running time of the current working path exceeds the preset minimum hold time and the time interval since the last switch exceeds the preset cooldown time.

[0040] Preferably, the method further includes a rapid authentication step:

[0041] When the XR terminal worn by the user first accesses the network resource pool, the management terminal of the network resource pool generates a global session token containing the user's permission set, validity period and digital signature based on a preset encryption key.

[0042] The global session token is synchronously distributed to the local storage units of all candidate network access devices within the coverage area;

[0043] When an XR terminal initiates a network handover request, it carries the global session token in the handshake message;

[0044] The target network access device uses data in its local storage unit or a pre-set public key to perform offline verification of the global session token; if the signature verification passes and the token is within its validity period, it directly inherits the user session state and establishes a connection, skipping the complete identity negotiation process.

[0045] Preferably, in step S5, the specific steps of recombination include:

[0046] S51. Maintain a sliding reassembly window based on the global sequence number at the receiving end, and set a maximum waiting delay threshold;

[0047] S52. Fill the out-of-order transmission units into the corresponding positions of the sliding reassembly window according to the global sequence number;

[0048] S53. Perform timeout determination for missing transmission units within the window: if the current time exceeds the maximum waiting delay threshold and the transmission unit has not yet arrived, it is determined to be a timeout;

[0049] S54. If the transmission unit that timed out is a high-priority transmission unit, forward error correction decoding recovery will be initiated first; if recovery fails, a fast retransmission request or an immediate decoding refresh request will be triggered; if the transmission unit that timed out is a dynamically scheduled priority transmission unit, time domain frame interpolation or motion compensation algorithm will be used to cover up the error, and a selective acknowledgment will be sent to the sending end to trigger background retransmission.

[0050] S55. Calculate the timeout frequency and retransmission rate, and feed them back to the transmitting end to drive the adaptive adjustment of the transmission unit size and redundancy ratio in the next stage.

[0051] Preferably, the method further includes the step of dynamically adjusting video stream parameters based on the network resource pool status:

[0052] Compare the global capacity metric with the bitrate of the current video stream;

[0053] When the global capacity metric is lower than the product of the current video stream's bitrate and the safety margin,

[0054] The encoding parameters of the XR video stream are generated to reduce the video bitrate or resolution.

[0055] Secondly, embodiments of this application provide a system for video transmission based on a hybrid network of millimeter wave, 5G, and Wi-Fi. The system includes a processor, a communication interface, and a memory. The memory stores a computer program. When the computer program is executed by the processor, it implements the method for video transmission based on a hybrid network of millimeter wave, 5G, and Wi-Fi as described in any embodiment of the first aspect.

[0056] This application presents a method and system for video transmission based on a hybrid network of millimeter wave, 5G, and Wi-Fi. It constructs a unified resource pool for millimeter wave, 5G, and Wi-Fi through a network interface abstraction layer, and dynamically allocates paths to transmission units using a multi-path transmission protocol combined with video stream segmentation, achieving aggregation and parallel transmission of heterogeneous network bandwidth resources. Compared to single-network or simple primary / backup solutions, this method overcomes the physical bandwidth limitations of a single frequency band, effectively solving the problems of spectrum congestion and signal obstruction in large-scale, multi-user scenarios, significantly improving the system's total throughput and transmission stability, thereby supporting concurrent transmission of higher-resolution XR video streams. Attached Figure Description

[0057] Figure 1This is a flowchart illustrating a method for video transmission based on a hybrid network of millimeter wave, 5G, and Wi-Fi, as provided in an embodiment of this application.

[0058] Figure 2 This is a timing diagram illustrating the seamless handover process of heterogeneous networks provided in an embodiment of this application. Detailed Implementation

[0059] The preferred embodiments of this application are described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit this application.

[0060] This application provides a system for video transmission based on a hybrid network of millimeter wave, 5G and Wi-Fi. The system aims to solve the bandwidth bottleneck and mobility continuity problem of multi-user high-concurrency video transmission in large-space XR scenarios such as large VR e-sports arenas, industrial simulation workshops or virtual museums with a floor area of ​​hundreds of square meters.

[0061] Specifically, the system includes, at the physical level:

[0062] (i) Network infrastructure, namely the heterogeneous network access layer, which is responsible for building a hybrid wireless network environment covering the entire site. Specifically, it includes millimeter wave access points (mmWave APs), 5G indoor small cells and Wi-Fi 6 / 6E access points (Wi-Fi APs) that are connected to the core switch via high-speed fiber or 10 Gigabit Ethernet to form a unified wired backbone backhaul network.

[0063] (ii) A central server, comprising at least one processor, a communication interface for connecting to the core switch, and a memory. The memory stores computer program instructions, and the processor executes these instructions to logically implement the following functions:

[0064] Network Resource Pool Manager: Used to uniformly manage heterogeneous network interfaces and implement global resource monitoring and scheduling.

[0065] Video distribution server: Responsible for real-time rendering, encoding, slicing, and multi-path distribution of XR video content.

[0066] (iii) XR terminal devices: user-worn terminal devices, such as VR all-in-one machines and AR glasses. These devices serve as the system's receiving and interactive ends and possess the following hardware characteristics:

[0067] Multi-mode communication capability: Built-in or via USB-C connection, it has a communication module that supports millimeter wave, 5G and Wi-Fi concurrent operation, and each module has an independent radio frequency front end.

[0068] Motion sensing capability: It integrates a high-precision inertial measurement unit (IMU), which typically includes a three-axis gyroscope, a three-axis accelerometer, and a magnetometer. This IMU is not only used for head tracking of the screen, but also for real-time acquisition of the user's motion posture and acceleration data, providing input for the network layer's switching prediction.

[0069] Based on the aforementioned system hardware architecture, this application further provides a method for video transmission based on a hybrid network of millimeter wave, 5G, and Wi-Fi. This method primarily runs on the aforementioned central server and XR terminal device. When the central server runs the computer program in its memory, it logically collaborates with the video distribution server through a network resource pool manager to execute specific transmission control procedures.

[0070] The following is combined with Figure 1 The specific execution steps of the method described in this embodiment are explained in detail below:

[0071] S1. Register the millimeter-wave network interface, 5G network interface, and Wi-Fi network interface in the current environment through the network interface abstraction layer to establish a unified network resource pool.

[0072] In practice, the central server deploys a network interface abstraction layer. Once the XR terminal enters the signal coverage area and completes initial access, this abstraction layer automatically scans the physical network interface information reported by the XR terminal, identifying and registering the currently available millimeter-wave, 5G cellular, and Wi-Fi interfaces. Subsequently, the system reads the underlying attributes of each physical interface, such as MAC address, frequency band information, and physical layer protocol version, and encapsulates them into virtual network objects in a unified format. Based on this, the resource pool manager assigns a globally unique logical ID to each virtual network object and establishes a mapping table between this logical ID and the physical interface. In this way, when upper-layer video applications perform data scheduling, they only need to operate on the logical ID without needing to concern themselves with the underlying transmission medium, thus achieving pooled management of heterogeneous resources.

[0073] S2. Monitor the status parameters of each network interface in the network resource pool in real time, and aggregate and calculate the global capacity index that characterizes the system's transmission capability. In specific implementation, the resource pool manager periodically obtains the real-time status parameters of each network interface through the control channel, including Physical Rate, Received Signal Strength (RSSI / RSRP), and Signal-to-Noise Ratio (SNR).

[0074] Specifically, the steps for aggregate computing to characterize the global capacity index of the system's transmission capability include:

[0075] S21. Periodically collect raw network status data for each network interface, wherein the raw data includes at least the physical negotiation rate, received signal strength, signal-to-noise ratio, short-time packet loss rate, and round-trip time.

[0076] In practice, raw status data for each network interface i is collected at a preset period, such as every 50ms, including the physical negotiation rate R_phy. i Received signal strength RSS i Or reference signal received power RSRP i Signal-to-noise ratio (SNR) i Short-term packet loss rate (Loss) i Round-trip time (RTT) i .

[0077] S22. Normalize the received signal strength, signal-to-noise ratio, short-time packet loss rate and round-trip delay respectively, and perform time-domain smoothing filtering on the normalized values ​​to obtain the normalized index values ​​of each dimension.

[0078] In practical implementation, to eliminate the influence of different dimensions, the above indicators are mapped and normalized to the [0,1] interval respectively. Specifically, for RSS / RSRP and SNR, a linear mapping with upper and lower limits can be used; for packet loss rate, a 1-Loss mapping is used, meaning less packet loss is better; for RTT, a reciprocal mapping is used; and for instantaneous network fluctuations, an exponential moving average (EMA) smoothing filter is applied to the normalized values. In this embodiment, the EMA time constant τ can be set to 200ms to 500ms, filtering out millisecond-level glitches and jitters while balancing the sensitivity and stability of the indicators.

[0079] S23. According to the preset weighting strategy, the normalized index values ​​of each dimension are weighted and summed to calculate the link quality coefficient of each network interface.

[0080] In practice, a weighted sum method can be used to calculate the overall quality score Q of interface i. i Let the weights of each indicator be w1, w2, w3, and w4, respectively. In this preferred embodiment, the weights are configured as follows: signal strength weight w1 = 0.35, signal-to-noise ratio weight w2 = 0.35, packet loss rate weight w3 = 0.2, and latency weight w4 = 0.1. The calculation formula is:

[0081] Q i =w1·Norm(RSS)+w2·Norm(SNR)+w3·Norm(1—Loss)+w4·Norm(RTT -1 )

[0082] S24. Calculate the effective bandwidth of each network interface based on the physical negotiation rate, link quality coefficient, and preset protocol overhead coefficient.

[0083] In practice, the effective bandwidth of a single interface is calculated based on the physical rate and quality coefficient, after deducting protocol overhead:

[0084] B eff_i =R_phy i ×Q i ×(1—α overhead )

[0085] Where, α overhead The protocol and header overhead coefficient is preferably between 0.05 and 0.15.

[0086] S25. Select network interfaces with link quality coefficients greater than a preset quality threshold as available interfaces, sum the effective bandwidth of all available interfaces, and apply a safety margin coefficient to the summation result to obtain the global capacity index B. total .

[0087] In practice, the first step is to perform availability screening, only including Q... i ≥Q th Threshold Q th =0.6, and interfaces that pass stability testing are included in the available set; then, the effective bandwidth of all available interfaces is accumulated and multiplied by the safety margin factor Ms1, Ms1=0.8, to obtain the final system global capacity index reported externally:

[0088] B total =(∑B eff_i )×Ms1

[0089] The B total It will be smoothly output to the video encoder and scheduler in real time for dynamic adjustment of the video bitrate.

[0090] S3. Divide the XR video stream to be transmitted into multiple independent transmission units.

[0091] Specifically, the steps of dividing an XR video stream into multiple independent transmission units include:

[0092] S31. Perform Deep Packet Inspection (DPI) on the XR video stream, and parse the header information of the Network Abstraction Layer (NAL) to identify video frame boundaries and frame types. The frame types include at least keyframes and reference frames. Taking H.264 / AVC or H.265 / HEVC encoding format as an example, the nal_unit_type field in the NAL Unit Header is read, and the video frame boundaries and frame types are identified based on this field.

[0093] S32. According to the video coding boundary or the preset maximum transmission unit limit, several network abstraction layer units are aggregated and encapsulated into a transmission unit.

[0094] In practice, an aggregation buffer is maintained to merge several smaller NAL units belonging to the same video frame. The aggregation terminates when the size of the merged data is close to but does not exceed the maximum transmission unit of the current link. This step ensures that each transmission unit is a complete atomic operation during physical layer transmission, reducing the probability of packet loss caused by underlying fragmentation.

[0095] S33. For transmission units containing key frames, mark them as high priority and calculate and generate forward error correction redundancy information to encapsulate in the transmission unit; for transmission units containing reference frames, mark them as dynamic scheduling priority.

[0096] Specifically, a differentiated strategy is implemented based on the identified frame type:

[0097] High-priority channel: When a transmission unit is identified as containing keyframe data, the system marks its Priority field as "High" and applies forward error correction (FEC) coding to the keyframe data block. In a preferred embodiment, the RaptorQ fountain code algorithm or the Reed-Solomon (RS) algorithm is used to generate additional redundant check packets.

[0098] For example, the redundancy rate is set to r=1.5, which means that for 10 original I-frame data packets, 5 additional FEC redundant packets are generated. This means that even if 33% of the data packets are lost during transmission, the receiving end can still recover the complete I-frame without loss through the FEC algorithm, without triggering a high-latency retransmission request.

[0099] Dynamic scheduling channel: When it is detected that the transmission unit contains only reference frame data, the system marks its Priority field as "Dynamic". Such data is usually not subject to additional FEC redundancy, or is subject to a low proportion of redundancy, such as setting the redundancy rate r=1.1, in order to save bandwidth resources. Occasional packet loss is subsequently handled by Hybrid Automatic Repeat Request (HARQ) or application layer retransmission.

[0100] S34. Add an encapsulation header to each of the transmission units. The encapsulation header includes at least a global sequence number, a priority identifier, a timestamp, and a forward error correction (FEC) index for receiver reassembly and error recovery.

[0101] S4. Based on the status parameters of each network interface and the attributes of the transmission unit, a transmission path is dynamically allocated to each transmission unit using a multipath transmission protocol, and the transmission unit is sent through at least one selected network interface.

[0102] In practice, the dynamic allocation of transmission paths follows the following priority strategy:

[0103] S41. Set the millimeter wave network path as the highest priority, utilizing its ultra-high bandwidth and microsecond-level latency to serve as the default primary transmission channel under line-of-sight conditions; the 5G network path is the second priority, providing wide coverage and high stability; the Wi-Fi network path is the basic backup path, used to maintain a minimum level of connection keep-alive.

[0104] S42. Monitor the packet loss rate, signal-to-noise ratio change, round-trip delay increment, and link blockage probability of the current working path in real time; determine that the current working path has been degraded only when any of the above indicators exceeds the preset threshold and continues to meet the preset number of continuous monitoring cycles.

[0105] For example: Set the monitoring period T c =50ms, judgment threshold N=3. That is, a degradation judgment is triggered only when a certain degradation indicator exceeds the threshold for 3 consecutive cycles totaling 150ms; at the same time, the current working path is monitored in real time; taking millimeter wave as an example, when any of the following conditions are met and last for N cycles, the path is judged to be degraded:

[0106] Short-term packet loss rate exceeding the limit: A link layer packet loss rate loss of >5% was detected;

[0107] The signal-to-noise ratio has deteriorated drastically: the current SNR has decreased by more than 6 dB compared to the average value of the previous stable time window;

[0108] Anomaly in latency jitter: The current round-trip latency has increased by more than 10ms compared to the baseline value;

[0109] Geometric occlusion risk: The line-of-sight probability (PLOS) calculated based on IMU attitude and spatial map is less than 0.4.

[0110] S43. When the millimeter wave network path is downgraded, the candidate 5G network path and Wi-Fi network path are comprehensively ranked using a weighted scoring method, and the system switches to the network path with the highest score.

[0111] Specifically, after the main path (millimeter wave) is deemed downgraded, the optimal candidate path needs to be selected, and a weighted score is calculated based on real-time quality.

[0112] Score = W1 × (1 - Loss) avg )+W2×(1 / RTT norm )

[0113] Wherein, W1 and W2 are weighting coefficients. Preferably, W1=0.6 and W2=0.4.

[0114] If the 5G path score is significantly higher than Wi-Fi (e.g., 20% higher), and the effective bandwidth of 5G is B eff If the current video bitrate requirement is met, switch to 5G; if 5G is on the edge of weak signal (e.g., packet loss rate > 8% or score too low), downgrade directly to Wi-Fi path as a backup.

[0115] S44. When making a path switching decision, the switching action is only allowed if the running time of the current working path exceeds the preset minimum hold time and the time interval since the last switch exceeds the preset cooldown time.

[0116] Specifically, in order to completely eliminate ping-pong handover, the system performs the following two time constraint checks before making a handover decision:

[0117] Minimum hold time: Set T min_hold =500ms. Once any path is selected and activated, it will be forcibly locked within the next 500ms, even if there are slight fluctuations in the indicators, i.e., not reaching the level of disconnection, and active switching will be prohibited.

[0118] Switch cooldown time: Set T cooldown =300ms. Within 300ms after the previous path switching action is completed, non-fatal switching requests are blocked.

[0119] Understandably, in actual implementation, if the current link experiences a physical layer disconnection, the cooldown time is ignored and an emergency switchover is triggered immediately.

[0120] S5. At the receiving end, the transmission units arriving through different network interfaces are reassembled to restore a continuous video stream.

[0121] In practice, the specific steps of the restructuring include:

[0122] S51. Maintain a sliding reassembly window based on a global sequence number at the receiving end and set a maximum waiting latency threshold. In specific implementation, the maximum waiting latency threshold can be dynamically configured according to the current XR application scenario. For example, in low-latency competitive scenarios such as FPS games, it can be configured to 50ms; while in general immersive scenarios such as panoramic videos and virtual exhibitions, it can be configured to 150ms.

[0123] S52. Fill the out-of-order transmission units into the corresponding positions of the sliding reassembly window according to the global sequence number. If a packet with a duplicate sequence number is received, retain the earliest arriving one and discard the subsequent arriving copies.

[0124] S53. Perform timeout determination for missing transmission units within the window: If the current time exceeds the maximum waiting delay threshold and the transmission unit has not yet arrived, it is determined to be a timeout, and the subsequent recovery process is triggered.

[0125] S54. If the transmission unit that timed out is a high-priority transmission unit, forward error correction decoding recovery will be initiated first; if recovery fails, a fast retransmission request or an immediate decoding refresh request will be triggered; if the transmission unit that timed out is a dynamically scheduled priority transmission unit, time domain frame interpolation or motion compensation algorithm will be used to cover up the error, and a selective acknowledgment will be sent to the sending end to trigger background retransmission.

[0126] In this step, the priority identifier that the missing unit should have carried is parsed, and traffic splitting is performed:

[0127] Branch A: For high-priority transmission units

[0128] First-level recovery step (FEC decoding): First, retrieve the arrived Forward Error Correction (FEC) redundant packets in the same group, and perform a local XOR operation using the RaptorQ or Reed-Solomon algorithm to attempt to recover the lost original data. This process is extremely fast and requires no network interaction.

[0129] Secondary recovery steps (fast retransmission / IDR): If the FEC redundancy packets are insufficient to recover the data, the system immediately sends a high-priority fast retransmission request to the sender; if the estimated retransmission time will cause severe playback stuttering, the system directly requests the sender to generate a new Instant Decoding Refresh (IDR) frame to quickly cut off the error propagation chain.

[0130] Branch B: For dynamically scheduled priority transmission units

[0131] Level 1 recovery step (error masking): The receiving end abandons waiting for the data packet and directly notifies the video rendering layer. The rendering layer uses time-domain frame interpolation technology or motion compensation algorithm to generate a frame to fill the gap and prevent the screen from freezing.

[0132] Secondary correction step (SACK feedback): Although the current frame has been skipped, in order to prevent subsequent frames from accumulating errors due to missing references, the receiver still sends a selective acknowledgment signal to the sender. After receiving it, the sender will retransmit the data in the background to correct the reference queue of subsequent frames or embed more error correction information in subsequent frames.

[0133] S55. Statistically analyze the timeout frequency and retransmission rate, and feed them back to the sending end to drive the adaptive adjustment of the transmission unit size and redundancy ratio in the next stage. In practice, if the timeout frequency is high and the FEC success rate is low, the sending end will reduce the transmission unit size in the next stage to reduce the probability of single packet loss and increase the redundancy ratio; if the retransmission rate is low but network jitter is high, the sending end may appropriately increase the maximum waiting delay threshold value of the receiving end.

[0134] Finally, the rearranged and integrity-verified data packets are merged and restored into a continuous video stream, which is then pushed to the upper-layer rendering engine for display in real time.

[0135] In some embodiments, the method further includes a seamless handover mechanism based on the user's mobility state to avoid momentary connection interruptions caused by millimeter-wave signal blockage. This mechanism specifically includes:

[0136] 1. Inertial data acquisition and attitude calculation: Real-time acquisition of measurement data from the inertial measurement unit of the user's XR terminal, and low-pass filtering and attitude calculation are performed on the data to estimate the short-term displacement and orientation of the XR terminal.

[0137] In practice, the high-precision inertial measurement unit (IMU) built into the XR terminal outputs acceleration, angular velocity, and magnetic field strength data in real time, and performs synchronization alignment and low-pass filtering on the raw IMU signal. In this preferred embodiment, the recommended cutoff frequency is set to approximately 10Hz to filter out human body tremors and high-frequency noise from the sensor, while retaining the true human motion components.

[0138] Subsequently, using the Extended Kalman Filter (EKF) algorithm or a lightweight inertial navigation algorithm, combined with preset site constraints, the short-term displacement vector and head orientation of the terminal in space are estimated in real time. At the same time, based on the calculated terminal pose, the line-of-sight vector of the terminal antenna relative to the millimeter-wave access point (AP) and the deviation angle relative to the antenna beam centerline are calculated in real time, serving as the geometric basis for subsequent judgments.

[0139] 2. Predictive modeling of link blockage probability: Based on the short-term displacement and orientation, combined with the preset environmental model or historical signal change characteristics, the prediction model is used to calculate the link blockage probability and prediction confidence within the preset time window in the future.

[0140] Model input: Input features include, but are not limited to, IMU motion features (such as sudden changes in angular velocity and turning rate), the rate of change of historical received signal strength RSS, and the estimated distance between the terminal and the AP.

[0141] Environmental model integration: If the AP deployment locations and the locations of major obstructions (such as pillars and walls) are known, geometric occlusion detection is performed by combining the current geometric line-of-sight vector.

[0142] Predicted output: Using logistic regression or a lightweight neural network as the classifier, output the future time window T. pred The probability of sight distance P within the range LOS This refers to the probability that the link is not blocked and the prediction confidence level.

[0143] 3. Handover Trigger Decision Logic: The link occlusion probability is compared with a preset occlusion threshold, or the decrease in the current link quality coefficient is monitored. When the prediction model's prediction result within the preset future time window satisfies the link occlusion probability being higher than the occlusion threshold and the prediction confidence level meeting the standard, or when the instantaneous decrease in the current link quality coefficient exceeds a preset safety tolerance, the backup link establishment process is immediately triggered before the current link is disconnected, reserving handover margin through advance handshake.

[0144] Prediction-based early triggering: Setting the prediction time window T pred The timeframe is 200ms, meaning it predicts the state 200ms in advance. It triggers immediately when the prediction model output meets the following conditions:

[0145] P LOS ≤P th And confidence level ≥ C th

[0146] Among them, the occlusion threshold P th It can be set to 0.4, which means there is a greater than 60% probability of non-line-of-sight occlusion occurring; while the confidence threshold C th In this embodiment, it is set to 0.7.

[0147] Based on signal mutation response triggering: As a supplement to the prediction model, the quality coefficient Q of the current link can also be monitored at the same time; if a short-term drop in the Q factor ΔQ>0.2 is detected, and the downward trend continues for more than N monitoring cycles, for example N=3, it is determined to be a sudden physical blockage and is triggered immediately.

[0148] Once any of the above triggering conditions are met, the system will immediately initiate the process of establishing a 5G or Wi-Fi backup link before the current millimeter-wave link is disconnected.

[0149] To ensure that at least T is reserved before the expected shading occurs. reserveA time window (e.g., 100ms-300ms) is used to complete the handshake negotiation, resource allocation, and initialization of parallel transmission of the two links.

[0150] If the backup link is within the timeout period T establish If the connection is not established within 200ms-500ms (e.g.), the system will force a fallback to a conservative mode (e.g., reduce the bitrate) and attempt to connect to other candidate links. The system will also record the failure event for policy optimization.

[0151] After triggering and establishing the backup link, the system enters dual-link parallel transmission. The specific steps of the dual-link parallel transmission include:

[0152] First, the same transmission unit is copied into two copies and sent in parallel through the current link and the backup link, respectively.

[0153] Subsequently, the receiving end performs integrity checks (CRC / HMAC) on the data packets. If only one copy passes the check, it is used directly. If both data packets from both links pass the check, the arrival delay, signal-to-noise ratio snapshot, or bit error rate soft information are further compared, and the copy of the data packet with the better link quality is selected as valid data. Specifically, by reading the link snapshot information and physical layer bit error rate soft information carried in the data packet header, the copy with lower arrival delay and higher link signal-to-noise ratio / bit error rate is prioritized and submitted as valid data to the video decoder.

[0154] The communication quality of the backup link is monitored in real time. The verification is deemed successful only when the effective bandwidth, packet loss rate, and round-trip delay of the backup link simultaneously meet the preset quality thresholds, and this satisfactory state continues for a preset number of monitoring periods.

[0155] In practice, the verification is considered successful only when the backup link meets all of the following conditions and the duration reaches the preset number of monitoring cycles N, for example, N=5 cycles, or 250ms:

[0156] Ample effective bandwidth: Real-time effective bandwidth B of the backup link eff_new It must be greater than γ times the video bitrate requirement. In this embodiment, a safety factor γ=1.2 is set, that is, a 20% bandwidth margin is reserved to cope with fluctuations.

[0157] Packet loss rate meets target: Short-term packet loss rate on backup links. new ≤3%.

[0158] Controllable latency: Average round-trip time (RTT) new ≤50ms.

[0159] After the verification passes, the data transmission weight of the current link is gradually reduced until transmission stops completely, completing the handover. During this process, the transmission of low-priority P / B frames on the old link is first stopped, and only the dual transmission of key frames (I frames) is retained to reduce the overall system throughput load; if no retransmission request is received from the receiver for several consecutive cycles (e.g., 100ms), data transmission on the old link is completely stopped, releasing its physical resources.

[0160] After the switching action is completed, the system enters a state of length T. monitor =300ms-500ms performance monitoring period; if the new link (original backup link) experiences a sharp performance decline during this period (such as a sudden increase in packet loss rate to more than 10%), a rollback will be triggered immediately to reactivate the old link or trigger a reconnection process, thereby minimizing the impact of network fluctuations on user experience.

[0161] Combination Figure 2 The seamless switching timing diagram shown is as follows:

[0162] First, at time t0, the system did not wait for the signal to actually deteriorate. Instead, it detected the link risk in advance by analyzing the IMU inertial measurement data and attitude calculation results reported by the XR terminal. Specifically, at this time, the geometric relationship between the user's line-of-sight vector and the obstruction calculated by the network resource pool manager changed, and the occlusion probability output by the prediction model began to rise. Therefore, it was determined that the current main link (old link) was about to enter an unstable state.

[0163] Subsequently, during the period from t0 to t1, the system continuously tracks the prediction confidence. When the pre-switching is triggered at time point t1, the prediction metric exceeds the preset threshold, for example, the predicted occlusion probability P within the next 200ms. LOS If the value is greater than 0.4 and the confidence level meets the standard, the system immediately instructs the XR terminal to initiate a connection request to the backup link (new link). At this time, the old link continues to transmit data normally to reserve sufficient handshake negotiation time.

[0164] After a brief handshake establishment process, at time point t2 (dual-link parallel), the new link connection is successful. During this stage, the server performs a dual-transmission operation: while the old link continues to send video stream data packets, the new link simultaneously sends video stream data packets with the same sequence number. The XR terminal receives both data streams simultaneously and performs a hierarchical optimization decision based on the global sequence number allocated in step S31 to ensure that the data stream obtained by the upper-layer application is continuous and error-free.

[0165] At time t3, the system detects that the channel quality indicators (such as SNR or packet loss rate) of the new link have met the preset stability threshold for M consecutive cycles (e.g., 250ms). After confirming that it can independently carry the complete video stream, it enters the traffic migration phase, that is, the system begins to gradually reduce the data transmission weight of the old link. For example, it first stops the transmission of low-priority P / B frames and only retains the dual transmission of key frames, instead of immediately cutting it off.

[0166] Finally, at time t4, after a preset monitoring period confirming no performance rollback of the new link, the system sends a command to the XR terminal to disconnect the physical connection of the old link and release the logical resources occupied by the old link in the resource pool. The entire process achieves a seamless transition with zero interruption for the user.

[0167] In some embodiments, the method further includes a rapid authentication step:

[0168] When a user's XR device first accesses the network resource pool, the network resource pool's management terminal generates a global session token containing a user's permission set, validity period, and digital signature based on a preset encryption key. Specifically, the token uses a lightweight JSON Web Token format, and its payload includes a unique user identifier, a quality of service permission set, a random number, and a precise validity period to reduce security risks. Simultaneously, the network resource pool's management terminal signs the token using a pre-set site private key or a high-strength HMAC key to prevent forgery.

[0169] The global session token is synchronously distributed to the local storage units of all candidate network access devices within the coverage area. In specific implementations, it can be pushed to the edge via two parallel logical channels:

[0170] Backbone multicast channel: Utilizing the wired backbone network of the local area network, the latest token is broadcast to all Wi-Fi APs and 5G micro base stations within the same physical venue via IP multicast technology.

[0171] Distributed cache synchronization channel: Utilize the Pub / Sub mechanism or master-slave replication mechanism of in-memory databases such as Redis Cluster to ensure that the local cache of all edge nodes completes data consistency synchronization within milliseconds.

[0172] When an XR terminal initiates a network handover request, it carries the global session token in the handshake message. For example, when a user switches from millimeter wave to 5G or Wi-Fi, the terminal carries this token in a specific field of the link layer handshake message.

[0173] The target network access device uses data in its local storage unit or a pre-set public key to perform offline verification of the global session token; if the signature verification passes and the token is within its validity period, it directly inherits the user session state and establishes a connection, skipping the complete identity negotiation process.

[0174] Specifically, after receiving the request, the target base station / AP does not query the central server, but directly queries the token in the local cache, or uses the locally stored public key to digitally verify the token signature; once the verification is successful and it is confirmed that the token has not expired and is not in the revocation list, the target base station immediately trusts the user, inherits its original QoS policy and allocates air interface resources, thereby skipping the complete identity negotiation process.

[0175] If it is necessary to force a user offline, the network resource pool management maintains a short-term revocation list or updates the global token version number, and synchronizes it to all edge nodes in real time through the above broadcast mechanism. The edge nodes then intercept expired tokens in real time.

[0176] In some embodiments, the method further includes the step of dynamically adjusting video stream parameters based on the network resource pool status:

[0177] The global capacity metric, i.e., the total available bandwidth capacity, is compared with the bitrate of the current video stream.

[0178] When the global capacity index is lower than the product of the current video stream's bitrate and the safety margin Ms2, the encoding parameters for generating the XR video stream are adjusted to reduce the video bitrate or resolution, thereby prioritizing smoothness and avoiding stuttering when network resources are limited. In this embodiment, Ms2 = 1.25.

[0179] The method and system for video transmission based on a hybrid network of millimeter wave, 5G, and Wi-Fi provided in this application construct a unified resource pool for millimeter wave, 5G, and Wi-Fi through a network interface abstraction layer. It also dynamically allocates paths to transmission units using a multi-path transmission protocol combined with video stream segmentation, achieving aggregation and parallel transmission of heterogeneous network bandwidth resources. Compared to a single network or a simple primary / backup solution, this method overcomes the physical bandwidth limitations of a single frequency band, effectively solving the problems of spectrum congestion and signal obstruction in large-space, multi-user scenarios, significantly improving the system's total throughput and transmission stability, thereby supporting concurrent transmission of higher-resolution XR video streams.

[0180] In the description of this application, it should be noted that the terms "vertical", "up", "down", "horizontal", etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this application.

[0181] In the description of this application, it should also be noted that, unless otherwise expressly specified and limited, the terms "set," "install," "connect," and "link" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this application according to the specific circumstances.

[0182] Finally, it should be noted that the above descriptions are merely preferred embodiments of this application and are not intended to limit this application. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A method for video transmission based on a hybrid network of millimeter wave, 5G, and Wi-Fi, characterized in that, Includes the following steps: S1. Register the millimeter-wave network interface, 5G network interface and Wi-Fi network interface in the current environment through the network interface abstraction layer to establish a unified network resource pool; S2. Monitor the status parameters of each network interface in the network resource pool in real time, and aggregate and calculate the global capacity index that characterizes the system's transmission capability; S3. Divide the XR video stream to be transmitted into multiple independent transmission units; S4. Based on the status parameters of each network interface and the attributes of the transmission unit, a transmission path is dynamically allocated to each transmission unit using a multipath transmission protocol, and the transmission unit is sent through at least one selected network interface. S5. At the receiving end, the transmission units arriving through different network interfaces are reassembled to restore a continuous video stream; The method also includes a seamless switching mechanism based on the user's mobility state, specifically including: The system collects measurement data from the inertial measurement unit of the XR terminal worn by the user in real time, and performs low-pass filtering and attitude calculation on the data to estimate the short-term displacement and orientation of the XR terminal. Based on the short-term displacement and orientation, combined with the preset environmental model or historical signal change characteristics, the prediction model is used to calculate the link blockage probability and prediction confidence within the future preset time window. The link occlusion probability is compared with a preset occlusion threshold. When the prediction result of the prediction model within the future preset time window satisfies the link occlusion probability being higher than the occlusion threshold and the prediction confidence level being met, the backup link establishment process is immediately triggered before the current link is disconnected, reserving a handshake margin in advance.

2. The method for video transmission based on a hybrid network of millimeter wave, 5G, and Wi-Fi according to claim 1, characterized in that, After triggering and establishing a backup link, dual-link parallel transmission is initiated. The specific steps of the dual-link parallel transmission include: The same transmission unit is copied into two copies, which are then transmitted in parallel through the current link and the backup link, respectively. The receiving end performs integrity verification on the data packets; if the data packets from both links pass the verification, it further compares the arrival delay, signal-to-noise ratio snapshot, or bit error rate soft information, and selects the data packet copy with better link quality as valid data. The communication quality of the backup link is monitored in real time. The verification is deemed successful only when the effective bandwidth, packet loss rate and round-trip delay of the backup link simultaneously meet the preset quality thresholds and the satisfaction status continues for a preset number of monitoring periods. After the verification is passed, the data transmission weight of the current link is gradually reduced until transmission stops completely, and the switchover is completed; if a performance regression is detected during the monitoring period after the switchover, the system is immediately rolled back to the current link.

3. The method for video transmission based on a hybrid network of millimeter wave, 5G, and Wi-Fi according to claim 1, characterized in that, Step S2, specifically the step of aggregating and calculating the global capacity index characterizing the system's transmission capability, includes: S21. Periodically collect raw network status data for each network interface, wherein the raw data includes at least physical negotiation rate, received signal strength, signal-to-noise ratio, short-time packet loss rate, and round-trip time. S22. Normalize the received signal strength, signal-to-noise ratio, short-time packet loss rate and round-trip delay respectively, and perform time-domain smoothing filtering on the normalized values ​​to obtain the normalized index values ​​of each dimension. S23. According to the preset weighting strategy, the normalized index values ​​of each dimension are weighted and summed to calculate the link quality coefficient of each network interface; S24. Calculate the effective bandwidth of each network interface based on the physical negotiation rate, link quality coefficient, and preset protocol overhead coefficient; S25. Select network interfaces with a link quality coefficient greater than a preset quality threshold as available interfaces, accumulate the effective bandwidth of all available interfaces, and apply a safety margin coefficient to the accumulation result to obtain the global capacity index.

4. The method for video transmission based on a hybrid network of millimeter wave, 5G, and Wi-Fi according to claim 1, characterized in that, Step S3, which involves dividing the XR video stream into multiple independent transmission units, specifically includes: S31. Perform deep packet inspection on the XR video stream and parse the header information of the network abstraction layer unit to identify video frame boundaries and frame types, wherein the frame types include at least keyframes and reference frames; S32. According to the video coding boundary or the preset maximum transmission unit limit, several network abstraction layer units are aggregated and encapsulated into a transmission unit; S33. For transmission units containing key frames, mark them as high priority and calculate and generate forward error correction redundancy information to encapsulate into the transmission unit; for transmission units containing reference frames, mark them as dynamic scheduling priority. S34. Add an encapsulation header to each of the transmission units, wherein the encapsulation header includes at least a global sequence number, a priority identifier, a timestamp, and a forward error correction index for receiver reassembly and error recovery.

5. The method for video transmission based on a hybrid network of millimeter wave, 5G, and Wi-Fi according to claim 1, characterized in that, In step S4, the dynamic allocation of transmission paths follows the following priority strategy: S41. Set the millimeter wave network path as the highest priority, the 5G network path as the second priority, and the Wi-Fi network path as the basic guarantee path; S42. Monitor the packet loss rate, signal-to-noise ratio change, round-trip delay increment, and link blockage probability of the current working path in real time; determine that the current working path has been degraded only when any of the above indicators exceeds the preset threshold and continues to meet the preset number of consecutive monitoring periods. S43. When the millimeter wave network path is downgraded, the candidate 5G network path and Wi-Fi network path are comprehensively ranked using a weighted scoring method, and the network path with the highest score is switched to. S44. When making a path switching decision, the switching action is only allowed if the running time of the current working path exceeds the preset minimum hold time and the time interval since the last switch exceeds the preset cooldown time.

6. The method for video transmission based on a hybrid network of millimeter wave, 5G, and Wi-Fi according to claim 1, characterized in that, The method also includes a rapid authentication step: When the XR terminal worn by the user first accesses the network resource pool, the management terminal of the network resource pool generates a global session token containing the user's permission set, validity period and digital signature based on a preset encryption key. The global session token is synchronously distributed to the local storage units of all candidate network access devices within the coverage area; When an XR terminal initiates a network handover request, it carries the global session token in the handshake message; The target network access device uses data in its local storage unit or a pre-set public key to perform offline verification of the global session token; if the signature verification passes and the token is within its validity period, it directly inherits the user session state and establishes a connection, skipping the complete identity negotiation process.

7. The method for video transmission based on a hybrid network of millimeter wave, 5G, and Wi-Fi according to claim 6, characterized in that, The method further includes: Monitor the rate of decline in the current link quality coefficient; When the instantaneous drop in the quality coefficient of the current link exceeds the preset safety tolerance, the establishment process of the backup link is triggered immediately before the current link is disconnected, and a handshake is performed in advance to reserve the switching margin.

8. The method for video transmission based on a hybrid network of millimeter wave, 5G, and Wi-Fi according to claim 1, characterized in that, In step S5, the specific steps of recombination include: S51. Maintain a sliding reassembly window based on the global sequence number at the receiving end, and set a maximum waiting delay threshold; S52. Fill the out-of-order transmission units into the corresponding positions of the sliding reassembly window according to the global sequence number; S53. Perform timeout determination for missing transmission units within the window: if the current time exceeds the maximum waiting delay threshold and the transmission unit has not yet arrived, it is determined to be a timeout; S54. If the transmission unit that timed out is a high-priority transmission unit, forward error correction decoding recovery will be initiated first; if recovery fails, a fast retransmission request or an immediate decoding refresh request will be triggered; if the transmission unit that timed out is a dynamically scheduled priority transmission unit, time domain frame interpolation or motion compensation algorithm will be used to cover up the error, and a selective acknowledgment will be sent to the sending end to trigger background retransmission. S55. Calculate the timeout frequency and retransmission rate, and feed them back to the transmitting end to drive the adaptive adjustment of the transmission unit size and redundancy ratio in the next stage.

9. The method for video transmission based on a hybrid network of millimeter wave, 5G, and Wi-Fi according to claim 1, characterized in that, It also includes the step of dynamically adjusting video stream parameters based on the network resource pool status: Compare the global capacity metric with the bitrate of the current video stream; When the global capacity index is lower than the product of the current video stream's bitrate and safety margin, the encoding parameters for generating the XR video stream are adjusted to reduce the video bitrate or resolution.

10. A system for video transmission based on a hybrid network of millimeter wave, 5G, and Wi-Fi, characterized in that, The system includes a processor, a communication interface, and a memory. The memory stores a computer program, which, when executed by the processor, implements the method for video transmission based on a hybrid network of millimeter wave, 5G, and Wi-Fi as described in any one of claims 1 to 9.

Images