Ultrahigh-definition video stream multi-mode low-delay division method based on dynamic resource scheduling

Abstract

The invention discloses an ultra-high-definition video stream multi-mode low-delay division method based on dynamic resource scheduling. A probe module is deployed through an edge computing node, multi-modal data such as terminal decoding computing power and rendering capacity are collected in real time, a dynamic terminal capacity portrait is constructed, and low-delay division of an ultra-high-definition video stream is achieved in combination with a double-target optimization model. And by utilizing decoding time-consuming simulation and rendering buffer area state verification, dynamically evaluating a slice lagging risk and generating negative feedback, driving dual-target weight coefficient adaptive adjustment, and optimizing code rate allocation and path selection. A video stream is divided into a high motion sensitive area and a static background area, low-delay and high-compression path transmission is adopted respectively, and real-time performance and bandwidth efficiency are both considered. And low-delay and high-quality division of the ultra-high-definition video stream at the multi-mode terminal is realized.

Description

Technical Field

[0001] The present application relates to the field of video stream transmission technology, and in particular to a method for multimodal low-latency partitioning of ultra-high-definition video streams based on dynamic resource scheduling. Background Art

[0002] With the rapid development of ultra-high-definition video technology, users are increasingly demanding a higher quality viewing experience. They not only pursue higher video clarity but also place stringent standards on the real-time and smooth delivery of video. Ultra-high-definition video streams are characterized by large data volumes and high bit rates. When segmenting across multimodal terminals, they face challenges such as large variations in terminal capabilities and complex and changing network environments. These issues can easily lead to excessive transmission delays, inconsistent video quality, and playback lag.

[0003] CN109685802B provides a low-latency video segmentation real-time preview method, which is mainly aimed at single-device video segmentation preview scenarios and does not take into account the heterogeneous characteristics of multi-modal terminals. This method does not involve the collection and adaptation of terminal decoding computing power, rendering buffer saturation and other capability parameters, and cannot adjust the processing strategy according to terminal differences. Its transmission mechanism does not distinguish between high-motion-sensitive areas and static background areas of the video, and adopts a single transmission path, which is difficult to meet the low-latency requirements in complex network environments. At the same time, there is a lack of a mechanism for dynamically adjusting bit rate allocation and routing paths based on network status, and there is no feasibility verification and negative feedback adjustment link on the terminal side. It cannot solve the problems of poor video quality consistency and large delay fluctuations in multi-terminal scenarios, and it is difficult to meet the efficient segmentation requirements of ultra-high-definition video streams in multi-modal terminals. Summary of the Invention

[0004] In order to solve the problems raised in the background technology, this application proposes a multimodal low-latency division method for ultra-high-definition video streams based on dynamic resource scheduling. This method can perceive the terminal capabilities and network status in real time, and realize low-latency and high-quality division of ultra-high-definition video streams in multimodal terminals by dynamically adjusting the bit rate allocation and routing paths.

[0005] The technical solutions of this application are as follows:

[0006] According to one aspect of the present application, a method for multimodal low-latency partitioning of ultra-high-definition video streams based on dynamic resource scheduling is provided, comprising:

[0007] Probe modules are deployed at edge computing nodes to collect multimodal terminal data in real time. Based on the collected multimodal terminal data, a terminal capability profile model is constructed to determine the comprehensive capability index threshold for each terminal to maintain the target frame rate in the future time domain, generating a dynamically updated terminal capability profile.

[0008] Using terminal capability profiles, combined with real-time network bandwidth and packet loss rate data, a dual-objective optimization model is established: the first objective is to minimize end-to-end transmission delay, constrained by the terminal's comprehensive capability index threshold; the second objective is to maximize multi-terminal video quality consistency, constrained by the edge node's real-time transcoding resource margin. The model outputs the optimal bitrate allocation for each terminal, as well as low-latency and high-compression paths.

[0009] The ultra-high-definition video stream is divided into slices of high-motion-sensitive areas and static background slices. For the high-motion-sensitive slices, the bit rate is allocated according to the optimal bit rate allocation scheme and transmitted over a low-latency path. For the static background slices, the bit rate is allocated according to the optimal bit rate allocation scheme and transmitted over a high-compression path.

[0010] The decoding computing power of the terminal capability portrait is used to simulate the decoding time, and the slice jamming risk is verified in combination with the rendering buffer status. The slice jamming risk value is generated, and the dual-objective weight coefficient is adjusted based on the slice jamming risk value to regenerate the optimal bitrate allocation plan, as well as the low-latency path and high-compression path.

[0011] As a further optimization of this application, the probe module adopts a lightweight architecture and is embedded in the underlying system of the edge computing node. It obtains the decoding computing power of the terminal device by calling the interface, monitors the filling status of the rendering buffer through the video rendering framework to evaluate the rendering capability, obtains the network jitter tolerance by measuring the data transmission delay fluctuations between the terminal and the edge node, and obtains the screen color gamut parameters through the terminal display management interface.

[0012] As a further optimization of the present application, the multimodal terminal feature extraction includes: calculating the decoder load rate, rendering buffer saturation, network jitter tolerance and color gamut coverage of the terminal;

[0013] Among them, the decoder load rate is the ratio of the actual running time of the decoder to the total running time, the rendering buffer saturation is the ratio of the current buffer size to the maximum capacity, the network jitter tolerance is the ratio of the network delay fluctuation to the average delay, and the color gamut coverage is the ratio of the terminal color gamut area to the standard color space color gamut area.

[0014] As a further optimization of this application, the construction formula of the terminal capability portrait model is:

[0015] ;

[0016] in, is the comprehensive capability index, is the characteristic of decoding computing power, Characteristic of rendering capability, is the characteristic of network jitter tolerance, is the characteristic of the screen color gamut parameters, is the dynamic adjustment coefficient, is the performance change rate within the time window;

[0017] By weighting and fusing the static comprehensive capability index with the dynamically predicted comprehensive capability index threshold, a dynamically updated terminal capability profile is generated. The calculation formula is:

[0018] ;

[0019] in, It is the terminal capability portrait index, is the dynamic weight coefficient, which is calculated as follows: , is the adjustment coefficient, is the terminal performance volatility, is the comprehensive capability index, is the comprehensive capability index threshold;

[0020] Among them, the generated terminal capability portrait is presented in the form of a multi-dimensional capability index, including: terminal capability portrait index, characteristics of decoding computing power, characteristics of rendering capability, characteristics of network jitter tolerance, and characteristics of screen color gamut parameters.

[0021] As a further optimization of this application, the first objective function of the dual-objective optimization model is:

[0022] ;

[0023] in, represents the end-to-end transmission delay, Indicates terminal The video streaming delay, Indicates the edge node to the terminal The delay introduced by transcoding the video stream, Indicates the network link transmission delay, is the total number of terminals;

[0024] The first objective function constraint is: comprehensive capability index ≥ terminal comprehensive capability index threshold;

[0025] The second objective function is:

[0026] ;

[0027] in, Indicates the consistency of video quality across multiple terminals. is the average bit rate of all terminals, Indicates terminal The difference between the bit rate and the average bit rate;

[0028] The second objective function constraint is ,in, For terminal The transcoding resources required for the video stream, The total amount of available transcoding resources on edge nodes.

[0029] As a further optimization of the present application, the step of generating the optimal bit rate allocation scheme includes:

[0030] Determine the bitrate cap based on the comprehensive capability index threshold of the terminal capability profile model;

[0031] A multi-objective optimization algorithm is used to iteratively generate candidate solutions, giving priority to satisfying the terminal comprehensive capability index threshold constraint;

[0032] Dynamically adjust the dual-objective weight coefficients, giving priority to reducing transmission delay when network delay fluctuates greatly, and giving priority to ensuring video quality consistency when terminal decoding computing power fluctuates greatly.

[0033] As a further optimization of the present application, the low-latency path selection for the high motion-sensitive area slice includes:

[0034] Prioritize 5G direct links or fiber-optic transmission paths with the lowest end-to-end transmission latency;

[0035] Ensure that the path bandwidth is sufficient and has low fluctuations;

[0036] Forward error correction technology is introduced to reduce retransmission delays caused by network packet loss.

[0037] As a further optimization of this application, the high compression path selection of the static background slice includes:

[0038] Prioritize transmission paths that support high-compression encoding;

[0039] Use edge node caching technology to reduce repeated transmission bandwidth consumption;

[0040] Long-term reference frame technology is introduced to reuse static background frames to improve compression efficiency.

[0041] As a further optimization of this application, the calculation formula of the slice jam risk value is:

[0042] ;

[0043] in, Indicates the slice jam risk value, and are the weight coefficients of decoding time and rendering buffer saturation, Indicates the decoding time. Indicates the maximum allowed decoding time of the terminal. Indicates the saturation of the render buffer, is the maximum allowed saturation of the buffer.

[0044] As a further optimization of this application, the dynamic adjustment formula of the dual-objective weight coefficient is:

[0045] ;

[0046] in, represents the adjusted weight coefficient, represents the weight coefficient before adjustment, is the adjustment coefficient, is the slice jam risk value.

[0047] The beneficial effects of this application are:

[0048] Advantage (1): Multimodal data fusion and dynamic profiling improve the adaptability of resource allocation.

[0049] This application uses a probe module to collect the terminal's decoding computing power, rendering capabilities, network jitter tolerance, and screen color gamut parameters in real time, builds a multi-dimensional terminal capability portrait model, and dynamically quantifies the terminal's performance in different scenarios. This multimodal data fusion mechanism can accurately perceive the real-time capability status of the terminal, and combine the time series prediction model to predict the terminal's comprehensive capability index threshold in the future time domain, thereby dynamically adjusting the bit rate allocation and transmission path. Compared with traditional static resource allocation strategies, this application can adapt to terminal performance fluctuations and network status changes, significantly improve the flexibility and accuracy of resource allocation, and ensure efficient matching of video stream transmission and terminal capabilities.

[0050] Advantage (2): Dual-objective optimization and content-aware partitioning strategies take into account both transmission efficiency and image quality consistency.

[0051] By establishing a dual-objective optimization model centered around minimizing end-to-end latency and maximizing multi-terminal video quality consistency, this application achieves a multi-objective balance within resource constraints. The model dynamically generates an optimal bitrate allocation scheme based on terminal capability profiles and real-time network status, and divides the video stream into high-motion-sensitive areas and static background areas based on video content analysis. This content-aware, differentiated transmission strategy not only reduces transmission latency in highly dynamic areas but also optimizes bandwidth utilization in static areas through high-compression encoding, thereby balancing transmission efficiency and image quality consistency in a multi-terminal environment and improving the overall playback experience.

[0052] Advantage (3): The closed-loop feedback mechanism effectively responds to performance fluctuations and changes in network status.

[0053] This application has a feedback mechanism that dynamically evaluates the risk of video stream slicing jams through decoding time simulation and rendering buffer status verification. When a decoding timeout or buffer overflow risk exceeds a threshold, the system generates a negative feedback vector carrying resource gap parameters, drives the dual-objective optimization model to dynamically adjust the weight coefficient and bitrate allocation scheme, and triggers edge node routing path switching. This mechanism ensures that the system can respond in real time to emergencies such as decreased terminal decoding capabilities and increased network jitter, proactively optimize transmission strategies, significantly reduce the probability of jams, and improve the stability and smoothness of video playback. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 The figure is a schematic diagram of the overall process of the multimodal low-latency partitioning method for ultra-high-definition video streams based on dynamic resource scheduling; Figure 2 Detailed flowchart of step S100 of the method for multimodal low-latency partitioning of ultra-high-definition video streams based on dynamic resource scheduling; Figure 3 Detailed flowchart of step S200 of the method for multimodal low-latency partitioning of ultra-high-definition video streams based on dynamic resource scheduling; Figure 4 Detailed flowchart of step S300 of the method for multimodal low-latency partitioning of ultra-high-definition video streams based on dynamic resource scheduling; Figure 5 This is a detailed flowchart of step S400 of the method for multimodal low-latency partitioning of ultra-high-definition video streams based on dynamic resource scheduling. DETAILED DESCRIPTION

[0055] The following will be combined with the accompanying drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making any creative efforts are within the scope of protection of this application.

[0056] In existing technologies, the division of ultra-high-definition video streams usually adopts the method of fixed bit rate allocation and static routing path selection, which is difficult to dynamically adjust according to the real-time capabilities of the terminal and the network status. For example, some terminals have limited decoding computing power and cannot carry high-bitrate video streams, which is prone to freezes caused by decoding timeouts; and when the network bandwidth fluctuates, the fixed routing path may not be able to guarantee the stability of transmission delay, affecting the user experience. In addition, the existing methods have shortcomings in handling the consistency of video quality across multiple terminals. It is difficult to ensure that the video viewing quality of different terminals is at a balanced level while meeting the delay requirements of each terminal. To solve the above problems, please refer to Figure 1, which shows a method for multimodal low-latency partitioning of ultra-high-definition video streams based on dynamic resource scheduling provided by an embodiment of the present application, the method comprising:

[0057] S100: Deploy probe modules at edge computing nodes to collect multimodal terminal data in real time. Based on the collected multimodal terminal data, build a terminal capability profile model, determine the comprehensive capability index threshold for each terminal to maintain the target frame rate in the future time domain, and generate a dynamically updated terminal capability profile.

[0058] S200: Leveraging terminal capability profiling, combined with real-time network bandwidth and packet loss rate data, a dual-objective optimization model is established. The first objective is to minimize end-to-end transmission delay, with the terminal comprehensive capability index threshold as the constraint. The second objective is to maximize the consistency of multi-terminal video quality, with the edge node's real-time transcoding resource margin as the constraint. The model outputs the optimal bitrate allocation plan for each terminal, as well as the low-latency path and high-compression path.

[0059] S300: Divide the ultra-high-definition video stream into high-motion-sensitive area slices and static background slices. For the high-motion-sensitive slices, the bit rate is allocated according to the optimal bit rate allocation scheme and transmitted using a low-latency path. For the static background slices, the bit rate is allocated according to the optimal bit rate allocation scheme and transmitted using a high-compression path.

[0060] Step S400: Use the decoding computing power of the terminal capability portrait to simulate the decoding time, combine the rendering buffer status to verify the slice jamming risk, generate a slice jamming risk value, adjust the dual-objective weight coefficient based on the slice jamming risk value, and regenerate the optimal bit rate allocation plan, as well as the low-latency path and high-compression path.

[0061] In the multimodal low-latency partitioning method for ultra-high-definition video streams based on dynamic resource scheduling, S100 deploys probe modules at edge computing nodes to collect multimodal terminal data in real time. It combines terminal decoding computing power, rendering capabilities, network jitter tolerance, screen color gamut and other characteristics to predict the maximum bearable bitrate threshold for maintaining the target frame rate in the future, and builds a dynamically updated terminal capability portrait.

[0062] Please refer to Figure 2 , which shows a flowchart of an exemplary method S100 for multimodal low-latency partitioning of ultra-high-definition video streams based on dynamic resource scheduling of the present application, including:

[0063] S110: Deploy probe modules and collect multimodal terminal data.

[0064] The probe module is responsible for collecting multimodal terminal data in real time, including decoding computing power, rendering capabilities, network jitter tolerance, and screen color gamut parameters.

[0065] In one possible implementation, the probe module adopts a lightweight architecture and is embedded in the underlying system of the edge computing node to minimize the occupation of system resources.

[0066] Specifically, the probe module monitors the terminal device's operating status and uses the system call interface to obtain decoding computing power. The probe module uses the video rendering framework to monitor the rendering buffer's fill status and record the buffer's occupancy rate to assess the terminal device's rendering capabilities. The probe module measures the fluctuations in data transmission latency between the terminal device and the edge computing node to obtain network jitter tolerance. The probe module also uses the terminal device's display management interface to obtain screen color gamut parameters.

[0067] S120: Extracting multimodal terminal features based on the collected multimodal terminal data.

[0068] Feature extraction is performed on the collected multimodal terminal performance data to further quantify the key performance indicators of the terminal and build key data support for subsequent terminal capability profiling.

[0069] In a possible implementation, based on the characteristics of decoding computing power, the decoder load rate of the terminal is calculated, that is, calculated by the ratio of the actual running time of the decoder to the total running time, so as to evaluate its decoding capability.

[0070] In a possible implementation, the rendering buffer saturation is calculated based on the characteristics of the rendering capability, that is, calculated by the ratio of the current buffer size to the maximum capacity to evaluate the degree to which it is close to saturation.

[0071] In a possible implementation, the network jitter tolerance is calculated based on a characteristic of the network jitter tolerance, that is, the network jitter tolerance is calculated by the ratio of the network delay fluctuation of the terminal to the average network delay of the terminal, so as to evaluate the network jitter tolerance characteristic.

[0072] In a possible implementation, the color gamut coverage of the terminal is calculated based on the characteristics of the screen color gamut parameters, that is, calculated by the ratio of the current color gamut area of ​​the terminal to the color gamut area of ​​the standard color space to evaluate the screen color gamut parameter characteristics.

[0073] S130: Build a terminal capability portrait model.

[0074] Based on the multimodal terminal features extracted from the S120, a terminal capability profile model is constructed to quantify the terminal's performance in different video playback scenarios. This model integrates characteristics such as the terminal's decoding power, rendering buffer saturation, network jitter tolerance, and screen color gamut to create a multidimensional data model that dynamically reflects the terminal's capabilities.

[0075] In one possible implementation, the terminal capability portrait model constructed is:

[0076] ;

[0077] in, is the comprehensive capability index, is the characteristic of decoding computing power, Characteristic of rendering capability, is the characteristic of network jitter tolerance, is the characteristic of the screen color gamut parameters, is the dynamic adjustment coefficient, is the performance change rate within the time window.

[0078] S140: Determine a comprehensive capability index threshold of the terminal when maintaining a target frame rate in a future time domain through a terminal capability profile model.

[0079] Based on the terminal capability profile model constructed by S130, the comprehensive capability index threshold for the terminal to maintain the target frame rate in the future time domain is determined. The comprehensive capability index threshold is determined by analyzing the terminal's decoding capabilities, rendering buffer status, network jitter tolerance, and screen color gamut characteristics, and combining it with the dynamic changes in multimodal terminal characteristics to determine the comprehensive capability index threshold.

[0080] In one possible implementation, based on the terminal capability profile model, a time series prediction model is used to predict the comprehensive capability index threshold in the future time domain. The input of the model includes a sequence of historical comprehensive capability indices, multimodal terminal characteristics, and target frame rate requirements. The output is the minimum comprehensive capability index threshold required to maintain the target frame rate in the future time domain. For example, if the target frame rate is 60fps, the model needs to predict whether the terminal can maintain a comprehensive capability index ≥ the minimum comprehensive capability index threshold in the next T seconds to ensure smooth video playback and stable image quality.

[0081] S150: Generate a terminal capability profile.

[0082] Based on the terminal capability portrait model constructed by S130 and the comprehensive capability index threshold predicted by S140, a multi-dimensional indicator system for quantifying the terminal capability status, namely the terminal capability portrait, is constructed by integrating the static performance characteristics of the terminal with the dynamic prediction results.

[0083] In one possible implementation, the process of generating a terminal capability profile includes:

[0084] First, the comprehensive capability index output by the terminal capability profile model of S130 and the comprehensive capability index threshold of S140 are input.

[0085] By weighting and fusing the static comprehensive capability index with the dynamically predicted comprehensive capability index threshold, a dynamically updated terminal capability profile is generated. In one possible implementation, the following formula is used for calculation:

[0086] ;

[0087] in, It is the terminal capability portrait index, is the dynamic weight coefficient, which is calculated as follows: , is the adjustment coefficient, is the terminal performance volatility, is the comprehensive capability index, is the comprehensive capability index threshold.

[0088] Output:

[0089] The generated terminal capability portrait is presented in the form of a multi-dimensional capability index, including: terminal capability portrait index, decoding computing power characteristics, rendering capability characteristics, network jitter tolerance characteristics, and screen color gamut parameter characteristics.

[0090] In the multimodal low-latency partitioning method for ultra-high-definition video streams based on dynamic resource scheduling, S200 combines terminal capability profiles and real-time network status to build a dual-objective optimization model. With the goals of minimizing end-to-end transmission delay and maximizing multi-terminal video quality consistency, it dynamically generates the optimal bitrate allocation plan and the corresponding low-latency or high-compression routing path.

[0091] Please refer to Figure 3 , which shows a flowchart of an exemplary method S200 for multi-modal low-latency partitioning of ultra-high-definition video streams based on dynamic resource scheduling of the present application, including:

[0092] S210: Establish a dual-objective optimization model.

[0093] A dual-objective optimization model is established based on terminal capability profiles, real-time network bandwidth, and packet loss rate data. This dual-objective optimization model includes two core optimization goals: the first is to minimize end-to-end transmission delay, and the second is to maximize the consistency of multi-terminal video quality.

[0094] In one possible implementation, the construction of the dual-objective optimization model is based on the following two core objectives:

[0095] The first optimization goal is to minimize the end-to-end transmission delay. Specifically, the end-to-end transmission delay is composed of multiple factors, including the transmission delay of the video stream, the transcoding delay of the edge node, and the transmission delay of the network link. The objective function can be expressed as:

[0096] ;

[0097] in, represents the end-to-end transmission delay, Indicates terminal The video streaming delay, Indicates the edge node to the terminal The delay introduced by transcoding the video stream, Indicates the network link transmission delay, is the total number of terminals.

[0098] The constraint condition of this goal is the terminal's minimum comprehensive capability index threshold, namely: .

[0099] The second optimization goal is to maximize the consistency of multi-terminal video quality. Specifically, video quality consistency refers to the similarity in image quality of the video streams received by all terminals. This objective function can be expressed as:

[0100] ;

[0101] in, Indicates the consistency of video quality across multiple terminals. is the average bit rate of all terminals, Indicates terminal The difference between the bit rate and the average bit rate. The smaller the difference, the higher the video quality consistency.

[0102] The constraint of this goal is the real-time transcoding resource margin of the edge node, namely: ,in, For terminal The transcoding resources required for the video stream, The total amount of available transcoding resources on edge nodes.

[0103] Based on the first optimization objective and the second optimization objective, in a possible implementation, the objective function of the dual-objective optimization model can be expressed as:

[0104] ;

[0105] in, represents the end-to-end transmission delay, Indicates the consistency of video quality across multiple terminals. is the weight coefficient, which is used to adjust the relative importance between the two objectives.

[0106] S220: Generate an optimal bit rate allocation scheme, as well as corresponding low-latency paths and high-compression paths.

[0107] The optimal bitrate allocation scheme is based on the output of the dual-objective optimization model, combined with the terminal capability profile index, real-time network bandwidth and packet loss rate data, and a multi-objective optimization algorithm is used to solve the optimal bitrate allocation scheme for each terminal.

[0108] In one possible implementation, the step of generating the optimal bit rate allocation solution includes:

[0109] Based on the comprehensive capability index threshold output by the terminal capability profile model, the bit rate upper limit and resource occupancy range of each terminal are determined.

[0110] A multi-objective optimization algorithm is used to iteratively calculate the initial population and generate multiple candidate bitrate allocation schemes. During the iteration process, priority is given to meeting the minimum comprehensive capability index threshold constraint of the terminal, while balancing the end-to-end transmission delay and video quality consistency objectives.

[0111] The weight coefficients of the dual objective functions are dynamically adjusted based on real-time network fluctuations and the stability of the terminal capability profile. For example, when network latency fluctuates significantly, reducing transmission latency is prioritized; when terminal decoding computing power fluctuates significantly, ensuring video quality consistency is prioritized.

[0112] The final bitrate allocation result is output. The generated optimal bitrate allocation plan is presented in the form of a list, including the bitrate value of each terminal, the corresponding target frame rate, the edge node resource usage, and the path selection priority.

[0113] Based on the generated optimal bit rate allocation plan, a low-latency path or a high-compression path is dynamically selected for each terminal, and the path status is monitored in real time to trigger dynamic switching.

[0114] In one possible implementation, the low-latency path uses a 5G direct link with forward error correction technology, suitable for scenarios with high decoding load or low network jitter tolerance. The high-compression path uses edge node caching and bitrate conversion technology, suitable for scenarios with low packet loss rate and sufficient decoding capacity.

[0115] In the multimodal low-latency partitioning method for ultra-high-definition video streams based on dynamic resource scheduling, S300 divides the ultra-high-definition video stream into high-motion-sensitive areas and static background areas, dynamically allocates bit rates based on terminal capability profiles and real-time network status, and selects low-latency paths or high-compression paths, thereby achieving low-latency and efficient partitioning of multimodal terminals.

[0116] Please refer to Figure 4 , which shows a flowchart of an exemplary method S300 for multimodal low-latency partitioning of ultra-high-definition video streams based on dynamic resource scheduling of the present application, including:

[0117] S310: Divide the ultra-high-definition video stream into high-motion-sensitive area slices and static background slices.

[0118] When segmenting ultra-high-definition video streams, the dynamic nature of the video content significantly impacts transmission efficiency and the playback experience. Therefore, using video content analysis technology, the video stream is divided into highly motion-sensitive areas and static background areas, enabling differentiated processing for each region. The highly motion-sensitive areas contain a large number of rapidly changing visual elements, while the static background areas consist of relatively stable background information.

[0119] In a possible implementation, the division of high motion-sensitive area slices and static background slices depends on motion feature analysis of video content. By identifying motion patterns between video frames, the video content is divided into high motion-sensitive areas and static background areas.

[0120] S320: Bit rate allocation and low-latency path selection in high motion-sensitive areas.

[0121] For high-motion-sensitive areas, bitrate allocation is performed according to the optimal bitrate allocation scheme, and low-latency paths are selected for transmission. Since high-motion-sensitive areas typically contain a large amount of dynamically changing information, the video data volume is large, placing high demands on transmission bandwidth and decoding performance. Therefore, based on the terminal capability profile model and real-time network status, the bitrate allocation strategy is dynamically adjusted, and the optimal low-latency path is selected to ensure smooth video playback.

[0122] In one possible implementation, the bit rate allocation for high motion-sensitive areas is based on the decoding computing power, network jitter tolerance, and current network bandwidth in the terminal capability portrait model, ensuring that the terminal can stably decode and render the video stream under a low-latency path.

[0123] Specifically, the steps for determining the optimal bit rate allocation scheme for high motion-sensitive areas include:

[0124] Analyze the terminal's comprehensive capability index and predict whether the terminal can maintain a comprehensive capability index ≥ the minimum comprehensive capability index threshold within the next T seconds;

[0125] Determine the upper limit of the bit rate that can be carried based on the decoding computing power characteristics of the terminal.

[0126] The bitrate allocation is dynamically adjusted based on the terminal's render buffer saturation. If the terminal's render buffer is close to saturation, the system can reduce the bitrate to reduce rendering latency; if the buffer load is low, the bitrate can be appropriately increased to optimize image quality.

[0127] For terminals with low tolerance to network jitter, lower bitrate video streams are preferentially allocated to reduce the risk of unstable transmission caused by network fluctuations.

[0128] Slices in high-motion-sensitive areas are extremely sensitive to transmission delays, so a low-latency path is selected to ensure real-time video streaming. The selection of a low-latency path is based on the following criteria:

[0129] Prioritize paths with the lowest end-to-end transmission latency, such as 5G direct links or fiber optic transmission, to reduce data transmission time in the network.

[0130] Ensure that the path bandwidth is sufficient and has minimal fluctuations to avoid video freezes caused by insufficient bandwidth.

[0131] Forward error correction technology is introduced in low-latency paths to reduce retransmission delays caused by network packet loss by adding redundant data, thereby improving transmission stability.

[0132] S330: Bit rate allocation and high compression path selection for the static background area.

[0133] For the static background area, the bit rate is allocated according to the optimal bit rate allocation scheme, and a high compression path is selected for transmission.

[0134] Because static background areas typically contain less dynamically changing information, their video data volume is relatively small, placing lower demands on transmission bandwidth and decoding performance. Therefore, by combining a terminal capability profile model with real-time network status, the bitrate allocation strategy is dynamically adjusted, and the optimal high-compression path is selected to improve resource utilization and transmission efficiency.

[0135] In one possible implementation, the bit rate allocation for the static background area is based on the rendering capability, screen color gamut parameters, and current network bandwidth in the terminal capability portrait model.

[0136] Specifically, the steps of determining the optimal bit rate allocation scheme for the static background area include:

[0137] Analyze the terminal's comprehensive capability index and predict whether the terminal can maintain a comprehensive capability index ≥ the minimum comprehensive capability index threshold within the next T seconds;

[0138] Dynamically adjust the bitrate allocation based on the terminal's render buffer saturation. If the terminal's render buffer is close to saturation, reduce the bitrate to reduce rendering latency; if the buffer load is low, increase the bitrate appropriately to optimize image quality.

[0139] For terminals with a higher tolerance to network jitter, lower bitrate video streams are preferentially allocated to reduce the risk of unstable transmission caused by network fluctuations.

[0140] Static background slices have high bandwidth efficiency requirements, so a high compression path is selected to optimize transmission efficiency. The selection of a high compression path is based on the following criteria:

[0141] Prioritize transmission paths that support high-compression encoding to reduce the amount of data transmitted.

[0142] Ensure that the path supports edge node caching technology to reduce bandwidth consumption caused by repeated transmission.

[0143] Long-term reference frame technology is introduced in the high compression path to reduce the amount of data transmission by multiplexing static background frames, thereby improving compression efficiency.

[0144] In the multimodal low-latency partitioning method for ultra-high-definition video streams based on dynamic resource scheduling, the S400 quantifies the risk value of slice jamming through decoding time simulation and rendering buffer status verification based on terminal capability portraits, dynamically adjusts the weight coefficient of the dual-objective optimization model, and regenerates the optimal bitrate allocation plan and low-latency / high-compression path, realizing real-time optimization of resource scheduling and transmission strategies to ensure the smoothness and consistency of video stream quality.

[0145] Please refer to Figure 5 , which shows a flowchart of an exemplary method S400 for multi-modal low-latency partitioning of ultra-high-definition video streams based on dynamic resource scheduling of the present application, including:

[0146] S410: Simulating decoding time consumption based on terminal capability profile.

[0147] The terminal's decoding capabilities are a key factor affecting video playback smoothness and image quality stability. Therefore, based on the terminal's capability profile, we simulate the decoding time of the video stream to assess whether the terminal can meet the target frame rate requirements under current resource conditions.

[0148] In a possible implementation, the decoding time-consuming simulation calculation formula is:

[0149] ;

[0150] in, Indicates the decoding time. is the bit rate of the video stream, is the amount of single-frame video data, This formula determines the decoding time by quantifying the relationship between the terminal's decoding capability and the amount of video stream data.

[0151] S420: Verify the slice jamming risk based on the decoding time and rendering buffer status of the terminal capability portrait, and generate a slice jamming risk value.

[0152] Based on the decoding time simulation, S420 further evaluates the risk of video stream lag by combining the terminal's render buffer status. The terminal device's render buffer stores upcoming video frames, and its fill level directly affects the smoothness of video playback. The terminal's render buffer status is monitored in real time to determine the render buffer's saturation level.

[0153] In one possible implementation, the slice jam risk assessment formula is:

[0154] ;

[0155] in, Indicates the slice jam risk value, and are the weight coefficients of decoding time and rendering buffer saturation, Indicates the decoding time. Indicates the maximum allowed decoding time of the terminal. Indicates the saturation of the render buffer, is the maximum allowed saturation of the buffer.

[0156] If the decoding computing power of the terminal is low, resulting in the decoding time exceeding the maximum allowed value, then The value of will be large, indicating a high risk of lag. Similarly, if the render buffer is highly saturated, close to the maximum allowed value, then The value of will also be large, indicating a risk of overflow.

[0157] After completing the slice jam risk assessment, the system is instructed to adjust the resource scheduling strategy according to the slice jam risk value.

[0158] S430: Dynamically adjust the dual-objective weight coefficients based on the slice jam risk value to regenerate the optimal bitrate allocation plan, as well as the low-latency path and high-compression path.

[0159] The weight coefficients of the objective function in the dual-objective optimization model determine the balance between end-to-end transmission latency and consistent video quality across multiple terminals. When the slice's stall risk value indicates a high risk of stalling on a terminal, the dual-objective weight coefficients are dynamically adjusted to optimize resource scheduling strategies and ensure smooth video streaming and consistent image quality.

[0160] In one possible implementation, the dynamic adjustment of the dual-objective weight coefficients is calculated based on the following formula:

[0161] ;

[0162] in, represents the adjusted weight coefficient, represents the weight coefficient before adjustment, is the adjustment coefficient, is the slice jam risk value.

[0163] After determining the new weight coefficients, the objective function of the dual-objective optimization model is updated, and the optimal bit rate allocation scheme and the corresponding low-latency path and high-compression path are regenerated using step S220.

[0164] Based on dynamic resource scheduling, S100-S400 achieves low-latency division and transmission optimization of ultra-high-definition video streams through multi-modal terminal data collection and terminal capability profile construction. Its core innovation lies in: using the probe module to collect the terminal's decoding computing power, rendering capabilities, network jitter tolerance and screen color gamut parameters in real time, building a dynamic terminal capability profile model, and predicting the bit rate threshold that the terminal can bear. Combined with the dual-objective optimization model, it dynamically generates the optimal bit rate allocation plan and low-latency / high-compression transmission path; through video content analysis, it divides the video stream into high-motion-sensitive areas and static background areas, and introduces a feedback mechanism. Based on decoding time simulation and rendering buffer status verification to determine the risk of jamming, it dynamically adjusts the optimization model weights and resource scheduling strategies. The advantages of this application are: (1) accurate perception of terminal performance is achieved through multimodal data fusion and dynamic profiling, thereby improving the adaptability of resource allocation; (2) dual-objective optimization and content-aware partitioning strategies take into account both transmission efficiency and image quality consistency, reducing end-to-end latency; (3) the closed-loop feedback mechanism effectively responds to terminal performance fluctuations and network status changes, significantly reducing the probability of freezes; ultimately, high-quality, low-latency ultra-high-definition video transmission is achieved in a multi-terminal, multi-network environment.

[0165] The basic principles of the present application have been described above in conjunction with specific embodiments. However, it should be noted that the advantages, strengths, and effects mentioned in this application are merely illustrative and not restrictive, and it should not be assumed that these advantages, strengths, and effects are required of each embodiment of this application. In addition, the specific details disclosed above are merely illustrative and facilitating understanding, and are not restrictive. The above details do not limit this application to necessarily being implemented using the above specific details.

[0166] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present application. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of the present application. Therefore, the present application is not intended to be limited to the aspects shown herein, but rather to be applied in the widest sense consistent with the principles and novel features of the present invention.

[0167] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent replacements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. A multimodal low-latency partitioning method for ultra-high-definition video streams based on dynamic resource scheduling, characterized in that: include: Deploy probe modules at edge computing nodes to collect multimodal terminal data in real time; Based on the collected multimodal terminal data, a terminal capability profile model is constructed to determine the comprehensive capability index threshold for each terminal to maintain the target frame rate in the future time domain, generating a dynamically updated terminal capability profile. Using terminal capability profiles, combined with real-time network bandwidth and packet loss rate data, a dual-objective optimization model is established: the first objective is to minimize end-to-end transmission delay, constrained by the terminal's comprehensive capability index threshold; the second objective is to maximize multi-terminal video quality consistency, constrained by the edge node's real-time transcoding resource margin. The model outputs the optimal bitrate allocation for each terminal, as well as low-latency and high-compression paths. The ultra-high-definition video stream is divided into slices of high-motion-sensitive areas and static background slices. For the high-motion-sensitive slices, the bit rate is allocated according to the optimal bit rate allocation scheme and transmitted over a low-latency path. For the static background slices, the bit rate is allocated according to the optimal bit rate allocation scheme and transmitted over a high-compression path. The decoding computing power of the terminal capability portrait is used to simulate the decoding time, and the slice jamming risk is verified in combination with the rendering buffer status. The slice jamming risk value is generated, and the dual-objective weight coefficient is adjusted based on the slice jamming risk value to regenerate the optimal bitrate allocation plan, as well as the low-latency path and high-compression path.

2. The method for multimodal low-latency partitioning of ultra-high-definition video streams based on dynamic resource scheduling according to claim 1 is characterized in that: The probe module adopts a lightweight architecture and is embedded in the underlying system of the edge computing node. It obtains the decoding computing power of the terminal device by calling the interface, monitors the filling status of the rendering buffer through the video rendering framework to evaluate the rendering capability, obtains the network jitter tolerance by measuring the data transmission delay fluctuation between the terminal and the edge node, and obtains the screen color gamut parameters through the terminal display management interface.

3. The method for multimodal low-latency partitioning of ultra-high-definition video streams based on dynamic resource scheduling according to claim 1, characterized in that: The multimodal terminal feature extraction includes: calculating the decoder load rate, rendering buffer saturation, network jitter tolerance and color gamut coverage of the terminal; Among them, the decoder load rate is the ratio of the actual running time of the decoder to the total running time, the rendering buffer saturation is the ratio of the current buffer size to the maximum capacity, the network jitter tolerance is the ratio of the network delay fluctuation to the average delay, and the color gamut coverage is the ratio of the terminal color gamut area to the standard color space color gamut area.

4. The method for multimodal low-latency partitioning of ultra-high-definition video streams based on dynamic resource scheduling according to claim 1, characterized in that: The construction formula of the terminal capability portrait model is: ； in, is the comprehensive capability index, is the characteristic of decoding computing power, Characteristic of rendering capability, is the characteristic of network jitter tolerance, is the characteristic of the screen color gamut parameter, is the dynamic adjustment coefficient, is the performance change rate within the time window; By weighting and fusing the static comprehensive capability index with the dynamically predicted comprehensive capability index threshold, a dynamically updated terminal capability profile is generated. The calculation formula is: ； in, It is the terminal capability portrait index, is the dynamic weight coefficient, which is calculated as follows: , is the adjustment coefficient, is the terminal performance volatility, is the comprehensive capability index, is the comprehensive capability index threshold; Among them, the generated terminal capability portrait is presented in the form of a multi-dimensional capability index, including: terminal capability portrait index, characteristics of decoding computing power, characteristics of rendering capability, characteristics of network jitter tolerance, and characteristics of screen color gamut parameters.

5. The method for multimodal low-latency partitioning of ultra-high-definition video streams based on dynamic resource scheduling according to claim 1, characterized in that: The first objective function of the dual-objective optimization model is: ； in, represents the end-to-end transmission delay, Indicates terminal The video streaming delay, Indicates the edge node to the terminal The delay introduced by transcoding the video stream, Indicates the network link transmission delay, is the total number of terminals; The first objective function constraint is: comprehensive capability index ≥ terminal comprehensive capability index threshold; The second objective function is: ； in, Indicates the consistency of video quality across multiple terminals. is the average bit rate of all terminals, Indicates terminal The difference between the bit rate and the average bit rate; The second objective function constraint is ,in, For terminal The transcoding resources required for the video stream, The total amount of available transcoding resources on edge nodes.

6. The method for multimodal low-latency partitioning of ultra-high-definition video streams based on dynamic resource scheduling according to claim 1, characterized in that: The step of generating the optimal bit rate allocation scheme includes: Determine the bitrate cap based on the comprehensive capability index threshold of the terminal capability profile model; A multi-objective optimization algorithm is used to iteratively generate candidate solutions, giving priority to satisfying the terminal comprehensive capability index threshold constraint; Dynamically adjust the dual-objective weight coefficients, giving priority to reducing transmission delay when network delay fluctuates greatly, and giving priority to ensuring video quality consistency when terminal decoding computing power fluctuates greatly.

7. The method for multimodal low-latency partitioning of ultra-high-definition video streams based on dynamic resource scheduling according to claim 1, characterized in that: The low-latency path selection of the high motion-sensitive area slice includes: Prioritize 5G direct links or fiber-optic transmission paths with the lowest end-to-end transmission latency; Ensure that the path bandwidth is sufficient and has low fluctuations; Forward error correction technology is introduced to reduce retransmission delays caused by network packet loss.

8. The method for multimodal low-latency partitioning of ultra-high-definition video streams based on dynamic resource scheduling according to claim 1, characterized in that: The high compression path selection for the static background slice includes: Prioritize transmission paths that support high-compression encoding; Use edge node caching technology to reduce repeated transmission bandwidth consumption; Long-term reference frame technology is introduced to reuse static background frames to improve compression efficiency.

9. The method for multimodal low-latency partitioning of ultra-high-definition video streams based on dynamic resource scheduling according to claim 1, characterized in that: The calculation formula of the slice jam risk value is: ； in, Indicates the slice jam risk value, and are the weight coefficients of decoding time and rendering buffer saturation, Indicates the decoding time. Indicates the maximum allowed decoding time of the terminal. Indicates the saturation of the render buffer, is the maximum allowed saturation of the buffer.

10. The method for multimodal low-latency partitioning of ultra-high-definition video streams based on dynamic resource scheduling according to claim 9, characterized in that: The dynamic adjustment formula of the dual-objective weight coefficient is: ； in, represents the adjusted weight coefficient, represents the weight coefficient before adjustment, is the adjustment coefficient, is the slice jam risk value.

Images