Audio and video communication performance optimization method and system based on multipath transmission

Abstract

The invention provides an audio and video communication performance optimization method and system based on multi-path transmission, and relates to the technical field of audio and video communication.The method comprises the steps that multiple network paths are actively monitored, path classification is conducted based on monitoring data, and M transmission path combinations are obtained; performing frame packaging processing on the audio and video streams to obtain H audio and video data packets, and then performing transmission path matching optimization to obtain M groups of data packet transmission sequences; and transmitting the M groups of data packet transmission sequences to a second communication end through M transmission path combinations, performing real-time acquisition of audio and video communication data through a real-time transmission window, and performing transmission path dynamic matching and data transmission by taking a path classification updating result of a plurality of network paths as a constraint. The technical problems of path distribution staticizing and insufficient load balancing in a multipath transmission audio and video communication scene in the prior art are solved. The technical effect that smooth user audio and video communication experience is provided under the complex network condition is achieved.

Description

Technical Field

[0001] The present invention relates to the technical field of audio and video communication, and in particular to a method and system for optimizing audio and video communication performance based on multi-path transmission. Background Art

[0002] In audio and video communication scenarios with multi-path transmission, existing technologies usually use fixed parameters or pre-set static path allocation strategies, while ignoring the real-time fluctuations in network performance. For example, when users conduct high-definition video conferences or live broadcasts, due to high bandwidth requirements and complex network environments, existing technologies cannot dynamically adjust path allocation based on real-time monitored path performance. This static path allocation causes some paths to be overloaded, while other paths are not fully utilized, resulting in a waste of network resources and low transmission efficiency.

[0003] At the same time, existing technologies also have shortcomings in multi-path load balancing. For example, when multiple data packets need to be transmitted through different paths, the failure to comprehensively consider the real-time performance of the paths (such as bandwidth, latency, and packet loss rate) can easily cause transmission delays or loss of key data packets. In this case, the audio and video streams are out of sync or freeze, which seriously affects the user experience.

[0004] In summary, due to the defects of static path allocation and insufficient load balancing, the existing technology is difficult to adapt to the dynamic network environment, resulting in a decrease in communication transmission quality, which in turn affects the user's audio and video communication experience. Summary of the invention

[0005] The present invention provides a method and system for optimizing audio and video communication performance based on multi-path transmission, which is used to solve the technical problem that in the audio and video communication scenario of multi-path transmission in the prior art, there are defects of static path allocation and insufficient load balancing, which leads to a decrease in communication transmission quality and further affects the user's audio and video communication experience.

[0006] In view of the above problems, the present invention provides a method and system for optimizing audio and video communication performance based on multi-path transmission.

[0007] According to a first aspect of the present invention, a method for optimizing audio and video communication performance based on multi-path transmission is provided, the method comprising: adaptively adjusting a sliding transmission window according to network conditions to obtain a real-time transmission window; using the real-time transmission window at a first communication end to perform real-time collection of audio and video communication data to obtain a first audio and video stream; actively monitoring multiple network paths and classifying paths based on the monitoring data to obtain M transmission path combinations, wherein the path classification is dynamically updated based on real-time performance monitoring results; performing frame packaging processing on the first audio and video stream to obtain H audio and video data packets; performing transmission path matching optimization on the H audio and video data packets to obtain M groups of data packet transmission sequences; transmitting the M groups of data packet transmission sequences to a second communication end through the M transmission path combinations; and so on, performing real-time collection of audio and video communication data through the real-time transmission window, and dynamically matching transmission paths and transmitting data based on the path classification update results of the multiple network paths as constraints.

[0008] According to a second aspect of the present invention, a system for optimizing audio and video communication performance based on multi-path transmission is provided, wherein the system comprises: a sliding window adjustment unit, configured to adaptively adjust the sliding transmission window according to network conditions to obtain a real-time transmission window; a communication data acquisition unit, configured to use the real-time transmission window to acquire audio and video communication data in real time at a first communication end to acquire a first audio and video stream; a transmission path classification unit, configured to actively monitor a plurality of network paths and classify the paths based on the monitoring data to acquire M transmission path combinations, wherein the path classification is dynamically updated based on the real-time performance monitoring result; a data packaging execution unit, configured to perform frame packaging processing on the first audio and video stream to acquire H audio and video data packets; a transmission path optimization unit, configured to perform transmission path matching optimization on the H audio and video data packets to acquire M groups of data packet transmission sequences; a data transmission execution unit, configured to transmit the M groups of data packet transmission sequences to a second communication end through the M transmission path combinations; and a transmission cycle execution unit, configured to perform real-time acquisition of audio and video communication data through the real-time transmission window, and dynamically match the transmission paths and transmit data based on the path classification update results of the plurality of network paths as constraints.

[0009] One or more technical solutions provided in the present invention have at least the following technical effects or advantages:

[0010] The method provided by the embodiment of the present invention obtains a real-time transmission window by adaptively adjusting the sliding transmission window according to network conditions; uses the real-time transmission window to collect audio and video communication data in real time at the first communication end to obtain a first audio and video stream; actively monitors multiple network paths and classifies the paths based on the monitoring data to obtain M transmission path combinations, wherein the path classification is dynamically updated based on the real-time performance monitoring results; performs frame packaging processing on the first audio and video stream to obtain H audio and video data packets; performs transmission path matching optimization on the H audio and video data packets to obtain M groups of data packet transmission sequences; transmits the M groups of data packet transmission sequences to the second communication end through the M transmission path combinations; and so on, collects audio and video communication data in real time through the real-time transmission window, and performs dynamic transmission path matching and data transmission based on the path classification update results of the multiple network paths as constraints. The transmission quality, stability and real-time performance of audio and video communications in multi-path transmission scenarios are significantly improved, and the technical effect of providing a smooth user audio and video communication experience under complex network conditions is ensured. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 A schematic flow chart of a method for optimizing audio and video communication performance based on multipath transmission provided by the present invention;

[0012] Figure 2 A schematic diagram of the structure of the audio and video communication performance optimization system based on multi-path transmission provided by the present invention.

[0013] Explanation of the reference numerals: sliding window adjustment unit 11 , communication data acquisition unit 12 , transmission path classification unit 13 , data packaging execution unit 14 , transmission path optimization unit 15 , data transmission execution unit 16 , transmission cycle execution unit 17 . DETAILED DESCRIPTION

[0014] The present invention provides a method and system for optimizing the performance of audio and video communication based on multipath transmission, which is used to solve the defects of static path allocation and insufficient load balancing in the audio and video communication scenarios of multipath transmission in the prior art, which leads to a decrease in communication transmission quality and thus affects the user's audio and video communication experience. The method achieves a technical effect of significantly improving the transmission quality, stability and real-time performance of audio and video communication in multipath transmission scenarios, and ensuring a smooth user audio and video communication experience under complex network conditions.

[0015] Below, the technical solutions in the present invention will be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments of the present invention. It should be understood that the present invention is not limited to the example embodiments described herein. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention. It should also be noted that, for the convenience of description, only the parts related to the present invention are shown in the accompanying drawings, rather than all of them.

[0016] Embodiment 1, as Figure 1 As shown, the present invention provides a method for optimizing audio and video communication performance based on multipath transmission, the method comprising:

[0017] A100: Adaptively adjust the sliding transmission window according to network conditions to obtain a real-time transmission window.

[0018] In this embodiment, the sliding transmission window is a mechanism for dynamically adjusting the range of data reception and playback, and its length is adaptively adjusted according to network conditions: when bandwidth decreases, latency increases, or packet loss rate increases, the window is lengthened to buffer more data; when network conditions are good, the window is shortened to reduce latency. The window uses a sliding update mechanism to remove old data and receive new data, ensuring the real-time and playback synchronization of audio and video data, while providing time constraints for network path performance monitoring and optimization. Through the adaptive adjustment of the sliding transmission window, a real-time transmission window is obtained, and playback while receiving is achieved, thereby ensuring low latency and smoothness of audio and video streams.

[0019] A200: Using the real-time transmission window at the first communication end to collect audio and video communication data in real time to obtain a first audio and video stream.

[0020] Specifically, in this embodiment, the real-time transmission window is used at the first communication end to dynamically collect audio and video communication data, continuously receive real-time data through the boundary update mechanism of the sliding transmission window, and generate a first audio and video stream to provide high-quality basic data for subsequent path transmission and optimization.

[0021] In this embodiment, the first communication end and the second communication end have two-way communication capabilities, and can serve as data receiving ends and data sending ends for each other according to actual transmission requirements. That is, when the first communication end serves as a data sending end, the second communication end serves as a data receiving end; when the second communication end serves as a data sending end, the first communication end serves as a data receiving end.

[0022] This two-way interactive design ensures two-way real-time transmission of audio and video communication data, meeting application requirements in scenarios such as video conferencing or two-way streaming media transmission.

[0023] A300: Actively monitor multiple network paths and classify the paths based on the monitoring data to obtain M transmission path combinations, where the path classification is dynamically updated based on real-time performance monitoring results.

[0024] In one embodiment, multiple network paths are actively monitored and path classification is performed based on the monitoring data to obtain M transmission path combinations. Step A300 of the method provided by the present invention further includes:

[0025] A310: Set a path performance update window according to the real-time transmission window, wherein the length of the path performance update window is 1 / K of the real-time transmission window.

[0026] A320: Preconfigured test data packages.

[0027] A330: In the process of intermittently transmitting the test data packet from the first communication end to the second communication end with the path performance update window as a time constraint, multiple transmission path performance data are collected for the multiple network paths to obtain multiple groups of path performance data, wherein each group of path performance indicators includes K path performance information, and the path performance information includes bandwidth, packet loss rate, packet disorder rate, packet repetition rate, packet header overhead and delay change rate.

[0028] A340: Pre-construct a path performance evaluation model, and obtain multiple path performance quantization coefficients by loading the multiple groups of path performance data into the path performance evaluation model.

[0029] A350: Aggregate the multiple network paths according to the multiple path performance quantization coefficients to obtain the M transmission path combinations.

[0030] In one embodiment, a path performance evaluation model is pre-built, and multiple path performance quantization coefficients are obtained by loading the multiple groups of path performance data into the path performance evaluation model. Step A340 of the method provided by the present invention further includes:

[0031] A341: Pre-construct a path performance evaluation function, the path performance evaluation function is as follows:

[0032] Among them, S is the path performance quantization coefficient, B is the bandwidth, P is the packet loss rate, O is the packet disorder rate, D is the packet repetition rate, H is the packet header overhead, and V is the delay variation rate.

[0033] A342: Use the path performance evaluation function to construct the path performance evaluation model.

[0034] A343: Load the multiple groups of path performance data into the path performance evaluation model to obtain multiple groups of node performance quantization coefficients, wherein each group of node performance quantization coefficients includes K node performance quantization coefficients.

[0035] A344: Perform time-series weighting on the multiple groups of node performance quantization coefficients to obtain the multiple path performance quantization coefficients.

[0036] Specifically, in this embodiment, multiple network paths are actively monitored and path classification is performed based on the monitoring data. The purpose is to divide the multiple network paths into M transmission path combinations by monitoring and evaluating the network path performance, so as to facilitate subsequent load distribution and data packet transmission optimization.

[0037] Specifically, this implementation uses a shorter time slice set based on the real-time transmission window to intermittently monitor network path performance, and its length is 1 / K of the real-time transmission window, ensuring that changes in network status are captured in a shorter time.

[0038] Preconfigured test data packets are used to measure the performance of network paths within the path performance update window. The content and size of the test data packets can be configured according to actual needs to simulate the actual transmitted data load, thereby accurately reflecting the path performance. At the same time, the test data packet header may contain information such as timestamp and path identifier for path performance analysis at the receiving end.

[0039] During the duration of the entire real-time transmission window, the path performance update window is reused K times, that is, the test is performed K times. Through K times of intermittent testing, it is ensured that performance monitoring covers the entire real-time transmission window.

[0040] When testing within the path performance update window, each test is performed by sending a preconfigured test data packet to the second communication end for the first communication end. The second communication end collects and records multiple path performance data of multiple network paths by parsing the received test data packet, and accumulates K test results to form multiple groups of complete path performance data, wherein each group of path performance indicators includes K path performance information, and the path performance information includes bandwidth, packet loss rate, packet disorder rate, packet repetition rate, packet header overhead and delay change rate.

[0041] The bandwidth indicates the available transmission capacity of the path, in Mbps; the packet loss rate is the proportion of lost packets detected by the receiving end; the packet out-of-order rate is the proportion of inconsistent packet receiving order with the sending order; the packet repetition rate is the proportion of packets repeatedly received by the receiving end; the packet header overhead is the amount of protocol header data generated during the path transmission process; and the delay variation rate is the range of path delay fluctuation.

[0042] This embodiment uses a path performance evaluation model to evaluate the multiple groups of path performance data to indirectly provide a reference for the classification of the multiple transmission paths.

[0043] The method for constructing the path performance evaluation model is to pre-construct a path performance evaluation function, and implement the path performance evaluation function as a calculation model to obtain the path performance evaluation model.

[0044] The multiple groups of path performance data are loaded into the path performance evaluation model, and the node performance quantization coefficients are calculated path by path to obtain multiple groups of node performance quantization coefficients, each group of node performance quantization coefficients includes K node performance quantization coefficients. The multiple groups of node performance quantization coefficients are time-series weighted to comprehensively consider the influence of historical monitoring data to obtain the multiple path performance quantization coefficients. This embodiment smoothes short-term abnormal fluctuations through weighted processing to improve the stability of performance quantization. It should be understood that based on the formula structure of the path performance evaluation function, the larger the path performance quantization coefficient S, the better the performance of the path, and accordingly, more load distribution in participating data transmission.

[0045] The multiple network paths are aggregated according to the multiple path performance quantization coefficients to assign paths with smaller differences in performance quantization coefficients to the same combination, and finally the M transmission path combinations are obtained, wherein M is a positive integer less than the number of paths of the multiple transmission paths.

[0046] This embodiment generates M transmission path combinations through path aggregation, thereby providing a basis for load distribution of audio and video data, thereby indirectly achieving the technical effect of improving transmission efficiency and stability.

[0047] A400: Perform frame packaging processing on the first audio and video stream to obtain H audio and video data packets.

[0048] Specifically, in the process of audio and video communication, the first audio and video stream is first divided into multiple audio and video frames by performing frame packaging processing on the first audio and video stream, wherein the audio frame is extracted by time slice and the video frame is extracted by frame rate; then, it is further divided into several data blocks according to the size of each frame and the load capacity of network transmission; finally, each data block is encapsulated into an audio and video data packet, and necessary identification information is added to each data packet during the encapsulation process, including a sequence number, a timestamp, and a load type, etc., to ensure the order and real-time nature of the data packet. Through the above processing, H audio and video data packets are finally generated, which can be used for subsequent multi-path transmission optimization processing to further improve the transmission efficiency and the playback effect of the audio and video stream.

[0049] A500: Optimize the transmission path matching of the H audio and video data packets to obtain M groups of data packet transmission sequences.

[0050] In one embodiment, the transmission path matching optimization is performed on the H audio and video data packets to obtain M groups of data packet transmission sequences. Step A500 of the method provided by the present invention further includes:

[0051] A510: Extract H data packet attribute information of the H audio and video data packets.

[0052] A520: Taking the transmission path ratio of the M transmission path combinations as a data packet grouping constraint, priority classification is performed on the H audio and video data packets according to the attribute information of the H data packets to obtain a set of M audio and video data packets.

[0053] A530: Divide the multiple path performance quantization coefficients into M groups of path performance quantization coefficients according to the M transmission path combinations.

[0054] A540: Allocate transmission loads for the M transmission path combinations according to the M groups of path performance quantization coefficients, and serialize the M audio and video data packet sets according to the allocation results to obtain the M groups of data packet transmission sequences.

[0055] In one embodiment, the transmission path ratio of the M transmission path combinations is used as a data packet grouping constraint, the H audio and video data packets are prioritized according to the H data packet attribute information, and M audio and video data packet sets are obtained. Step A520 of the method provided by the present invention further includes:

[0056] A521: Add the bandwidths of the M transmission path combinations to obtain M total bandwidths.

[0057] A522: Perform ratio calculation based on the M total bandwidths to obtain the transmission path ratio.

[0058] A523: Prioritize the H audio and video data packets according to the attribute information of the H data packets to obtain H priority coefficients.

[0059] A524: Serialize the H audio and video data packets according to the H priority coefficients to obtain an initial audio and video data packet sequence.

[0060] A525: Taking the transmission path ratio as a constraint, split the initial audio and video data packet sequence to obtain M audio and video data packet subsequences, wherein the M audio and video data packet subsequences respectively correspond to the composition of the M audio and video data packet sets.

[0061] In one embodiment, the transmission load of the M transmission path combinations is allocated according to the M groups of path performance quantization coefficients, and the M audio and video data packet sets are serialized according to the allocation results to obtain the M groups of data packet transmission sequences. Step A540 of the method provided by the present invention also includes:

[0062] A541: Normalize the M groups of path performance quantization coefficients to obtain M transmission load ratios.

[0063] A542: Using the M transmission load ratios as constraints, split the M audio and video data packet subsequences to obtain M groups of initial transmission sequences.

[0064] A543: Perform nested sorting and updating on the M groups of initial transmission sequences according to the attribute information of the H data packets to obtain the M groups of data packet transmission sequences.

[0065] Specifically, in this embodiment, firstly, H data packet attribute information of the H audio and video data packets are extracted to provide a basis for subsequent priority classification, wherein the data packet attribute information includes the timestamp, size and priority of the data packet, the timestamp reflects the position of the data packet in the audio and video stream to ensure the time order of playback during transmission, the priority is used to identify the importance of the data packet, for example, audio packets and key frame data packets usually have higher priority, and the data packet size is used to optimize path traffic during subsequent load distribution.

[0066] Bandwidths of the M transmission path combinations are summed to obtain M total bandwidths, and ratio calculation is performed based on the M total bandwidths to obtain the transmission path ratio.

[0067] According to the attribute information of the H audio and video data packets, the priority of each data packet is quantified, and the priority coefficient is calculated for each data packet. The calculation of the priority coefficient comprehensively considers the importance and size of the data packet and quantizes it into a numerical form. The quantized priority coefficient is used to sort the H audio and video data packets, and the H audio and video data packets are serialized according to the H priority coefficients to obtain an initial audio and video data packet sequence. In this embodiment, the priority quantization method includes but is not limited to normalization calculation and weighted calculation.

[0068] Taking the transmission path ratio as a constraint, the initial audio and video data packet sequence is divided into M audio and video data packet subsequences. The size and composition of each data packet subsequence match the transmission path ratio. For example, if the transmission ratio of a certain path combination is 30%, the audio and video data packet subsequence corresponding to the path combination will contain about 30% of the data packets. The segmentation result ensures that the load of each path combination matches its transmission capacity, while maintaining the priority order of the data packets.

[0069] The multiple path performance quantization coefficients are divided into M groups of path performance quantization coefficients according to the M transmission path combinations, and the M groups of path performance quantization coefficients are normalized to obtain M transmission load ratios, wherein each transmission load ratio corresponds to a comprehensive load distribution of multiple transmission paths in a transmission path combination.

[0070] The same method of dividing the initial audio and video data packet sequence into M audio and video data packet subsequences is adopted with the transmission path ratio as a constraint, and the M transmission load ratios are used as a constraint to correspondingly divide the M audio and video data packet subsequences to obtain M groups of initial transmission sequences.

[0071] It should be understood that each initial transmission sequence is essentially the transmission allocation of audio and video data packets on a transmission path.

[0072] The specific method of removing the sorting constraint processing on the first initial transmission sequence, obtaining F audio and video data packets, and performing nested sorting update on the F audio and video data packets is as follows:

[0073] First, extract the timestamp, size and priority of F audio and video data packets from H audio and video data packets. Next, ensure that the transmission order of the audio and video data packets is consistent with the playback time logic of the audio and video stream, and prioritize the F audio and video data packets according to the timestamp from small to large. During the sorting process, if the timestamps are the same, sort them from high to low according to the priority to ensure that key data (such as audio data packets or video key frames) are transmitted first. If the timestamp and priority are the same, sort them from large to small according to the size of the data packet to improve the transmission efficiency of the data transmission path, and finally determine the first transmission sequence of the audio and video data packets.

[0074] By analogy, the M groups of initial transmission sequences are nested and sorted and updated according to the attribute information of the H data packets to obtain the M groups of data packet transmission sequences.

[0075] This embodiment generates M groups of data packet transmission sequences, and the order of data packets in each group of sequences is optimized to adapt to the dynamic performance of the corresponding path combination, thereby ensuring the transmission quality of audio and video communications.

[0076] A600: Transmit the M groups of data packet transmission sequences to the second communication end through the M transmission path combinations.

[0077] Specifically, in this embodiment, each transmission path combination is composed of multiple transmission paths, and the load distribution of the transmission paths is based on the normalized result of their performance quantization coefficients. For example, if the transmission load ratio of a path combination is 30%, the data packet transmission sequence in the path combination will pass through each transmission path in the path combination to complete the load distribution and transmission of the data packets in a manner consistent with the ratio.

[0078] During the transmission process, the transmission order of the data packets is strictly maintained according to their priority and timestamp to ensure the order of the data packets in the transmission path combination. At the same time, the load is distributed according to the dynamic adjustment mechanism of the path performance within the path combination to ensure the maximum transmission efficiency and transmission stability. The transmission result ensures the integrity and real-time performance of the data packets, and provides reliable basic data for the decoding and playback of the audio and video streams at the second communication end.

[0079] This embodiment achieves the technical effect of significantly improving the transmission quality, stability and real-time performance of audio and video communications in multi-path transmission scenarios through real-time audio and video stream acquisition, path performance monitoring and dynamic optimization, ensuring a smooth user audio and video communication experience under complex network conditions.

[0080] A700: Similarly, real-time collection of audio and video communication data is performed through the real-time transmission window, and dynamic matching of transmission paths and data transmission are performed based on the path classification update results of the multiple network paths as constraints.

[0081] In one embodiment, the method steps provided by the present invention further include:

[0082] A710: In the process of transmitting the M groups of data packet transmission sequences to the second communication end through the M transmission path combinations, the transmission status of the M transmission path combinations is passively monitored, and the path classification is updated according to the monitoring results to obtain M updated path combinations.

[0083] A720: Using the real-time transmission window at the first communication end to collect audio and video communication data in real time to obtain a second audio and video stream.

[0084] A730: After performing frame packing processing on the second audio and video stream, transmission path matching optimization and data transmission are performed based on the M update path combinations as constraints.

[0085] A740: Similarly, real-time collection of audio and video communication data is performed through the real-time transmission window, and dynamic matching of transmission paths and data transmission are performed based on the update results of the M transmission path combinations as constraints.

[0086] Specifically, based on step A300, it can be seen that the path classification in this embodiment is dynamically updated based on the real-time performance monitoring results to ensure that each path allocation matches the current network status. In the multi-path transmission of the first audio and video stream, the path allocation is completed based on the performance test results of the test data packet; and the subsequent multi-path transmission of the second audio and video stream to the Nth audio and video stream, its path allocation refers to the transmission status data during the previous audio and video stream transmission process.

[0087] Specifically, when data is transmitted through a combination of M transmission paths, this embodiment adopts the same method of collecting multiple transmission path performance data for the multiple network paths to obtain multiple groups of path performance data, performs real-time passive monitoring of the transmission status of each transmission path, and obtains test data consistent with the data structure of the path performance data.

[0088] Then, the same method as that for generating the M transmission path combinations is adopted to update the combination of the plurality of transmission paths, so as to obtain the M updated path combinations.

[0089] At the first communication end, the real-time transmission window is used to perform real-time collection of audio and video communication data to obtain a second audio and video stream. After frame packaging processing is performed on the second audio and video stream, the same method as obtaining the M groups of data packet transmission sequences is used to analyze and obtain M groups of update transmission sequences.

[0090] Thereby, transmission path matching optimization and data transmission of M groups of update transmission sequences are realized with the M update path combinations as constraints.

[0091] By analogy, the real-time collection of audio and video communication data is performed through the real-time transmission window, and the dynamic matching of transmission paths and data transmission are performed based on the update results of the M transmission path combinations as constraints.

[0092] This embodiment realizes real-time monitoring and dynamic optimization of data transmission in multi-path transmission during audio and video communication, ensuring that the path allocation and transmission of each audio and video stream can dynamically adapt to the current network status, thereby significantly improving the transmission quality and real-time technical effects of audio and video communication.

[0093] Embodiment 2 is based on the same inventive concept as the method for optimizing audio and video communication performance based on multipath transmission in the aforementioned embodiment. Figure 2 As shown, the present invention provides an audio and video communication performance optimization system based on multi-path transmission, wherein the system comprises:

[0094] The sliding window adjustment unit 11 is used to perform adaptive adjustment of the sliding transmission window according to network conditions to obtain a real-time transmission window.

[0095] The communication data collection unit 12 is used to collect audio and video communication data in real time using the real-time transmission window at the first communication end to obtain a first audio and video stream.

[0096] The transmission path classification unit 13 is used to actively monitor multiple network paths and classify the paths based on the monitoring data to obtain M transmission path combinations, wherein the path classification is dynamically updated based on the real-time performance monitoring results.

[0097] The data packing execution unit 14 is used to perform frame packing processing on the first audio and video stream to obtain H audio and video data packets.

[0098] The transmission path optimization unit 15 is used to perform transmission path matching optimization on the H audio and video data packets to obtain M groups of data packet transmission sequences.

[0099] The data transmission execution unit 16 is configured to transmit the M groups of data packet transmission sequences to the second communication end through the M transmission path combinations.

[0100] The transmission cycle execution unit 17 is used to perform real-time collection of audio and video communication data through the real-time transmission window and perform dynamic transmission path matching and data transmission based on the path classification update results of the multiple network paths as constraints.

[0101] In one embodiment, the transmission cycle execution unit 17 is further configured to:

[0102] In the process of transmitting the M groups of data packet transmission sequences to the second communication end through the M transmission path combinations, the transmission status of the M transmission path combinations is passively monitored, and the path classification is updated according to the monitoring results to obtain M updated path combinations; the real-time transmission window is used at the first communication end to collect audio and video communication data in real time to obtain a second audio and video stream; after frame packaging processing of the second audio and video stream, the transmission path matching optimization and data transmission are performed with the M updated path combinations as constraints; and so on, the real-time collection of audio and video communication data is performed through the real-time transmission window, and the update results of the M transmission path combinations are used as constraints to dynamically match the transmission path and transmit data.

[0103] In one embodiment, the transmission path classification unit 13 is further used for:

[0104] A path performance update window is set according to the real-time transmission window, wherein the length of the path performance update window is 1 / K of the real-time transmission window; a test data packet is preconfigured; in the process of intermittently transmitting the test data packet from the first communication end to the second communication end with the path performance update window as a time constraint, multiple transmission path performance data are collected for the multiple network paths to obtain multiple groups of path performance data, wherein each group of path performance indicators includes K path performance information, and the path performance information includes bandwidth, packet loss rate, packet disorder rate, packet repetition rate, packet header overhead and delay change rate; a path performance evaluation model is prebuilt, and multiple path performance quantization coefficients are obtained by loading the multiple groups of path performance data into the path performance evaluation model; the multiple network paths are aggregated according to the multiple path performance quantization coefficients to obtain the M transmission path combinations.

[0105] In one embodiment, the transmission path optimization unit 15 is further configured to:

[0106] Extract H data packet attribute information of the H audio and video data packets; take the transmission path ratio of the M transmission path combinations as the data packet grouping constraint, and classify the H audio and video data packets according to the H data packet attribute information to obtain M audio and video data packet sets; divide the multiple path performance quantization coefficients into M groups of path performance quantization coefficients according to the M transmission path combinations; distribute the transmission load for the M transmission path combinations according to the M groups of path performance quantization coefficients, and serialize the M audio and video data packet sets according to the distribution results to obtain the M groups of data packet transmission sequences.

[0107] In one embodiment, the transmission path optimization unit 15 is further configured to:

[0108] The bandwidths of the M transmission path combinations are added to obtain M total bandwidths; a ratio is calculated based on the M total bandwidths to obtain the transmission path ratio; the priorities of the H audio and video data packets are quantized according to the attribute information of the H data packets to obtain H priority coefficients; the H audio and video data packets are serialized according to the H priority coefficients to obtain an initial audio and video data packet sequence; the initial audio and video data packet sequence is divided based on the transmission path ratio as a constraint to obtain M audio and video data packet subsequences, wherein the M audio and video data packet subsequences respectively correspond to the composition of the M audio and video data packet sets.

[0109] In one embodiment, the transmission path optimization unit 15 is further configured to:

[0110] The M groups of path performance quantization coefficients are normalized to obtain M transmission load ratios; the M audio and video data packet subsequences are divided based on the M transmission load ratios as constraints to obtain M groups of initial transmission sequences; the M groups of initial transmission sequences are nested and sorted and updated according to the H data packet attribute information to obtain the M groups of data packet transmission sequences.

[0111] In one embodiment, the transmission path classification unit 13 is further used for:

[0112] A path performance evaluation function is pre-constructed, and the path performance evaluation function is as follows:

[0113]

[0114] Among them, S is the path performance quantization coefficient, B is the bandwidth, P is the packet loss rate, O is the packet disorder rate, D is the packet repetition rate, H is the packet header overhead, and V is the delay change rate; the path performance evaluation function is used to construct the path performance evaluation model; the multiple groups of path performance data are loaded into the path performance evaluation model to obtain multiple groups of node performance quantization coefficients, wherein each group of node performance quantization coefficients includes K node performance quantization coefficients; the multiple groups of node performance quantization coefficients are time-series weighted to obtain the multiple path performance quantization coefficients.

[0115] Any of the methods or steps described above may be stored as computer instructions or programs in various types of computer memories, and the computer instructions or programs may be recognized by various types of computer processors to implement any of the methods or steps described above.

[0116] Based on the above specific embodiments of the present invention, any improvements and modifications made to the present invention by those skilled in the art without departing from the principles of the present invention shall fall within the patent protection scope of the present invention.

Claims

1. A method for optimizing audio and video communication performance based on multipath transmission, characterized in that: The method comprises: Adaptively adjust the sliding transmission window according to network conditions to obtain a real-time transmission window; Using the real-time transmission window at the first communication end to collect audio and video communication data in real time to obtain a first audio and video stream; Actively monitor multiple network paths and classify the paths based on the monitoring data to obtain M transmission path combinations, where the path classification is dynamically updated based on real-time performance monitoring results; Performing frame packaging processing on the first audio and video stream to obtain H audio and video data packets; Performing transmission path matching optimization on the H audio and video data packets to obtain M groups of data packet transmission sequences; Transmitting the M groups of data packet transmission sequences to the second communication end through the M transmission path combinations; By analogy, the real-time collection of audio and video communication data is performed through the real-time transmission window, and the transmission path dynamic matching and data transmission are performed based on the path classification update results of the multiple network paths as constraints.

2. The method for optimizing audio and video communication performance based on multipath transmission according to claim 1, characterized in that: The method comprises: In the process of transmitting the M groups of data packet transmission sequences to the second communication end through the M transmission path combinations, passively monitoring the transmission status of the M transmission path combinations, and performing path classification update according to the monitoring results to obtain M updated path combinations; Using the real-time transmission window at the first communication end to collect audio and video communication data in real time to obtain a second audio and video stream; After performing frame packaging processing on the second audio and video stream, performing transmission path matching optimization and data transmission with the M update path combinations as constraints; By analogy, the real-time collection of audio and video communication data is performed through the real-time transmission window, and the dynamic matching of transmission paths and data transmission are performed based on the update results of the M transmission path combinations as constraints.

3. The method for optimizing audio and video communication performance based on multipath transmission according to claim 1, characterized in that: Actively monitoring multiple network paths and classifying the paths based on the monitoring data to obtain M transmission path combinations, the method comprising: Setting a path performance update window according to the real-time transmission window, wherein the length of the path performance update window is 1 / K of the real-time transmission window; Preconfigured test data packages; In the process of intermittently transmitting the test data packet from the first communication end to the second communication end with the path performance update window as a time constraint, multiple transmission path performance data are collected for the multiple network paths to obtain multiple groups of path performance data, wherein each group of path performance indicators includes K path performance information, and the path performance information includes bandwidth, data packet loss rate, data packet disorder rate, data packet repetition rate, data packet header overhead and delay change rate; Pre-constructing a path performance evaluation model, and obtaining a plurality of path performance quantization coefficients by loading the plurality of groups of path performance data into the path performance evaluation model; The multiple network paths are aggregated according to the multiple path performance quantization coefficients to obtain the M transmission path combinations.

4. The method for optimizing audio and video communication performance based on multipath transmission as claimed in claim 3, characterized in that: Performing transmission path matching optimization on the H audio and video data packets to obtain M groups of data packet transmission sequences, the method comprising: Extracting H data packet attribute information of the H audio and video data packets; Taking the transmission path ratio of the M transmission path combinations as a data packet grouping constraint, and performing priority classification on the H audio and video data packets according to the attribute information of the H data packets to obtain a set of M audio and video data packets; Dividing the plurality of path performance quantization coefficients into M groups of path performance quantization coefficients according to the M transmission path combinations; The transmission loads of the M transmission path combinations are distributed according to the M groups of path performance quantization coefficients, and the M audio and video data packet sets are serialized according to the distribution results to obtain the M groups of data packet transmission sequences.

5. The method for optimizing audio and video communication performance based on multipath transmission according to claim 4, characterized in that: Taking the transmission path ratio of the M transmission path combinations as a data packet grouping constraint, and performing priority classification on the H audio and video data packets according to the H data packet attribute information to obtain M audio and video data packet sets, the method includes: Adding bandwidths of the M transmission path combinations to obtain M total bandwidths; Performing ratio calculation based on the M total bandwidths to obtain the transmission path ratio; Quantify the priorities of the H audio and video data packets according to the attribute information of the H data packets to obtain H priority coefficients; Serialize the H audio and video data packets according to the H priority coefficients to obtain an initial audio and video data packet sequence; Taking the transmission path ratio as a constraint, the initial audio and video data packet sequence is divided to obtain M audio and video data packet subsequences, wherein the M audio and video data packet subsequences respectively correspond to the composition of the M audio and video data packet sets.

6. The method for optimizing audio and video communication performance based on multipath transmission as claimed in claim 5, characterized in that: The method comprises: distributing transmission loads on the M transmission path combinations according to the M groups of path performance quantization coefficients, and serializing the M audio and video data packet sets according to the distribution results to obtain the M groups of data packet transmission sequences. The method comprises: Normalizing the M groups of path performance quantization coefficients to obtain M transmission load ratios; Taking the M transmission load ratios as constraints, dividing the M audio and video data packet subsequences to obtain M groups of initial transmission sequences; The M groups of initial transmission sequences are updated by nested sorting according to the attribute information of the H data packets to obtain the M groups of data packet transmission sequences.

7. The method for optimizing audio and video communication performance based on multipath transmission as claimed in claim 3, characterized in that: Pre-constructing a path performance evaluation model, and obtaining a plurality of path performance quantization coefficients by loading the plurality of groups of path performance data into the path performance evaluation model, the method comprising: A path performance evaluation function is pre-constructed, and the path performance evaluation function is as follows: Among them, S is the path performance quantization coefficient, B is the bandwidth, P is the packet loss rate, O is the packet disorder rate, D is the packet repetition rate, H is the packet header overhead, and V is the delay variation rate; Using the path performance evaluation function to construct the path performance evaluation model; Loading the multiple groups of path performance data into the path performance evaluation model to obtain multiple groups of node performance quantization coefficients, wherein each group of node performance quantization coefficients includes K node performance quantization coefficients; The multiple groups of node performance quantization coefficients are time-series weighted to obtain the multiple path performance quantization coefficients.

8. An audio and video communication performance optimization system based on multipath transmission, characterized in that: The steps for implementing the method according to any one of claims 1 to 7 include: A sliding window adjustment unit, used to adaptively adjust the sliding transmission window according to network conditions to obtain a real-time transmission window; A communication data collection unit, configured to collect audio and video communication data in real time using the real-time transmission window at the first communication end to obtain a first audio and video stream; A transmission path classification unit, used to actively monitor multiple network paths and classify the paths based on the monitoring data to obtain M transmission path combinations, wherein the path classification is dynamically updated based on real-time performance monitoring results; A data packing execution unit, configured to perform frame packing processing on the first audio and video stream to obtain H audio and video data packets; A transmission path optimization unit, used to perform transmission path matching optimization on the H audio and video data packets to obtain M groups of data packet transmission sequences; A data transmission execution unit, configured to transmit the M groups of data packet transmission sequences to a second communication end through the M transmission path combinations; The transmission cycle execution unit is used to perform real-time collection of audio and video communication data through the real-time transmission window, and dynamically match the transmission path and transmit data based on the path classification update results of the multiple network paths as constraints.

Images