Method, apparatus and electronic device for multi-connection based streaming transmission

Abstract

Embodiments of the present application provide a multi-connection-based streaming media transmission method and device, electronic equipment and computer readable storage medium, and relate to the field of communication. The method comprises: determining the dependency degree of each streaming media segment in the to-be-transmitted streaming media; determining each candidate network connection and transmission strategy corresponding to the to-be-transmitted streaming media, the transmission strategy being associated with a target function and a constraint condition of the target function; determining the target network connection of each streaming media segment when the target function reaches an optimal solution under the constraint condition according to the network state of each candidate network connection and the dependency degree of each streaming media segment; and transmitting the streaming media segment through the target network connection of the streaming media segment. Embodiments of the present application flexibly determine the target network connection of each I frame, P frame and B frame in a GoP, and can effectively avoid the occurrence of first frame delay, reduce lag and improve user experience.

Description

Technical Field

[0001] This application relates to the field of communication technology, and more specifically, to a method, apparatus, electronic device, computer-readable storage medium, and computer program product for streaming media transmission based on multiple connections. Background Technology

[0002] Multi-channel multi-connection technology can significantly increase client transmission bandwidth and transmission robustness, thereby allowing the delivery of higher bitrate video streams.

[0003] The existing technology offers the following two solutions:

[0004] Option 1 involves sending a copy of the streaming media on each connection. However, since the receiving bitrate at the decoding end of the streaming media depends on the connection with the smallest bandwidth, this option wastes a significant amount of bandwidth across multiple connections when sending based on all connections due to the limitation of the connection with the smallest bandwidth.

[0005] Option 2 configures the proportion of streaming media traffic carried on different connections according to the real-time measured connection bandwidth and latency jitter. However, this option does not fully consider the impact of each streaming media segment on the user experience, resulting in a lower user experience. Summary of the Invention

[0006] This application provides a multi-connection-based streaming media transmission method, apparatus, electronic device, computer-readable storage medium, and computer program product, which can solve the above-mentioned problems of the prior art. The technical solution is as follows:

[0007] According to one aspect of the embodiments of this application, a multi-connection-based streaming media transmission method is provided, the method comprising:

[0008] Determine the degree of dependency of each streaming media segment in the streaming media to be transmitted. The degree of dependency is used to characterize the interval between the corresponding streaming media segment and the reference unit. The reference unit is the streaming media segment that is farthest from the corresponding streaming media segment among all streaming media segments that have a dependency relationship with the corresponding streaming media segment.

[0009] The candidate network connections and transmission strategies corresponding to the streaming media to be transmitted are determined. The transmission strategies are associated with the objective function and the constraints of the objective function. The objective function is associated with the first evaluation index of each streaming media segment. The constraints are used to determine at least one available network connection corresponding to the streaming media segment from the candidate network connections. The constraints are associated with at least one second evaluation index of the streaming media segment. The first evaluation index is determined based on the network status of the available network connections.

[0010] Based on the network state of each candidate network connection and the degree of dependence of each streaming media segment, the target network connection of each streaming media segment is determined when the objective function reaches the optimal solution under the constraints.

[0011] For each streaming segment, the streaming segment is transmitted via the target network connection of the streaming segment.

[0012] According to another aspect of the embodiments of this application, a multi-connection-based streaming media transmission device is provided, the device comprising:

[0013] The dependency degree determination module is used to determine the dependency degree of each streaming media segment in the streaming media to be transmitted. The dependency degree is used to characterize the interval between the corresponding streaming media segment and the reference unit. The reference unit is the streaming media segment that is farthest from the corresponding streaming media segment among all streaming media segments that have a dependency relationship with the corresponding streaming media segment.

[0014] The connection strategy preparation module is used to determine the candidate network connections and transmission strategies corresponding to the streaming media to be transmitted. The transmission strategy is associated with the objective function and the constraints of the objective function. The objective function is associated with the first evaluation index of each streaming media segment. The constraints are used to determine at least one available network connection corresponding to the streaming media segment from the candidate network connections. The constraints are associated with at least one second evaluation index of the streaming media segment. The first evaluation index is determined based on the network status of the available network connections.

[0015] The target network connection determination module is used to determine the target network connection of each streaming media segment when the objective function reaches the optimal solution under the constraints, based on the network status of each candidate network connection and the degree of dependence of each streaming media segment.

[0016] The transmission module is used to transmit each streaming media segment via the target network connection of the streaming media segment.

[0017] As an alternative embodiment, the degree of dependence is used to weight the first evaluation metric for the corresponding streaming media segment;

[0018] The objective function is associated with the target evaluation index of the streaming media to be transmitted. The target evaluation index is the sum of the weighted first evaluation index of each streaming media segment.

[0019] The target network connection determination module is specifically used to: determine the target evaluation index associated with the objective function based on the network state of each candidate network connection, and determine the target network connection of each streaming media segment when the optimal solution is reached under constraints.

[0020] As an optional embodiment, the target network connection determination module is specifically used to: determine the target network connection of each streaming media segment from the candidate network connections in descending order of the degree of dependence of each streaming media segment;

[0021] The steps for the target network connection determination module to determine the target network connection for each streaming media segment include:

[0022] Determine the network status of each candidate network connection;

[0023] Based on the network state of each candidate network connection, the target network connection of the streaming media segment is determined when the first evaluation index of the streaming media segment associated with the objective function reaches the optimal solution under the constraints.

[0024] Update the network status of the target network connection.

[0025] As an optional embodiment, the dependency determination module is used to:

[0026] For each streaming media segment in the streaming media to be transmitted, if the streaming media to be transmitted includes streaming media segments that are dependent on the corresponding streaming media segment, then a reference unit is determined from the streaming media segments that are dependent on the reference unit, and the number of units between the corresponding streaming media segment and the reference unit is taken as the degree of dependence of the streaming media segment.

[0027] If the resource to be transmitted does not include any streaming media segments that are dependent on the corresponding streaming media segment, then the default value will be used as the degree of dependency of the corresponding streaming media segment.

[0028] As an optional embodiment, the constraints are also associated with at least one third evaluation metric for the candidate network connection, which is related to the network state of the corresponding candidate network connection.

[0029] As an optional embodiment, the target network connectivity determination module includes:

[0030] The available set determination unit is used to determine the third evaluation index of each candidate network connection based on the network state of each candidate network connection, and to take the candidate network connections that meet the constraints of the third evaluation index as available network connections, so as to obtain the available set of the first iteration.

[0031] The reference set unit is used to select the available set after removing available network connections for each available network connection in the available set of the current iteration.

[0032] The indicator determination unit is used to determine a first evaluation indicator for the transmission of streaming media segments based on the network status of the available set; and to determine a first evaluation indicator and a second evaluation indicator for the transmission of streaming media segments based on the network status of the reference set.

[0033] The judgment unit is used to determine the degree of change of the first evaluation index of the streaming media segment based on the available set and the reference set, and to determine whether the second evaluation index of the streaming media segment based on the reference set meets the constraints.

[0034] An iterative unit is used to remove the reference network connection that causes the most significant change from the available set of the current iteration if it is determined that there is at least one reference network connection and the second evaluation metric of the streaming media segment transmitted based on the reference set corresponding to the reference network connection meets the constraints, so as to obtain the available set of the next iteration and proceed to the next iteration.

[0035] The target determination unit is used to stop the iteration if it is determined that there is no reference network connection in the current iteration, and to use the available network connections in the available set of the current iteration as the target network connections of the streaming media segment.

[0036] As an optional embodiment, the first evaluation metric is the number of available network connections for the streaming media segment;

[0037] The second evaluation metric includes: the overall failure rate of the streaming media segment, which is used to characterize the probability that a streaming media segment will fail when transmitted based on the corresponding available network connection;

[0038] The third evaluation metric includes: the receive latency and transmission rate of the candidate network connection.

[0039] As an optional embodiment, the first evaluation metric is the increase in traffic for transmitting streaming media segments across all available network connections;

[0040] The second evaluation metric includes: the overall failure rate of streaming segments;

[0041] The third evaluation metric includes: the receive latency and transmission rate of the candidate network connection.

[0042] As an optional implementation, the first evaluation metric is the overall failure rate of the streaming media segment;

[0043] The second evaluation metric includes: the number of available network connections corresponding to the streaming segment;

[0044] The third evaluation metric includes: the receive latency and transmission rate of the candidate network connection.

[0045] As an optional implementation, the first evaluation metric is the overall failure rate of the streaming media segment;

[0046] The second evaluation metric includes: unit traffic of a streaming media segment, which is the traffic of the streaming media segment when all available network connections are used to transmit the streaming media segment;

[0047] The third evaluation metric includes: the receive latency and transmission rate of the candidate network connection.

[0048] According to another aspect of the embodiments of this application, an electronic device is provided, which includes a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the above-described multi-connection-based streaming media transmission method.

[0049] According to another aspect of the embodiments of this application, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the steps of the above-described multi-connection-based streaming media transmission method.

[0050] According to one aspect of the embodiments of this application, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps of the above-described multi-connection-based streaming media transmission method.

[0051] The beneficial effects of the technical solutions provided in this application are:

[0052] The streaming media transmission method based on multiple connections in this application determines the dependency degree of each streaming media segment in the streaming media to be transmitted. The dependency degree represents the interval between the corresponding streaming media segment and the reference unit. The dependency degree can reflect the range of the largest neighboring frames that the streaming media segment may affect when the transmission fails. In particular, when the streaming media segment is a single frame, the dependency degree of I-frames, B-frames and P-frames will be significantly different. Further, it determines each candidate network connection and transmission strategy corresponding to the streaming media to be transmitted. Based on the network status of each candidate network connection and the dependency degree of each streaming media segment, it determines the target network connection of each streaming media segment when the objective function reaches the optimal solution under the constraints. This enables flexible determination of the target network connection for I-frames, P-frames and B-frames within each GoP, and can effectively avoid the occurrence of first frame delay, reduce stuttering and improve user experience. Attached Figure Description

[0053] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments of this application will be briefly introduced below.

[0054] Figure 1 This application provides a schematic diagram of the system architecture for implementing a multi-connection-based streaming media transmission method.

[0055] Figure 2A flowchart illustrating a multi-connection-based streaming media transmission method provided in an embodiment of this application;

[0056] Figure 3 A view of the dependencies between streaming media segments in a GoP provided for embodiments of this application;

[0057] Figure 4 A flowchart illustrating a process for determining a target network connection for a streaming media segment, provided as an embodiment of this application;

[0058] Figure 5 A schematic diagram of a multi-connection-based streaming media transmission device is provided in an embodiment of this application;

[0059] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0060] The embodiments of this application are described below with reference to the accompanying drawings. It should be understood that the embodiments described below with reference to the accompanying drawings are exemplary descriptions for explaining the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions of the embodiments of this application.

[0061] Those skilled in the art will understand that, unless otherwise stated, the singular forms “a,” “an,” and “the” used herein may also include the plural forms. It should be further understood that the terms “comprising” and “including” as used in embodiments of this application mean that the corresponding feature can be implemented as the presented feature, information, data, step, operation, element, and / or component, but do not exclude implementation as other features, information, data, step, operation, element, component, and / or combinations thereof supported by the art. It should be understood that when we say that an element is “connected” or “coupled” to another element, the one element can be directly connected or coupled to the other element, or it can mean that the one element and the other element establish a connection relationship through an intermediate element. Furthermore, “connected” or “coupled” as used herein can include wireless connection or wireless coupling. The term “and / or” as used herein indicates at least one of the items defined by the term; for example, “A and / or B” can be implemented as “A,” or as “B,” or as “A and B.”

[0062] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0063] First, let's introduce and explain several terms used in this application:

[0064] Streaming media refers to a type of multimedia in which multimedia data is continuously transmitted from a streaming media provider to the client, and the client can start playing the multimedia without downloading the entire multimedia data to the local device.

[0065] A channel is a data transmission channel established between a physical network interface card and an access point.

[0066] A connection is an abstract data transmission channel uniquely determined by a source triplet <src_ip source IP address, src_port source port, protocol> and a destination triplet <dst_ip destination IP address, dst_port destination port, protocol>.

[0067] Multiple connections mean that there are multiple connections between the source end and the destination end.

[0068] A video frame is a static complete picture that can be decompressed and rendered by a decoder. A dynamically changing video is formed by quickly refreshing different frames on a display.

[0069] Group of Pictures (GoP) is a video frame compression unit defined in the MPEG-4 standard, and it is the smallest unit that can be completely decoded, rendered and displayed by a Moving Pictures Experts Group (MPEG) decoder. When a GoP is incomplete, for example, an I-frame is missing, it may cause problems such as inability to decode, decreased definition, or screen corruption.

[0070] An I-frame, also called a key frame, is the most important video frame carried in a GoP. Usually, each GoP has one and only one I-frame, which is located at the position of the first frame. Generally, the absence of an I-frame will make the GoP undecodable.

[0071] A P-frame, or forward predictive frame, is a frame compressed by referring to pixel information of several previous frames. It depends on previous frames and the I-frame for decoding. Generally, loss of a P-frame will cause screen corruption or stuttering of the video in the GoP.

[0072] A B-frame, or bi-directional predictive frame, is a frame compressed by referring to pixel information of several previous frames and several subsequent frames. Generally, loss of a non-reference B-frame will not cause video stuttering or screen corruption, while loss of a reference B-frame may lead to screen corruption or stuttering in the GoP.

[0073] Receiving delay is the time elapsed from when a video frame is output from an encoder to the transmitting interface, goes through smooth transmission inside the interface, propagates through the network connection to the opposite end, until the opposite end receives the complete frame.

[0074] First-frame delay is the time consumed by a mobile device from initiating a handshake to displaying the first frame of picture on the display screen, which is usually an important influencing factor of user experience.

[0075] Stuttering refers to the maximum inter-frame latency in normally playing video, which is typically greater than 200ms.

[0076] Cloud live streaming user experience refers to the user's subjective evaluation of the video playback. Generally speaking, the lower the latency of the first frame, the better the user experience; the less stuttering, the better the user experience.

[0077] The streaming media transmission method, apparatus, electronic device, computer-readable storage medium, and computer program product based on multiple connections provided in this application are intended to solve the above-mentioned technical problems of the prior art.

[0078] The technical solutions of this application and their effects are described below through several exemplary embodiments. It should be noted that the following embodiments can be referenced, borrowed from, or combined with each other. Identical terms, similar features, and similar implementation steps in different embodiments will not be repeated.

[0079] Figure 1 A schematic diagram of a system architecture for implementing a multi-connection-based streaming media transmission method provided in an embodiment of this application is shown. The system 100 includes multiple devices that can communicate with each other, for example, via a network 150. For example, the communication system 100 includes a first device 110 and a second device 120 interconnected via the network 150. Figure 1 In this embodiment, the first device 110 and the second device 120 perform unidirectional data transmission. For example, the first device 110 may encode streaming media, such as a video image stream captured by the first device 110, for transmission over network 150 to the second device 120. The encoded video data is transmitted in the form of one or more streaming media segments. The second device 120 may receive the encoded streaming media segments from network 150, decode the encoded streaming media segments to recover the streaming media, and display video images based on the recovered streaming media.

[0080] In another embodiment, the communication system 100 includes a third device 130 and a fourth device 140 that perform bidirectional transmission of encoded video data, which may occur, for example, during a video conference. For bidirectional data transmission, each of the third device 130 and the fourth device 140 may encode streaming media data (e.g., a stream of video images captured by the device) for transmission over network 150 to the other device. Each of the third device 130 and the fourth device 140 may also receive encoded streaming media segments transmitted by the other device, decode the encoded streaming media segments to recover video data, and display video images on an accessible display device based on the recovered streaming media.

[0081] exist Figure 1 In the embodiments described herein, the first device 110, the second device 120, the third device 130, and the fourth device 140 may be computer devices such as servers, personal computers, and smartphones, but the principles disclosed in this application are not limited to these. The embodiments of this application are applicable to PCs (Personal Computers), mobile phones, tablets, media players, and / or dedicated video conferencing equipment. Network 150 refers to any number of networks that transmit encoded video data between the first device 110, the second device 120, the third device 130, and the fourth device 140, including, for example, wired and / or wireless communication networks. Communication network 150 may exchange data in circuit-switched and / or packet-switched channels. This network may include telecommunications networks, local area networks, wide area networks, and / or the Internet. For the purposes of this application, unless explained below, the architecture and topology of network 150 may be irrelevant to the operation of the disclosures herein.

[0082] This application provides a multi-connection-based streaming media transmission method, such as... Figure 2 As shown, the method includes:

[0083] S101. Determine the degree of dependency of each streaming media segment in the streaming media to be transmitted.

[0084] The streaming media to be transmitted in the embodiments of this application can be an audio stream, video stream, text stream, image stream, animation stream, etc. The streaming media can include at least one streaming media segment. Taking a video stream as an example, the streaming media segment in the embodiments of this application can refer to a video frame or a GoP (GoP). That is, the streaming media transmission method in the embodiments of this application is applicable to frame-level scheduling and also to GoP-level scheduling.

[0085] The degree of dependency is used to characterize the interval between the corresponding streaming media segment and the reference unit, which is the streaming media segment that is furthest from the corresponding streaming media segment among all streaming media segments that have a dependency relationship with the corresponding streaming media segment.

[0086] In streaming media transmission, the source end typically does not send complete frames to the destination end each time, but only the parts of the subsequent frame that differ from the previous frame. Therefore, in streaming media transmission, if a streaming segment A needs to reference another streaming segment B during decoding, it is said that streaming segment A and streaming segment B have a dependency relationship, with streaming segment B being the dependent streaming segment and streaming segment A being the dependent streaming segment. It should be understood that the dependency relationship usually only exists within video frames of the same GoP. Therefore, in determining the degree of dependency of each streaming segment in this embodiment, the determination is the degree of dependency of each streaming segment within a GoP. Furthermore, for each streaming segment, the streaming segments that have a dependency relationship with that streaming segment refer to the streaming segments whose decoding order is after that streaming segment. That is to say, the decoding order of the reference unit in this embodiment must also be after the corresponding streaming segment.

[0087] Please see Figure 3 This example illustrates a view of the dependencies between streaming media segments (hereinafter referred to as frames) within a GoP. In the figure, GoP1 includes 11 video frames. For frame I0, the furthest B within the GoP can be seen. 11 The frame depends on frame P8, and frame P8 depends on frame I0. That is, starting from frame I0, the frame with the furthest dependency relationship is the reference unit B. 11 The frames are spaced 11 times. In this embodiment, the furthest backward dependency is defined as the frame's dependency level d, representing the maximum range of neighboring frames that may be affected when the frame's first transmission fails. That is, the dependency level of frame I0 is denoted as d(I0).

[0088] In one embodiment, determining the dependency level of each streaming media segment in the streaming media to be transmitted includes:

[0089] For each streaming media segment in the streaming media to be transmitted, if the resource to be transmitted includes streaming media segments that are dependent on the corresponding streaming media segment, then a reference unit is determined from the dependent streaming media segments, and the number of units between the corresponding streaming media segment and the reference unit is taken as the degree of dependency of the streaming media segment. For example, in Figure 3 In the illustrated embodiment, the dependency level of the I0 frame is 11.

[0090] It is important to note that when the streaming media segment is a GoP, since there are no dependencies between GoPs, the degree of dependency can be replaced by a preset value. In one embodiment, if the resource to be transmitted does not include streaming media segments that are dependent on the corresponding streaming media segment, then the preset value is used as the degree of dependency of the corresponding streaming media segment.

[0091] For example, in Figure 3 In the illustrated embodiment, none of the B-frames are used as dependent frames, so there are no reference units, and the dependency level of all B-frames can be set to a preset value. When the streaming media segment is a GoP, since each GoP is independent, the dependency level can also be set to a preset value. In one embodiment, the preset value in this application is a non-zero value, such as 1, and this application does not impose specific limitations.

[0092] S102. Determine the candidate network connections and transmission strategies corresponding to the streaming media to be transmitted.

[0093] It should be understood that the source and destination ends of the streaming media are predetermined before transmission. The source end is the sender of the streaming media, and the destination end is the receiver. Furthermore, a triplet of the source and destination ends can be obtained. Multiple candidate connections are obtained by combining these triplets. It is understood that the network states of different candidate connections may differ. The network state in this embodiment may include transmission rate, reception latency, remaining bandwidth, etc.

[0094] This application embodiment can pre-create multiple transmission strategies, each associated with an objective function and constraints. Different transmission strategies have different objective functions and constraints, allowing the source end to select different transmission strategies based on different transmission purposes.

[0095] It should be noted that since this application requires obtaining the network status of candidate network connections to determine the target network connection, as a special case, that is, when the network status of candidate networks is not obtained—for example, when transmitting the first GoP video frame, a simple multi-transmission strategy can be adopted. That is, the first GoP I frame is sent on each candidate network connection at the default transmission rate. After obtaining the network status of each candidate network connection, the target network connection is selected based on the obtained network status of the candidate network connections.

[0096] In an alternative embodiment, when the network status of a network connection, such as bandwidth, latency, and packet loss rate, is obtained, the network connection can be used as a candidate network connection.

[0097] The objective function of this application is associated with a first evaluation metric for each streaming media segment in the streaming media to be transmitted. It is understood that the first evaluation metric is an indicator used to evaluate the transmission quality of the streaming media segment, such as the number of connections, success rate, and traffic surge. In some embodiments, the objective function can be used to independently evaluate the first evaluation metric for each streaming media segment. In other embodiments, the first evaluation metric for all streaming media segments evaluated by the objective function can be used to evaluate the transmission quality of the streaming media to be transmitted.

[0098] The objective function in this application embodiment also has constraints, that is, this application aims to obtain the optimal solution of the objective function under certain constraints. The constraints are associated with at least one second evaluation metric for the streaming media segment. It is understood that the second evaluation metric is different from the first evaluation metric and is used to evaluate the transmission quality of the streaming media segment.

[0099] The constraints in this application embodiment are used to determine at least one available network connection corresponding to a streaming media segment from each candidate network connection. That is, by combining different candidate network connections and judging whether the second evaluation index meets the constraints when transmitting streaming media segments based on the combination, it is determined whether the combination can be used as an available network connection for the streaming media segment. Thus, it can be seen that each streaming media segment has its own corresponding available network connection, and there are at least one combination of available network connections.

[0100] The first evaluation metric in this embodiment is determined based on the network status of available network connections. Through constraints associated with the second evaluation metric, available network connections for each streaming media segment are filtered from candidate network connections. Then, the first evaluation metric for each streaming media segment is determined based on the network status of the available network connections. In other words, after obtaining at least one set of available network connections for each streaming media segment, the optimal set of available network connections is found by calculating the first evaluation metric corresponding to each set of available network connections.

[0101] S103. Based on the network state of each candidate network connection and the degree of dependence of each streaming media segment, determine the target network connection of each streaming media segment when the objective function reaches the optimal solution under the constraints.

[0102] Compared to existing technologies that determine the optimal solution of the objective function under constraints solely based on the network state of candidate network connections, this application's embodiments also consider the degree of dependency of each streaming media segment. The degree of dependency in determining the optimal solution is applied in the following two ways in this application's embodiments:

[0103] In the first approach, the degree of dependence is used to weight the first evaluation metric for streaming segments in the objective function.

[0104] In this approach, the weight of the first evaluation metric for a streaming media segment can be determined by utilizing the degree of dependency of the segments. Therefore, when the degree of dependency varies, the weighted first evaluation metric for each segment also differs. The objective function is then correlated with the weighted first evaluation metrics of each segment, allowing the optimal solution to be determined based on the weighted first evaluation metrics of all segments. This approach differentiates the importance of the first evaluation metric for different streaming media segments in obtaining the overall optimal solution, thereby determining the target network connections for each segment and improving the overall transmission performance of the streaming media.

[0105] Method 2 uses the degree of dependency to determine the order of operations for each streaming media segment.

[0106] For any two streaming media segments, the target network connection of the streaming media segment with the earlier processing order is prioritized. When the streaming media to be transmitted includes streaming media segments that belong to multiple video frames in a GoP, since the dependency degree of I-frames is greater than that of P-frames, and the dependency degree of P-frames is greater than that of B-frames, and the latency of the first frame (generally I-frame) is usually an important factor affecting user experience, this application embodiment can set the higher the dependency degree, the earlier the processing order. The earlier the streaming media segment is processed, the more candidate network connections can be selected, and the better the network condition, which can effectively avoid the occurrence of first frame latency and improve the viewer experience.

[0107] S104. For each streaming media segment, transmit the streaming media segment through the target network connection of the streaming media segment.

[0108] Once the target network connection for each streaming media segment is determined, the streaming media segment can be transmitted from the source to the destination based on the target network connection for each streaming media segment.

[0109] The streaming media transmission method based on multiple connections in this application determines the dependency degree of each streaming media segment in the streaming media to be transmitted. The dependency degree represents the interval between the corresponding streaming media segment and the reference unit. The dependency degree can reflect the range of the largest neighboring frames that the streaming media segment may affect when the transmission fails. In particular, when the streaming media segment is a single frame, the dependency degree of I-frames, B-frames and P-frames will be significantly different. Further, it determines each candidate network connection and transmission strategy corresponding to the streaming media to be transmitted. Based on the network status of each candidate network connection and the dependency degree of each streaming media segment, it determines the target network connection of each streaming media segment when the objective function reaches the optimal solution under the constraints. This enables flexible determination of the target network connection for I-frames, P-frames and B-frames within each GoP, and can effectively avoid the occurrence of first frame delay, reduce stuttering and improve user experience.

[0110] Based on the above embodiments, as an optional embodiment, the degree of dependence is used to weight the first evaluation metric of the corresponding streaming media segment. That is, the degree of dependence in this application embodiment is used to determine the weight of the corresponding streaming media segment, thereby weighting the first evaluation metric of the corresponding streaming media segment.

[0111] Accordingly, the objective function is associated with the target evaluation index of the streaming media to be transmitted, which is the sum of the weighted first evaluation index of each streaming media segment.

[0112] In this embodiment, connection scheduling is performed with the overall evaluation metric (i.e., target evaluation metric) of the streaming media to be transmitted as the objective. Specifically, the target evaluation metric is obtained by weighting and summing the first evaluation metrics of the streaming media segments based on the degree of dependency of each streaming media segment. By using the degree of dependency of each streaming media segment as the weight for the first evaluation metric of the streaming media segment, this embodiment enables the first evaluation metric of the streaming media segment to better reflect the overall target evaluation metric.

[0113] It should be understood that the relationship between the degree of dependence and the weight is related to the type of the primary evaluation metric. In some cases, a smaller primary evaluation metric means better transmission performance. For example, a smaller number of target network connections for a streaming media segment means fewer copies of the segment, resulting in better transmission. In this case, the degree of dependence can be inversely proportional to the weight; that is, the greater the degree of dependence, the smaller the weight of the corresponding streaming media segment. Thus, even if the number of target network connections for the streaming media segment is higher, its impact on the target evaluation metric will be reduced due to the weight. Clearly, this approach is beneficial for identifying more target network connections for streaming media segments with a high degree of dependence, thereby improving the transmission performance of these segments and enhancing the user experience.

[0114] Based on this, according to the network state of each candidate network connection and the degree of dependence of each streaming media segment, the target network connections of each streaming media segment are determined when the objective function reaches the optimal solution under the constraints, including:

[0115] Based on the network state of each candidate network connection, the objective evaluation index associated with the objective function is determined, and the target network connection of each streaming media segment is determined when the optimal solution is reached under constraints.

[0116] Since the objective evaluation index associated with the objective function is the sum of a weighted first evaluation index based on each streaming media segment, the embodiments of this application are equivalent to determining the target network connection of each streaming media segment from the perspective of the overall transmission effect of the streaming media to be transmitted.

[0117] Taking the streaming media to be transmitted, which includes streaming media segments belonging to multiple video frames in a GoP, as an example, since the dependence of I-frames is greater than that of P-frames, and the dependence of P-frames is greater than that of B-frames, when calculating the target evaluation index in this embodiment of the application, I-frames are more important to the target evaluation index than P-frames, and P-frames are more important to the target evaluation index than B-frames. Therefore, when determining the target transmission index of the streaming media to be transmitted, determining the weight based on the degree of dependence is beneficial to better determine the target network connection of I-frames and P-frames while ensuring the overall target evaluation index, thereby improving the user's browsing experience.

[0118] Based on the above embodiments, as an optional embodiment, the target network connections for each streaming media segment are determined according to the network state of each candidate network connection and the degree of dependence of each streaming media segment, including:

[0119] Based on the order of dependence of each streaming segment from highest to lowest, the target network connection for each streaming segment is determined sequentially from the candidate network connections.

[0120] In other words, in this embodiment of the application, the target network connection of the streaming media segment with a high degree of dependence will be determined first. Since the network status of the network connection may deteriorate as the number of streaming media segments transmitted increases, the streaming media segment with the target network connection determined first will have more candidate network connections to choose from, and the network status of the selected target network connection will also be better.

[0121] The steps for determining the target network connection for each streaming media segment in this embodiment include:

[0122] Determine the network status of each candidate network connection;

[0123] Based on the network state of each candidate network connection, the target network connection of the streaming media segment is determined when the first evaluation index of the streaming media segment associated with the objective function reaches the optimal solution under the constraints.

[0124] Update the network status of the target network connection.

[0125] Because this application independently determines the target network connection for each streaming media segment using an objective function, it is necessary to determine the latest network state of each candidate network connection each time the target network connection for a streaming media segment is determined. Then, based on the network state of each candidate network connection, the target network connection of the streaming media segment associated with the objective function is determined when the first evaluation index of the streaming media segment reaches the optimal solution under the constraints. After determining the target network connection of the streaming media segment, this application embodiment can directly send the streaming media segment through the target network connection. That is, the degree of dependence also determines the transmission order of the streaming media segments in this application embodiment.

[0126] It should be noted that the objective function of this application embodiment is only related to the first evaluation index of a corresponding streaming media segment. Therefore, when determining the optimal solution of the objective function, the amount of computation is greatly reduced compared to the objective function related to the objective evaluation index (obtained based on the first evaluation index of all streaming media segments). This application embodiment can be applied to schemes with a larger number of connections.

[0127] Based on the above embodiments, the embodiments of this application further include:

[0128] If the number of candidate network connections is determined to be no greater than a preset threshold, the degree of dependence is used to weight the first evaluation metric of the corresponding streaming media segment, or to determine the operation order of the streaming media segment.

[0129] If the number of candidate network connections is determined to be greater than a preset threshold, the degree of dependence is used to determine the operation order of streaming media segments.

[0130] In other words, when the number of candidate network connections does not exceed the preset threshold, the degree of dependency can be used for both weighting and determining the order of operations. However, when the number of candidate network connections exceeds the preset threshold, in order to reduce the amount of computation and improve the efficiency of computation, the dependent program is only used to determine the order of operations.

[0131] Based on the above real-time examples, as an optional embodiment, the constraints are also associated with at least one third evaluation metric for the candidate network connection, which is related to the network state of the corresponding candidate network connection.

[0132] The constraints of this application embodiment are associated not only with the second evaluation index of the streaming media segment, but also with at least one third evaluation index of the candidate network connection. The third evaluation index is related to the network state of the corresponding candidate network connection. That is, the third evaluation index of each candidate network connection is only related to its own network state and is not related to other candidate network connections.

[0133] In determining the optimal solution of the objective function, a third evaluation metric can be determined for each candidate network connection based on its network state. Candidate network connections whose third evaluation metric does not meet the constraints are first excluded. For the remaining candidate network connections, since this application is based on a multi-connection transmission method, different candidate network connections are combined into different connection sets in different ways. It should be noted that the number of candidate network connections in a connection set can be one or more. A second evaluation metric for transmitting streaming media segments is determined based on the network state of the connection set. If the second evaluation metric meets the constraints, the connection set can be considered a group of available network connections.

[0134] After obtaining a set of available network connections, the first evaluation index of the corresponding streaming media segment is determined based on the network status of the available network connections. Based on the first evaluation index of all streaming media segments and the degree of dependence, a solution to the objective function can be obtained. By comparing all solutions, the optimal solution can be obtained, and then the target network connection corresponding to each streaming media segment at the optimal solution can be determined.

[0135] Since the first and second evaluation metrics for transmitting streaming media segments may differ depending on the available network connections, by continuously combining different network connections, we can eventually obtain the target network connection for each streaming media segment that enables the objective function to reach the optimal solution under constraints.

[0136] Based on the above embodiments, as an optional embodiment, when the degree of dependence of the embodiments of this application is used to determine the operation order of each streaming media segment, the target network connection of each streaming media segment can be determined based on a greedy algorithm.

[0137] A greedy algorithm is one that, when solving a problem, always makes the choice that seems best at the current moment. In other words, it doesn't consider the overall optimal solution, but rather arrives at a locally optimal solution in some sense. The characteristic of a greedy algorithm is that it proceeds step by step, making the optimal choice based on the current situation and a certain optimization metric, without considering all possible overall scenarios. This saves a significant amount of time that would otherwise be spent exhaustively searching for the optimal solution. Greedy algorithms use a top-down, iterative approach, making successive greedy choices. Each greedy choice simplifies the problem into a smaller subproblem, and through each step of the greedy choice, an optimal solution to the problem can be obtained.

[0138] When applying the greedy algorithm to this application, since the greedy algorithm only involves evaluation and sorting, the complexity of sorting and evaluation is O(nlog(n)), which is extremely low.

[0139] Based on the above embodiments, as an optional embodiment, please refer to... Figure 4 The figure exemplifies the process of determining the target network connection of a streaming media segment based on the network status of each candidate network connection, when the first evaluation index of the streaming media segment associated with the objective function reaches the optimal solution under constraints, as shown in the figure. The process includes:

[0140] S201. Determine the third evaluation index for each candidate network connection based on the network status of each candidate network connection.

[0141] S202. Select candidate network connections that meet the constraints of the third evaluation metric as available network connections to obtain the available set for the first iteration.

[0142] In determining the target network connection for each streaming media segment, this embodiment first determines a third evaluation metric based on the network status of each candidate network connection. Candidate network connections that meet the preset conditions for the third evaluation metric are included as available network connections in the first iteration's available set. As the iteration progresses, the number of available network connections in the available set gradually decreases.

[0143] S203. For each available network connection in the available set of the current iteration, the available set after removing the available network connections is used as the reference set.

[0144] In each iteration of this application, each available network connection in the current available set is removed from the available set to obtain a reference set corresponding to that available network connection. For example, if the available set in the current iteration includes available network connections A, B, and C, then the reference set corresponding to available network connection A includes available network connections B and C, the reference set corresponding to available network connection B includes available network connections A and C, and the reference set corresponding to available network connection C includes available network connections A and B. This application determines multiple reference sets through an elimination method, and then determines the available network connections to be removed in this iteration by assessing the degree of change in the first evaluation index of the reference sets and the available set.

[0145] S204. Based on the network status of the available set, determine the first evaluation index for the transmission of streaming media segments based on the available set; based on the network status of the reference set, determine the first evaluation index and the second evaluation index for the transmission of streaming media segments based on the reference set.

[0146] According to the embodiments of this application, the first evaluation index of streaming media segment transmission based on the available set can be determined by utilizing the network status of all available network connections in the available set in the current iteration. Furthermore, after the previous step of obtaining the reference set, at least one reference set can be obtained. For each reference set, the first evaluation index and the second evaluation index of streaming media segment transmission based on the reference set can be determined by utilizing the network status of all available network connections in the reference set.

[0147] S205. Determine the degree of change of the first evaluation metric for the transmission of streaming media segments based on the available set and the reference set, and determine whether the second evaluation metric for the transmission of streaming media segments based on the reference set meets the constraints.

[0148] It is understandable that the trend of the first evaluation metric may differ when the first evaluation metric is different. For example, when the first evaluation metric is the number of target network connections, the first evaluation metric gradually decreases; when the first evaluation metric is the failure rate, the first evaluation metric gradually increases. This step only needs to focus on the degree of change of the first evaluation metric, and does not need to focus on whether this change is positive or negative. In an optional embodiment, the degree of change can be represented by the absolute value of the difference between the first evaluation metric corresponding to the available set and the reference set.

[0149] S206. Determine whether there is at least one reference network connection and whether the second evaluation index of the streaming media segment transmitted based on the reference set corresponding to the reference network connection meets the constraint conditions. If it exists, execute S207; if it does not exist, execute S208.

[0150] S207. Remove the reference network connection that causes the most significant change from the available set of the current iteration to obtain the available set for the next iteration, and return to step S203 for the next iteration.

[0151] In this embodiment, for each available network connection, the decision to stop the iteration is made based on whether the second evaluation metric for the transmission of a streaming media segment based on the reference set corresponding to the available network connection meets the constraints. If the second evaluation metric for the transmission of a streaming media segment based on the reference set corresponding to an available network connection meets the constraints, then the available network connection is taken as a reference network connection. After determining all reference network connections for the current iteration, the reference network connection that causes the most significant change in the first evaluation metric is selected from all reference network connections and removed from the available set for the current iteration. This yields the available set for the next iteration, and the next iteration begins.

[0152] S208. If it is determined that there is no reference network connection in the current iteration, the iteration is stopped, and the available network connections in the available set of the current iteration are used as the target network connections for the streaming media segment.

[0153] If there is no reference network connection in the available set of the current iteration, it means that removing any available network connection from the available set of the current iteration will result in the constraint not being met. Therefore, the iteration needs to be stopped, and the available network connections in the available set of the current iteration are used as the target network connections for the streaming media segment.

[0154] Based on the above embodiments, as an optional embodiment, the transmission strategy of this application embodiment is a first transmission strategy. The purpose of the first transmission strategy is to ensure low latency and high transmission success rate with as few frame copies as possible.

[0155] The first evaluation metric for the first transmission strategy is the number of available network connections corresponding to the streaming media segment.

[0156] It should be understood that for each additional network connection a streaming media segment is transmitted based on, the source needs to create an additional copy of the streaming media segment. Obviously, creating a copy increases the transmission cost. Therefore, when the number of available network connections corresponding to a streaming media segment is used as the primary evaluation metric, the number of available network connections corresponding to the streaming media segment should be as small as possible. Correspondingly, when the primary evaluation metric of each streaming media segment reaches its minimum value, the objective function is considered to have reached its optimal solution.

[0157] As can be seen from the above embodiments, when the objective function of the first transmission strategy is associated with the target evaluation index of the streaming media to be transmitted, the expression of the objective function of the first transmission strategy in this application embodiment can be:

[0158]

[0159] Where f represents any streaming media segment in the streaming media F to be transmitted, d f J represents the degree to which streaming segment f is dependent. f This represents the set of available network connections corresponding to a streaming media segment f.

[0160] The second evaluation metric of the objective function of the first transmission strategy is the overall failure rate of the streaming media segment, which is used to characterize the probability that a streaming media segment will fail when transmitted based on the corresponding available network connection.

[0161] In calculating the overall failure rate, embodiments of this application can obtain the probability of transmission failure for each available network connection, which serves as the transmission failure rate for each available network. The overall failure rate for transmission based on a set of available network connections can be obtained by multiplying the transmission failure rates for a set of available network connections. In one embodiment, the expression for the overall failure rate can be:

[0162]

[0163] Where, r f φ represents the transmission rate of a streaming media segment f over a network connection. j (f,r f ) indicates that network connection j uses a transmission rate r f The success rate of transmitting streaming media segment f.

[0164] In one embodiment, the expression for the success rate of transmitting streaming media segment f via network connection j can be:

[0165]

[0166] Among them, b j This represents the remaining bandwidth of network connection j. f B represents the set of streaming media segments that network connection j needs to transmit simultaneously when transmitting streaming media segment f. j p represents the total bandwidth of network connection j. j This represents the bandwidth-delay product (BDP). j This represents the random packet loss rate. The expression contains... This represents the queue buffer overflow rate.

[0167] This application embodiment can be based on the total bandwidth B of network connection j. j and the propagation delay δ measured on network connection j j To obtain the bandwidth-delay product, in one embodiment, the expression for the bandwidth-delay product of network connection j can be: p j = j ×δ j .

[0168] In one embodiment, when B is not obtained j In this case, the expression for transmission success rate can also be modified to:

[0169]

[0170] That is, the expression for the transmission failure rate is:

[0171] It should be noted that if only B is obtained... j However, no b was obtained. j Alternatively, b can be simply considered as j =B j In some embodiments, upon obtaining B j In this case, b can be obtained by combining the size of the contention flow. j Alternatively, you can rely on the default configuration value to set B. j and bj .

[0172] The third evaluation metric for the first transmission strategy in this application embodiment includes the reception delay and transmission rate of the candidate network connection. It should be noted that the speed at which network connection j sends streaming media segment f is denoted as... Corresponding reception delay If r1>r2, then σ j (r1)<σ j (r1). That is, the faster a network connection transmits streaming media segments, the smaller the reception latency.

[0173] In one embodiment, the expression for the reception delay of streaming media segment f on network connection j can be: Among them, s f Represented as the size of a streaming segment, δ j This indicates propagation delay.

[0174] The expression associated with the first transmission strategy in this application embodiment includes:

[0175]

[0176]

[0177]

[0178]

[0179] The target evaluation metric of the first transmission strategy in this application is the weighted sum of the number of available network connections for each streaming media segment, calculated as the inverse of the dependency level of each streaming media segment. The goal of the first transmission strategy is to minimize the target evaluation metric. The first transmission strategy also includes three constraints:

[0180] Constraint (1) is that the maximum transmission delay of any streaming media segment f is less than the maximum tolerable delay κ of that streaming media segment f. f ;

[0181] Constraint (2) is that the transmission rate of any available network connection j is less than the remaining bandwidth b of that available network connection j. j ;

[0182] Constraint (3) is that the overall failure rate of any streaming segment f is lower than the failure threshold ∈.

[0183] It should be understood that for network connection j, if its estimated delay is greater than the maximum tolerable delay... Therefore, it is unnecessary to transmit streaming media segment f on network connection j. Thus, unsuitable connections j can be pre-selected using the maximum tolerable delay, i.e., set J can be used. f , must meet

[0184] For network connections, the transmission rate must be kept as low as possible, which is determined by the relationship between the receiving delay and the maximum tolerable delay of the streaming media segment. The minimum transmission speed for each network connection can be calculated. Make To put it more simply, assuming the original bitrate of the video is v, the minimum sending speed... That is, the minimum transmission speed is an integer multiple of the original bit rate.

[0185] Based on the above embodiments, when the objective function of the first transmission strategy is associated with the first evaluation index of the streaming media segment, the expression of the objective function of the first transmission strategy in this application embodiment can be:

[0186] minlo(|J f |

[0187] Where f represents any streaming media segment in the streaming media F to be transmitted, d f J represents the degree to which streaming segment f is dependent. f This represents the set of target network connections corresponding to streaming media segment f.

[0188] The expression for the first transmission strategy in this application embodiment includes:

[0189] minlo(|J f |

[0190]

[0191]

[0192]

[0193] The first transmission strategy of this application embodiment is to minimize the number of available network connections for streaming media segment f under three constraints:

[0194] Constraint (1) is that the maximum transmission delay of streaming media segment f is less than the maximum tolerable delay κ of streaming media segment f. f ;

[0195] Constraint (2) is that the transmission rate of any available network connection j is less than the remaining bandwidth b of that available network connection j. j ;

[0196] Constraint (3) is that the overall failure rate of the streaming segment f is lower than the failure threshold ∈.

[0197] Based on the above embodiments, the transmission strategy of this application embodiment is a second transmission strategy. The purpose of the second transmission strategy is to ensure low latency and high transmission success rate with as little traffic increase as possible.

[0198] The first evaluation metric for the second transmission strategy is the increase in traffic for transmitting streaming media segments across all available network connections.

[0199] The traffic increase referred to in this application embodiment refers to the ratio of the bandwidth required for transmitting streaming media segments via a network connection to the bandwidth already used. It can be understood that, given a fixed required bandwidth, the larger the used bandwidth, the smaller this ratio, and thus the smaller the traffic increase. A smaller traffic increase results in less network fluctuation caused by the transmission of streaming media segments. Therefore, when this application embodiment uses the traffic increase of all available network connections for transmitting streaming media segments as the first evaluation metric, it is desirable for the traffic increase corresponding to each streaming media segment to be as small as possible. Correspondingly, when the first evaluation metric for each streaming media segment reaches its minimum value, the objective function is considered to have reached its optimal solution.

[0200] As can be seen from the above embodiments, when the objective function of the second transmission strategy is associated with the target evaluation index of the streaming media to be transmitted, the expression of the objective function of the second transmission strategy in this application embodiment can be:

[0201]

[0202] in, This represents the increase in traffic for transmitting streaming media segment f on available network connection j. It can be understood that this increase occurs when available network connections are combined in different ways to form different sets of available connections J. f hour, The values ​​may differ. The purpose of this application embodiment is to determine the available set of each streaming media segment that minimizes the objective function. The available network connections included in this available set are the target network connections.

[0203] The second evaluation metric for the second transmission strategy includes the overall failure rate of streaming media segments, and the third evaluation metric includes the reception latency and transmission rate of candidate network connections.

[0204] The expression associated with the second transmission strategy in this application embodiment includes:

[0205]

[0206]

[0207]

[0208]

[0209] The target evaluation metric of the second transmission strategy in this application is the result of a weighted sum of the number of available network connections for each streaming media segment and the inverse of the dependency level of that segment. The goal of the second transmission strategy is to minimize the target evaluation metric. The second transmission strategy also includes three constraints:

[0210] Constraint (1) is that the maximum transmission delay of any streaming media segment f is less than the maximum tolerable delay κ of that streaming media segment f. f ;

[0211] Constraint (2) is that the transmission rate of any available network connection j is less than the remaining bandwidth b of that available network connection j. j ;

[0212] Constraint (3) is that the overall failure rate of any streaming segment f is lower than the failure threshold ∈.

[0213] Based on the above embodiments, when the objective function of the second transmission strategy is associated with the first evaluation index of the streaming media segment, the expression of the objective function of the second transmission strategy in this application embodiment can be:

[0214]

[0215] As can be seen, the objective function of this application embodiment only considers minimizing the traffic increase of the first evaluation metric of a streaming media segment—the streaming media segment f transmitted by all available connections.

[0216] The expression for the second transmission strategy in this application embodiment includes:

[0217]

[0218]

[0219]

[0220]

[0221] The second transmission strategy in this application embodiment minimizes the number of available network connections for the streaming media segment f under three constraints:

[0222] Constraint (1) is that the maximum transmission delay of streaming media segment f is less than the maximum tolerable delay κ of streaming media segment f. f ;

[0223] Constraint (2) is that the transmission rate of any available network connection j is less than the remaining bandwidth b of that available network connection j. j ;

[0224] Constraint (3) is that the overall failure rate of the streaming segment f is lower than the failure threshold ∈.

[0225] Based on the above embodiments, the transmission strategy of this application embodiment is a third transmission strategy. The purpose of the third transmission strategy is to minimize the transmission failure rate under the limitation of the maximum number of copies of streaming media segments.

[0226] The first evaluation metric for the third transmission strategy is the overall failure rate of streaming media segments.

[0227] As can be seen from the above embodiments, when the objective function of the third transmission strategy is associated with the target evaluation index of the streaming media to be transmitted, the expression of the objective function of the third transmission strategy in this application embodiment can be:

[0228]

[0229] The second evaluation metric for the third transmission strategy includes the number of available network connections corresponding to the streaming media segment; the third evaluation metric includes the reception latency and transmission rate of the candidate network connections.

[0230] The expression associated with the third transmission strategy in this application embodiment includes:

[0231]

[0232]

[0233]

[0234]

[0235] Where N represents the maximum number of available network connections for each streaming media segment. The target evaluation metric of the third transmission strategy in this application is the result of a weighted sum of the overall failure rate of the streaming media segments, calculated using the inverse of the dependency level of each streaming media segment. The goal of the third transmission strategy is to minimize the target evaluation metric. The third transmission strategy also includes three constraints:

[0236] Constraint (1) is that the maximum transmission delay of any streaming media segment f is less than the maximum tolerable delay κ of that streaming media segment f. f ;

[0237] Constraint (2) is that the transmission rate of any available network connection j is less than the remaining bandwidth b of that available network connection j. j ;

[0238] Constraint (3) is that the overall failure rate of any streaming segment f is lower than the failure threshold ∈.

[0239] Based on the above embodiments, when the objective function of the third transmission strategy is associated with the first evaluation index of the streaming media segment, the expression of the objective function of the third transmission strategy in this application embodiment can be:

[0240]

[0241]

[0242]

[0243]

[0244] The third transmission strategy in this application embodiment minimizes the number of available network connections for the streaming media segment f under three constraints:

[0245] Constraint (1) is that the maximum transmission delay of streaming media segment f is less than the maximum tolerable delay κ of streaming media segment f. f ;

[0246] Constraint (2) is that the transmission rate of any available network connection j is less than the remaining bandwidth b of that available network connection j. j ;

[0247] Constraint (3) is that the overall failure rate of the streaming segment f is lower than the failure threshold ∈.

[0248] Based on the above embodiments, the transmission strategy of this application embodiment is a fourth transmission strategy. The goal of the fourth transmission strategy is to minimize the transmission failure rate under the constraint of traffic increment.

[0249] The first evaluation metric for the fourth transmission strategy is the overall failure rate of streaming media segments.

[0250] As can be seen from the above embodiments, when the objective function of the third transmission strategy is associated with the target evaluation index of the streaming media to be transmitted, the expression of the objective function of the third transmission strategy in this application embodiment can be:

[0251]

[0252] The second evaluation metric for the fourth transmission strategy includes: unit traffic of the streaming media segment, where unit traffic is the traffic of the streaming media segment when all available network connections transmit the streaming media segment. In this embodiment, the traffic of available connections transmitting the streaming media segment can be represented by the transmission speed of available connections transmitting the streaming media segment, so all available network connections J of the streaming media segment ff The bandwidth used to transmit streaming media segments can be expressed as:

[0253] The third evaluation metric for the fourth transmission strategy includes the receive latency and transmission rate of the candidate network connection.

[0254] The expression associated with the fourth transmission strategy in this application embodiment includes:

[0255]

[0256]

[0257]

[0258]

[0259] In the above constraint (3), parameter D represents the traffic threshold of the streaming media to be transmitted. The target evaluation index of the fourth transmission strategy of this application is the result of weighted summation of the overall failure rate of the streaming media segments by the inverse of the dependence degree of each streaming media segment. The goal of the fourth transmission strategy is to minimize the target evaluation index. The fourth transmission strategy also includes three constraints:

[0260] Constraint (1) is that the maximum transmission delay of any streaming media segment f is less than the maximum tolerable delay κ of that streaming media segment f. f ;

[0261] Constraint (2) is that the transmission rate of any available network connection j is less than the remaining bandwidth b of that available network connection j. j ;

[0262] Constraint (3) is that the sum of the unit traffic of all streaming media segments is less than the traffic threshold of the streaming media to be transmitted.

[0263] Based on the above embodiments, when the objective function of the fourth transmission strategy is associated with the first evaluation index of the streaming media segment, the expression of the objective function of the fourth transmission strategy in this application embodiment can be:

[0264]

[0265]

[0266]

[0267]

[0268] The fourth transmission strategy in this application embodiment minimizes the number of available network connections for the streaming media segment f under three constraints:

[0269] Constraint (1) is that the maximum transmission delay of streaming media segment f is less than the maximum tolerable delay κ of streaming media segment f. f ;

[0270] Constraint (2) is that the transmission rate of any available network connection j is less than the remaining bandwidth b of that available network connection j. j ;

[0271] Constraint (3) is that the unit flow of streaming media segment f is lower than the flow threshold D′ of streaming media segment.

[0272] This application provides a streaming media transmission device based on multiple connections, such as... Figure 5 As shown, the streaming media transmission device may include: a dependency determination module 101, a connection strategy preparation module 102, a target network connection determination module 103, and a transmission module 104, wherein...

[0273] The dependency degree determination module 101 is used to determine the dependency degree of each streaming media segment in the streaming media to be transmitted. The dependency degree is used to characterize the interval between the corresponding streaming media segment and the reference unit. The reference unit is the streaming media segment that is farthest from the corresponding streaming media segment among the streaming media segments that have a dependency relationship with the corresponding streaming media segment.

[0274] The connection strategy preparation module 102 is used to determine each candidate network connection and transmission strategy corresponding to the streaming media to be transmitted. The transmission strategy is associated with an objective function and constraints of the objective function. The objective function is associated with a first evaluation index of each streaming media segment. The constraints are used to determine at least one available network connection corresponding to the streaming media segment from each candidate network connection. The constraints are associated with at least one second evaluation index of the streaming media segment. The first evaluation index is determined based on the network status of the available network connection.

[0275] The target network connection determination module 103 is used to determine the target network connection of each streaming media segment when the objective function reaches the optimal solution under the constraints, based on the network status of each candidate network connection and the degree of dependence of each streaming media segment.

[0276] The transmission module 104 is used to transmit the streaming media segment via the target network connection of the streaming media segment for each streaming media segment.

[0277] The apparatus in this application embodiment can execute the method provided in this application embodiment, and the implementation principle is similar. The actions performed by each module in the apparatus of each embodiment of this application correspond to the steps in the method of each embodiment of this application. For detailed functional descriptions of each module of the apparatus, please refer to the descriptions in the corresponding methods shown above, which will not be repeated here.

[0278] As an alternative embodiment, the degree of dependence is used to weight the first evaluation metric for the corresponding streaming media segment;

[0279] The objective function is associated with the target evaluation index of the streaming media to be transmitted. The target evaluation index is the sum of the weighted first evaluation index of each streaming media segment.

[0280] The target network connection determination module is specifically used to: determine the target evaluation index associated with the objective function based on the network state of each candidate network connection, and determine the target network connection of each streaming media segment when the optimal solution is reached under constraints.

[0281] As an optional embodiment, the target network connection determination module is specifically used to: determine the target network connection of each streaming media segment from the candidate network connections in descending order of the degree of dependence of each streaming media segment;

[0282] The steps for the target network connection determination module to determine the target network connection for each streaming media segment include:

[0283] Determine the network status of each candidate network connection;

[0284] Based on the network state of each candidate network connection, the target network connection of the streaming media segment is determined when the first evaluation index of the streaming media segment associated with the objective function reaches the optimal solution under the constraints.

[0285] Update the network status of the target network connection.

[0286] As an optional embodiment, the dependency determination module is used to:

[0287] For each streaming media segment in the streaming media to be transmitted, if the streaming media to be transmitted includes streaming media segments that are dependent on the corresponding streaming media segment, then a reference unit is determined from the streaming media segments that are dependent on the reference unit, and the number of units between the corresponding streaming media segment and the reference unit is taken as the degree of dependence of the streaming media segment.

[0288] If the resource to be transmitted does not include any streaming media segments that are dependent on the corresponding streaming media segment, then the default value will be used as the degree of dependency of the corresponding streaming media segment.

[0289] As an optional embodiment, the constraints are also associated with at least one third evaluation metric for the candidate network connection, which is related to the network state of the corresponding candidate network connection.

[0290] As an optional embodiment, the target network connectivity determination module includes:

[0291] The available set determination unit is used to determine the third evaluation index of each candidate network connection based on the network state of each candidate network connection, and to take the candidate network connections that meet the constraints of the third evaluation index as available network connections, so as to obtain the available set of the first iteration.

[0292] The reference set unit is used to select the available set after removing available network connections for each available network connection in the available set of the current iteration.

[0293] The indicator determination unit is used to determine a first evaluation indicator for the transmission of streaming media segments based on the network status of the available set; and to determine a first evaluation indicator and a second evaluation indicator for the transmission of streaming media segments based on the network status of the reference set.

[0294] The judgment unit is used to determine the degree of change of the first evaluation index of the streaming media segment based on the available set and the reference set, and to determine whether the second evaluation index of the streaming media segment based on the reference set meets the constraints.

[0295] An iterative unit is used to remove the reference network connection that causes the most significant change from the available set of the current iteration if it is determined that there is at least one reference network connection and the second evaluation metric of the streaming media segment transmitted based on the reference set corresponding to the reference network connection meets the constraints, so as to obtain the available set of the next iteration and proceed to the next iteration.

[0296] The target determination unit is used to stop the iteration if it is determined that there is no reference network connection in the current iteration, and to use the available network connections in the available set of the current iteration as the target network connections of the streaming media segment.

[0297] As an optional embodiment, the first evaluation metric is the number of available network connections for the streaming media segment;

[0298] The second evaluation metric includes: the overall failure rate of the streaming media segment, which is used to characterize the probability that a streaming media segment will fail when transmitted based on the corresponding available network connection;

[0299] The third evaluation metric includes: the receive latency and transmission rate of the candidate network connection.

[0300] As an optional embodiment, the first evaluation metric is the increase in traffic for transmitting streaming media segments across all available network connections;

[0301] The second evaluation metric includes: the overall failure rate of streaming segments;

[0302] The third evaluation metric includes: the receive latency and transmission rate of the candidate network connection.

[0303] As an optional implementation, the first evaluation metric is the overall failure rate of the streaming media segment;

[0304] The second evaluation metric includes: the number of available network connections corresponding to the streaming segment;

[0305] The third evaluation metric includes: the receive latency and transmission rate of the candidate network connection.

[0306] As an optional implementation, the first evaluation metric is the overall failure rate of the streaming media segment;

[0307] The second evaluation metric includes: unit traffic of a streaming media segment, which is the traffic of the streaming media segment when all available network connections are used to transmit the streaming media segment;

[0308] The third evaluation metric includes: the receive latency and transmission rate of the candidate network connection.

[0309] This application provides an electronic device, including a memory, a processor, and a computer program stored in the memory. The processor executes the computer program to implement the steps of a multi-connection-based streaming media transmission method. Compared with related technologies, this method achieves the following: The multi-connection-based streaming media transmission method of this application determines the dependency degree of each streaming media segment in the streaming media to be transmitted. The dependency degree characterizes the interval between the corresponding streaming media segment and the reference unit. The dependency degree can reflect the range of the largest neighboring frames that the streaming media segment may affect when the transmission fails. In particular, when the streaming media segment is a single frame, the dependency degree of I-frames, B-frames, and P-frames will have obvious differences. Further, it determines each candidate network connection and transmission strategy corresponding to the streaming media to be transmitted. Based on the network status of each candidate network connection and the dependency degree of each streaming media segment, it determines the target network connection of each streaming media segment when the objective function reaches the optimal solution under the constraints. This enables flexible determination of the target network connection for each I-frame, P-frame, and B-frame within each GoP, and can effectively avoid the occurrence of first-frame delay, reduce stuttering, and improve user experience.

[0310] In one alternative embodiment, an electronic device is provided, such as Figure 6 As shown, Figure 6The illustrated electronic device 4000 includes a processor 4001 and a memory 4003. The processor 4001 and the memory 4003 are connected, for example, via a bus 4002. Optionally, the electronic device 4000 may further include a transceiver 4004, which can be used for data interaction between the electronic device and other electronic devices, such as sending and / or receiving data. It should be noted that in practical applications, the transceiver 4004 is not limited to one type, and the structure of the electronic device 4000 does not constitute a limitation on the embodiments of this application.

[0311] Processor 4001 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. Processor 4001 may also be a combination that implements computational functions, such as including one or more microprocessor combinations, a combination of a DSP and a microprocessor, etc.

[0312] Bus 4002 may include a pathway for transmitting information between the aforementioned components. Bus 4002 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. Bus 4002 can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 6 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0313] The memory 4003 may be ROM (Read Only Memory) or other types of static storage devices capable of storing static information and instructions, RAM (Random Access Memory) or other types of dynamic storage devices capable of storing information and instructions, or EEPROM (Electrically Erasable Programmable Read Only Memory), CD-ROM (Compact Disc Read Only Memory) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media, other magnetic storage devices, or any other medium capable of carrying or storing computer programs and capable of being read by a computer, without limitation herein.

[0314] The memory 4003 stores computer programs that execute embodiments of this application, and its execution is controlled by the processor 4001. The processor 4001 executes the computer programs stored in the memory 4003 to implement the steps shown in the foregoing method embodiments.

[0315] This application provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it can implement the steps and corresponding content of the aforementioned method embodiments.

[0316] This application also provides a computer program product, including a computer program that, when executed by a processor, can implement the steps and corresponding content of the aforementioned method embodiments.

[0317] The terms "first," "second," "third," "fourth," "1," "2," etc. (if present) in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in a sequence other than that shown in the figures or text.

[0318] It should be understood that although arrows indicate various operation steps in the flowcharts of this application's embodiments, the order in which these steps are implemented is not limited to the order indicated by the arrows. Unless explicitly stated herein, in some implementation scenarios of this application's embodiments, the implementation steps in each flowchart can be executed in other orders as required. Furthermore, some or all steps in each flowchart, based on the actual implementation scenario, may include multiple sub-steps or multiple stages. Some or all of these sub-steps or stages can be executed at the same time, and each sub-step or stage can also be executed at different times. In scenarios where execution times differ, the execution order of these sub-steps or stages can be flexibly configured according to requirements, and this application's embodiments do not limit this.

[0319] The above are only optional implementation methods for some implementation scenarios of this application. It should be noted that for those skilled in the art, other similar implementation methods based on the technical concept of this application, without departing from the technical concept of this application, also fall within the protection scope of the embodiments of this application.

Claims

1. A streaming media transmission method based on multiple connections, characterized in that, include: Determine the degree of dependency of each streaming media segment in the streaming media to be transmitted. The degree of dependency is used to characterize the number of unit intervals between the corresponding streaming media segment and the reference unit in the decoding order within the same picture group GoP. The reference unit is the streaming media segment that is the furthest away from the corresponding streaming media segment in the decoding order among the streaming media segments that have a dependency relationship with the corresponding streaming media segment. If the streaming media to be transmitted does not include streaming media segments that are dependent on the corresponding streaming media segments, then a preset non-zero value is used as the degree of dependence of the corresponding streaming media segments. The process involves determining candidate network connections and transmission strategies corresponding to the streaming media to be transmitted. Each transmission strategy is associated with an objective function and its constraints. The objective function is the sum of weighted first evaluation metrics for each streaming media segment, with the degree of dependence inversely proportional to the weights. The constraints are used to determine at least one available network connection for each streaming media segment from the candidate network connections. These constraints are associated with at least one second evaluation metric for the streaming media segment and a third evaluation metric for the candidate network connection. The first evaluation metric is determined based on the network status of the available network connection. The second evaluation metric is different from the first evaluation metric and is used to evaluate the transmission quality of the streaming media segment. The third evaluation metric is related to the network status of the corresponding candidate network connection. Based on the network state of each candidate network connection and the degree of dependence of each streaming media segment, the target network connection of each streaming media segment is determined when the objective function reaches the optimal solution under the constraints. For each streaming segment, the streaming segment is transmitted through the target network connection of the streaming segment.

2. The method according to claim 1, characterized in that, The degree of dependence is used to weight the first evaluation metric for the corresponding streaming media segment; The objective function is associated with the target evaluation index of the streaming media to be transmitted, and the target evaluation index is the sum of the weighted first evaluation index of each streaming media segment; The step of determining the target network connections for each streaming media segment when the objective function reaches its optimal solution under the constraints, based on the network state of each candidate network connection and the degree of dependency of each streaming media segment, includes: Based on the network state of each candidate network connection, the target evaluation index associated with the objective function is determined, and the target network connection of each streaming media segment is determined when the optimal solution is reached under the constraints.

3. The method according to claim 1, characterized in that, The step of determining the target network connections for each streaming media segment when the objective function reaches its optimal solution under the constraints, based on the network state of each candidate network connection and the degree of dependency of each streaming media segment, includes: Based on the order of dependence of each streaming segment from highest to lowest, the target network connection for each streaming segment is determined sequentially from the candidate network connections; The steps for determining the target network connection for each streaming segment include: Determine the network status of each candidate network connection; Based on the network state of each candidate network connection, the target network connection of the streaming media segment is determined when the first evaluation index of the streaming media segment associated with the objective function reaches the optimal solution under the constraints. Update the network status of the target network connection.

4. The method according to claim 1, characterized in that, Determining the degree of dependency of each streaming media segment in the streaming media to be transmitted includes: For each streaming media segment in the streaming media to be transmitted, if the streaming media to be transmitted includes streaming media segments that are dependent on the corresponding streaming media segment, then the reference unit is determined from the streaming media segments that are dependent on the reference unit, and the number of units between the corresponding streaming media segment and the reference unit is taken as the degree of dependence of the streaming media segment.

5. The method according to any one of claims 1-4, characterized in that, The constraints are also associated with at least one third evaluation metric for candidate network connections, which is related to the network state of the corresponding candidate network connection.

6. The method according to claim 5, characterized in that, When the first evaluation index of the streaming media segment associated with the objective function is determined based on the network state of each candidate network connection and reaches the optimal solution under the constraints, the target network connection of the streaming media segment includes: Based on the network state of each candidate network connection, a third evaluation metric is determined for each candidate network connection. Candidate network connections that meet the constraints of the third evaluation metric are selected as available network connections to obtain the available set for the first iteration. For each available network connection in the available set of the current iteration, the available set after removing the available network connection is used as the reference set; Based on the network status of the available set, a first evaluation metric is determined for the transmission of the streaming media segment based on the available set; based on the network status of the reference set, a first evaluation metric and a second evaluation metric are determined for the transmission of the streaming media segment based on the reference set. Determine the degree of change of a first evaluation metric for the transmission of the streaming media segment based on the available set and the reference set, and determine whether a second evaluation metric for the transmission of the streaming media segment based on the reference set meets the constraints. If it is determined that there is at least one reference network connection, and the second evaluation metric transmitted by the streaming media segment based on the reference set corresponding to the reference network connection meets the constraint condition, then the reference network connection that makes the change most significant is removed from the available set of the current iteration to obtain the available set of the next iteration, and the next iteration is performed. If it is determined that there is no reference network connection in the current iteration, the iteration is stopped, and the available network connections in the available set of the current iteration are used as the target network connection for the streaming media segment.

7. The method according to claim 5, characterized in that, The first evaluation metric is the number of available network connections for a streaming media segment; The second evaluation metric includes: the overall failure rate of a streaming media segment, which is used to characterize the probability that a streaming media segment will fail when transmitted based on the corresponding available network connection; The third evaluation metric includes: the receive latency and transmission rate of the candidate network connection.

8. The method according to claim 5, characterized in that, The first evaluation metric is the increase in traffic for transmitting streaming media segments across all available network connections; The second evaluation metric includes: the overall failure rate of streaming segments; The third evaluation metric includes: the receive latency and transmission rate of the candidate network connection.

9. The method according to claim 5, characterized in that, The first evaluation metric is the overall failure rate of streaming media segments; The second evaluation metric includes: the number of available network connections corresponding to a streaming segment; The third evaluation metric includes: the receive latency and transmission rate of the candidate network connection.

10. The method according to claim 5, characterized in that, The first evaluation metric is the overall failure rate of streaming media segments; The second evaluation metric includes: unit traffic of the streaming media segment, wherein the unit traffic is the traffic of the streaming media segment when all available network connections transmit the streaming media segment; The third evaluation metric includes: the receive latency and transmission rate of the candidate network connection.

11. A streaming media transmission device based on multiple connections, characterized in that, include: The dependency degree determination module is used to determine the dependency degree of each streaming media segment in the streaming media to be transmitted. The dependency degree is used to characterize the number of unit intervals between the corresponding streaming media segment and the reference unit in the decoding order within the same picture group GoP. The reference unit is the streaming media segment that is the farthest away from the corresponding streaming media segment in the decoding order among the streaming media segments that have a dependency relationship with the corresponding streaming media segment. If the streaming media to be transmitted does not include streaming media segments that are dependent on the corresponding streaming media segments, then a preset non-zero value is used as the degree of dependence of the corresponding streaming media segments. A connection strategy preparation module is used to determine each candidate network connection and transmission strategy corresponding to the streaming media to be transmitted. The transmission strategy is associated with an objective function and constraints on the objective function. The objective function is the sum of weighted first evaluation indicators for each streaming media segment, and the degree of dependence is inversely proportional to the weight. The constraints are used to determine at least one available network connection corresponding to the streaming media segment from each candidate network connection. The constraints are associated with at least one second evaluation indicator for the streaming media segment and a third evaluation indicator for the candidate network connection. The first evaluation indicator is determined based on the network status of the available network connection. The second evaluation indicator is different from the first evaluation indicator and is used to evaluate the transmission quality of the streaming media segment. The third evaluation indicator is related to the network status of the corresponding candidate network connection. The target network connection determination module is used to determine the target network connection of each streaming media segment when the objective function reaches the optimal solution under the constraints, based on the network status of each candidate network connection and the degree of dependence of each streaming media segment. A transmission module is used to transmit each streaming media segment via a target network connection of the streaming media segment.

12. An electronic device comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the steps of the multi-connection-based streaming media transmission method according to any one of claims 1-10.

13. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the multi-connection-based streaming media transmission method according to any one of claims 1-10.

14. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the multi-connection-based streaming media transmission method according to any one of claims 1-10.

Images