Video stream transmission method and device, equipment, storage medium and program product

Abstract

The invention relates to a video stream transmission method and device, equipment, a storage medium and a program product. The method comprises the following steps: establishing a plurality of video transmission connections according to different network transmission paths between a sending end and a receiving end; each video transmission connection comprises a plurality of video stream transmission tracks; determining at least one target video transmission connection for transmitting a video stream from the plurality of video transmission connections according to the network state parameter of each video transmission connection in the plurality of video transmission connections; the video stream has a video stream identifier, and the video stream identifier corresponds to the track identifier of one of the plurality of video stream transmission tracks; determining a target video stream transmission track represented by a track identifier corresponding to the video stream identifier in the at least one target video transmission connection; and transmitting the video stream to a receiving end through the target video stream transmission track. By adopting the method to schedule the video streams, the reliability of the transmission process of each video stream and the network bandwidth utilization rate can be improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a video stream transmission method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Art

[0002] With the development of computer technology, multiple video streams need to be transmitted simultaneously in many application scenarios. Due to factors such as network coverage and interference, a single network may not be sufficient to carry multiple video streams. Multiple network transmission channels can be used to transmit multiple video streams, thereby improving the bandwidth and reliability of video transmission.

[0003] However, in the related art, one video stream is usually bound to one network transmission channel, which cannot fully guarantee the reliability of the transmission process of each video stream, and the video stream cannot be split and transmitted on multiple paths, resulting in low network bandwidth utilization. Summary of the invention

[0004] Based on this, it is necessary to provide a video stream transmission method, apparatus, computer equipment, computer readable storage medium and computer program product to address the above technical problems, which can improve the reliability of the video stream transmission process and the network bandwidth utilization.

[0005] In a first aspect, the present application provides a video stream transmission method. The method comprises:

[0006] Establish multiple video transmission connections according to different network transmission paths between the sending end and the receiving end; the multiple video transmission connections are used to transmit multiple video streams generated locally, each of the video transmission connections includes multiple video stream transmission tracks, and each of the video stream transmission tracks is used to transmit one of the multiple video streams;

[0007] For any one of the multiple video streams, determining at least one target video transmission connection for transmitting the video stream from the multiple video transmission connections according to a network status parameter of each video transmission connection in the multiple video transmission connections; the video stream has a video stream identifier, and the video stream identifier corresponds to a track identifier of one of the multiple video stream transmission tracks;

[0008] Determine a target video stream transmission track represented by a track identifier corresponding to the video stream identifier in the at least one target video transmission connection;

[0009] The video stream is transmitted to the receiving end through the target video stream transmission track.

[0010] In a second aspect, the present application further provides a video stream transmission device. The device comprises:

[0011] A first establishing module is used to establish multiple video transmission connections according to different network transmission paths between a sending end and a receiving end; the multiple video transmission connections are used to transmit multiple video streams generated locally, each of the video transmission connections includes multiple video stream transmission tracks, and each of the video stream transmission tracks is used to transmit one of the multiple video streams;

[0012] A scheduling module, configured to determine, for any one of the multiple video streams, at least one target video transmission connection for transmitting the video stream from the multiple video transmission connections according to a network status parameter of each video transmission connection in the multiple video transmission connections; the video stream has a video stream identifier, and the video stream identifier corresponds to a track identifier of one of the multiple video stream transmission tracks; and determine a target video stream transmission track represented by a track identifier corresponding to the video stream identifier in the at least one target video transmission connection;

[0013] The sending module is used to transmit the video stream to the receiving end through the target video stream transmission track.

[0014] In one of the embodiments, the network status parameters include network delay and recommended sending rate;

[0015] The scheduling module is also used to determine the video transmission connection with the lowest network delay among the multiple video transmission connections; for any video stream among the multiple video streams, the preset encoding rate of the video stream is used as the initial residual encoding rate, and if the residual encoding rate of the video stream is less than or equal to the recommended sending rate of the video transmission connection with the lowest network delay, the video transmission connection with the lowest network delay is used as the target video transmission connection for transmitting the video stream.

[0016] In one of the embodiments, the scheduling module is also used to update the recommended sending rate of the target video transmission connection based on the recommended sending rate of the target video transmission connection and the remaining encoding rate of the video stream; the updated recommended sending rate is used to participate in the path scheduling of the next video stream in the multiple video streams.

[0017] In one of the embodiments, the scheduling module is also used to use the video transmission connection with the lowest network delay as a target video transmission connection for transmitting the video stream if the remaining coding rate of the video stream is greater than the recommended sending rate of the video transmission connection with the lowest network delay, and determine a splitting factor according to the preset coding rate and the recommended sending rate of the target video transmission connection, and split some video frames from the video stream according to the splitting factor; update the remaining coding rate of the video stream according to the preset coding rate, the remaining coding rate and the splitting factor, and continue to determine at least one target video transmission connection for transmitting the video stream from the other video transmission connections according to the updated remaining coding rate and the recommended sending rates of other video transmission connections among the multiple video transmission connections except the target video transmission connection.

[0018] In one of the embodiments, the scheduling module is further used to use the video stream transmission track represented by the track identifier corresponding to the video stream identifier in each determined target video transmission connection as the target video stream transmission track for transmitting the video stream.

[0019] In one of the embodiments, the sending module is also used to transmit the split partial video frames through the video stream transmission track represented by the track identifier corresponding to the video stream identifier in the target video transmission connection; and transmit video frames other than the partial video frames in the video stream through the video stream transmission track represented by the track identifier corresponding to the video stream identifier in the target video transmission connection that is further determined.

[0020] In one of the embodiments, the scheduling module is also used to obtain preset splitting parameters, calculate the splitting factor according to the splitting parameters, the preset encoding rate and the recommended sending rate of the target video transmission connection; split the video stream into multiple video frames according to the splitting factor, and each video frame serves as the split partial video frame.

[0021] In one of the embodiments, the scheduling module is further used to calculate the scheduled rate of the video stream based on the preset encoding rate and the splitting factor; calculate the difference between the remaining encoding rate and the scheduled rate; and use the calculated difference as the updated remaining encoding rate of the video stream.

[0022] In one of the embodiments, the sending module is also used to determine, in each scheduling period, for any one of the multiple video streams, from the multiple video transmission connections, at least one target video transmission connection for transmitting the video stream in the current scheduling period according to the network status parameters of each video transmission connection in the multiple video transmission connections; if the target video transmission connection determined in the current scheduling period is different from the target video transmission connection determined in the previous scheduling period, the first video frame in the video stream to be scheduled to the target video transmission connection determined in the current scheduling period is synchronously transmitted to the receiving end through the target video transmission connection determined in the current scheduling period and the target video transmission connection determined in the previous scheduling period.

[0023] In a third aspect, the present application further provides a computer device, wherein the computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the above-mentioned video stream transmission method are implemented.

[0024] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the above-mentioned video stream transmission method are implemented.

[0025] In a fifth aspect, the present application further provides a computer program product, wherein the computer program product comprises a computer program, and when the computer program is executed by a processor, the steps of the above-mentioned video stream transmission method are implemented.

[0026] The above-mentioned video stream transmission method, device, computer equipment, storage medium and computer program product establish multiple video transmission connections according to different network transmission paths between the sending end and the receiving end, divide multiple video stream transmission tracks in each video transmission connection, and assign a track identifier to each video stream transmission track, and assign a video stream identifier to each video stream to be transmitted, and determine at least one target video transmission connection for transmitting the video stream from the multiple video transmission connections according to the network status parameters of each video transmission connection in the multiple video transmission connections. That is, the video stream can be dynamically scheduled to multiple video transmission connections according to the network status parameters, and the reliability of the transmission process of each video stream can be improved by making full use of multi-path dynamic transmission, and the same video stream can be transmitted to the receiving end across multiple video transmission connections, which can improve the network bandwidth utilization.

[0027] In a sixth aspect, the present application also provides a video stream transmission method. The method comprises:

[0028] Establish multiple video transmission connections according to different network transmission paths between the sending end and the receiving end; the multiple video transmission connections are used to receive multiple video streams generated by the sending end, each of the video transmission connections includes multiple video stream transmission tracks, and each of the video stream transmission tracks is used to receive one of the multiple video streams;

[0029] Receiving multiple video streams transmitted on each of the video transmission connections; the video streams have video stream identifiers, and the video stream identifiers correspond to track identifiers of one of the multiple video stream transmission tracks;

[0030] When video streams with the same video stream identifier are received through multiple different video transmission connections, the video streams with the same video stream identifier are merged to obtain one of the multiple video streams.

[0031] In a seventh aspect, the present application further provides a video stream transmission device. The device comprises:

[0032] A second establishing module is used to establish multiple video transmission connections according to different network transmission paths between the sending end and the receiving end; the multiple video transmission connections are used to receive multiple video streams generated by the sending end, each of the video transmission connections includes multiple video stream transmission tracks, and each of the video stream transmission tracks is used to receive one of the multiple video streams;

[0033] A receiving module, configured to receive multiple video streams transmitted on each of the video transmission connections; the video streams have video stream identifiers, and the video stream identifiers correspond to track identifiers of one of the multiple video stream transmission tracks;

[0034] The merging module is used to merge the video streams with the same video stream identifier when receiving the video streams with the same video stream identifier through multiple different video transmission connections to obtain one of the multiple video streams.

[0035] In one of the embodiments, the merging module is also used to establish a corresponding video stream buffer for the video stream identifier when a video stream with the same video stream identifier is received through multiple different video transmission connections; store the received video stream with the same video stream identifier in the video stream buffer; sort the video streams in the video stream buffer according to the timestamps of the video frames and then merge them to obtain one of the multiple video streams.

[0036] In one of the embodiments, the merging module is also used to obtain the buffering duration of each of the multiple target video stream transmission tracks that receive the video stream with the same video stream identifier; determine the buffering duration threshold of the video stream buffer according to the buffering duration of each of the multiple target video stream transmission tracks; when the buffering duration of the video stream buffer is greater than or equal to the buffering duration threshold, execute the step of sorting the video streams in the video stream buffer according to the timestamps of the video frames and then merging them to obtain one of the multiple video streams, and clear the buffering duration of the video stream buffer.

[0037] In one of the embodiments, the merging module is also used to determine the different video transmission connections to which the multiple target video stream transmission tracks respectively belong; for each target video stream transmission track, calculate the sum of the buffering time of the target video stream transmission track and the network delay of the corresponding video transmission connection; determine the buffering time threshold of the video stream buffer based on the maximum and minimum values ​​of the sum of the cache time and the network delay calculated for each target video stream transmission track.

[0038] In an eighth aspect, the present application further provides a computer device, wherein the computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the above-mentioned video stream transmission method are implemented.

[0039] In a ninth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the above-mentioned video stream transmission method are implemented.

[0040] In a tenth aspect, the present application further provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, the steps of the above-mentioned video stream transmission method are implemented.

[0041] The above-mentioned video stream transmission method, device, computer equipment, storage medium and computer program product establish multiple video transmission connections according to different network transmission paths between the sending end and the receiving end, divide multiple video stream transmission tracks in each video transmission connection, and assign a track identifier to each video stream transmission track, and assign a video stream identifier to each video stream to be transmitted. The sending end dynamically schedules the video stream to multiple video transmission connections, fully utilizes multi-path dynamic transmission to improve the reliability of each video stream transmission process, and the same video stream can be transmitted to the receiving end across multiple video transmission connections, which can improve network bandwidth utilization; the receiving end merges the video streams received from multiple different video transmission connections according to the video stream identifier to obtain one of the multiple video streams, and can quickly identify the video streams belonging to the same video stream, thereby improving the video stream identification efficiency of the receiving end. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 A diagram showing an application environment of a video stream transmission method in an embodiment;

[0043] Figure 2 A diagram showing an application environment of a video stream transmission method in another embodiment;

[0044] Figure 3 A schematic diagram of a process of a video stream transmission method in one embodiment;

[0045] Figure 4 A schematic diagram of a video transmission connection between a transmitting end and a receiving end in one embodiment;

[0046] Figure 5 A schematic diagram of a video stream scheduling process in an embodiment;

[0047] Figure 6 A schematic diagram of video stream splitting and transmission in one embodiment;

[0048] Figure 7 is a flow chart of a video stream transmission method in another embodiment;

[0049] Figure 8 is a structural block diagram of a video stream transmission device in one embodiment;

[0050] Fig. 9 is a structural block diagram of a video stream transmission device in another embodiment;

[0051] Fig.10 is an internal structure diagram of a server in one embodiment;

[0052] Fig.11 FIG. 4 is a diagram showing the internal structure of a terminal in an embodiment. DETAILED DESCRIPTION

[0053] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0054] The video stream transmission method provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown. Among them, the sending end 102 establishes multiple video transmission connections according to different network transmission paths between the sending end 102 and the receiving end 104; the multiple video transmission connections are used to transmit multiple video streams generated locally, each video transmission connection includes multiple video stream transmission tracks, and each video stream transmission track is used to transmit one of the multiple video streams; for any video stream in the multiple video streams, the sending end 102 determines at least one target video transmission connection for transmitting the video stream from the multiple video transmission connections according to the network state parameters of each video transmission connection in the multiple video transmission connections; the video stream has a video stream identifier, and the video stream identifier corresponds to the track identifier of one of the multiple video stream transmission tracks; the sending end 102 determines the target video stream transmission track represented by the track identifier corresponding to the video stream identifier in at least one target video transmission connection; the sending end 102 transmits the video stream to the receiving end 104 through the target video stream transmission track.

[0055] like Figure 2 As shown, the video stream transmission method provided in the embodiment of the present application can also be applied to Figure 2 In the application environment shown. Among them, the terminal 202 can be used as a sending end, and the server 204 can be used as a receiving end. Among them, the server 204 can be an independent physical server, or it can be a server cluster or distributed system composed of multiple physical servers, or it can be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The terminal 202 can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, etc., but is not limited to this. The terminal and the server can be directly or indirectly connected via wired or wireless communication, and this application is not limited here.

[0056] In one embodiment, Figure 3 As shown, a video stream transmission method is provided, and the method is applied to Figure 1 The sending end in the example is used to illustrate, including the following steps:

[0057] Step 302, establish multiple video transmission connections according to different network transmission paths between the sending end and the receiving end; the multiple video transmission connections are used to transmit multiple video streams generated locally, each video transmission connection includes multiple video stream transmission tracks, and each video stream transmission track is used to transmit one of the multiple video streams.

[0058] The video transmission connection refers to a video channel for transmitting video streams on the network transmission path between the sending end and the receiving end. Each video transmission connection is established on a different network transmission path, and a video transmission connection can be established on one network transmission path. Specifically, the network transmission path is determined based on a local IP address of the sending end and the network communication port of the sending end, and a local IP address of the receiving end and the network communication port of the receiving end, and a video transmission connection is created on the network transmission path; wherein a local IP address and a network communication port can determine a network address. For example, Figure 4 is a schematic diagram of the video transmission connection between the transmitter and the receiver, such as Figure 4 As shown, the sending end has N network cards and the receiving end has one network card. Then, according to the channel between each network card of the sending end and the network card of the receiving end, N network transmission paths are established, and a video transmission connection is established on each network transmission path. For example, according to the channel between network card 1 of the sending end and the network card of the receiving end, network transmission path 1 is established, and a video transmission connection 1 is established on network transmission path 1. For example, according to the channel between network card N of the sending end and the network card of the receiving end, network transmission path N is established, and a video transmission connection N is established on network transmission path N.

[0059] The video stream transmission track is a logical division of data transmission on the video transmission connection to obtain a logically isolated video transmission data channel. Each video transmission connection includes multiple video stream transmission tracks. The number of video stream transmission tracks can be set according to actual needs. In this embodiment, the number of video stream transmission tracks is set to be the same as the number of multiple video streams. That is, if the sending end has a total of M video streams, each video transmission connection includes M video stream transmission tracks. For example, Figure 4 is a schematic diagram of the video transmission connection between the transmitter and the receiver, such as Figure 4As shown, M video stream transmission tracks are divided on each network transmission path, and a track identifier is assigned to each video stream transmission track. In this embodiment, T_nm is used to represent the track identifier of the m-th video stream transmission track in the n-th video transmission connection. Taking network transmission path 1 as an example, network transmission path 1 is divided into M video stream transmission tracks, and the track identifier of the first video stream transmission track is recorded as T_11, and the track identifier of the m-th video stream transmission track is recorded as T_1m. Taking network transmission path N as an example, network transmission path N is divided into M video stream transmission tracks, and the track identifier of the first video stream transmission track is recorded as T_n1, and the track identifier of the m-th video stream transmission track is recorded as T_nm.

[0060] Video streams can be video streams from multiple different video sources such as cameras, screen capture, video files, etc. Multi-channel video streams are multi-channel video streams generated by the same sending end. For example, the sending end includes multiple cameras, and the multiple video streams captured by the multiple cameras are transmitted to the receiving end through multiple video transmission connections. The sending end generates the original data of the video stream, and packets it according to the RTP protocol, adds the video stream identifier during packetization, and then packets it according to the UDP protocol and encapsulates it, using the same local IP address and network communication port number, encapsulating it into a UDP message, and then sending the video stream out. The embodiment of the present application adopts the protocol stack of real-time video transmission itself, which has its own anti-packet loss capabilities, and no additional aggregation transmission overhead.

[0061] In some embodiments, the multi-channel video streams can be multi-channel video streams collected in a remote real-time control scenario. For example, the sending end is a terminal and the receiving end is a video server. The terminal transmits the video streams collected by multiple cameras to the video server through multiple video transmission connections. The video server displays the multi-channel video streams on a monitoring platform. On-site staff issue control instructions to the terminal based on the video images displayed on the monitoring platform. The terminal executes actions according to the control instructions, thereby realizing the remote real-time control function.

[0062] In some embodiments, the multi-channel video streams may be multi-channel video streams collected in a real-time vehicle monitoring scenario. For example, the sending end is a vehicle controller, the receiving end is a server, and multiple cameras are set at different positions of the vehicle. The vehicle controller obtains the video streams collected by the multiple cameras and transmits the multi-channel video streams to the server through multiple video transmission connections. The server displays the multi-channel video streams on the vehicle monitoring platform, thereby realizing real-time monitoring of the surrounding environment of the scene where the vehicle is located, which can be applied to a vehicle navigation system or a vehicle automatic driving system.

[0063] Specifically, the sender establishes multiple video transmission connections based on the sender's multiple local IP addresses and the sender's network communication ports, as well as the receiver's multiple local IP addresses and the receiver's network communication ports, and logically divides the data transmission on each video transmission connection to obtain multiple logically isolated video stream transmission tracks.

[0064] Step 304: for any one of the multiple video streams, determine at least one target video transmission connection for transmitting the video stream from the multiple video transmission connections based on the network status parameters of each video transmission connection in the multiple video transmission connections; the video stream has a video stream identifier, and the video stream identifier corresponds to a track identifier of one of the multiple video stream transmission tracks.

[0065] Among them, the network status parameters are used to measure the network performance of the video transmission connection. For example, the network status parameters may include parameters such as bandwidth, network delay, jitter, packet loss, and recommended sending rate. The network status parameters are updated regularly or irregularly. In a scheduling cycle (approximately 100ms), M video streams need to be scheduled, and scheduling is required multiple times, that is, scheduling is performed for each video stream. Each scheduling duration is about 1-2ms. The number of schedulings in a scheduling cycle may be M times, or may be greater than M times. In the scheduling process of multiple video streams, the same network status parameters are used, and the network status parameters obtained in the next scheduling cycle may change. For example, including M video streams, network status parameter 1 is obtained at time t1. For any of the M video streams, the M video streams are scheduled according to the network status parameter 1 at time t1. At time t1+T, the updated network status parameter 2 is obtained, and the M video streams are scheduled according to the network status parameter 2 at time t1+T. Among them, T represents the scheduling cycle, which is approximately 100ms.

[0066] Each video stream includes a video stream identifier for distinguishing different video streams. In order to facilitate the receiving end to distinguish the received video streams, this embodiment sets each video stream transmission track to include a track identifier. The track identifier is used to identify different video stream transmission tracks in the same video transmission connection, and the video stream identifier corresponds to the track identifier of one of the multiple video stream transmission tracks. When scheduling the video stream, the mth video stream is scheduled on a video stream transmission track in at least one video transmission connection whose track identifier corresponds to the video stream identifier of the mth video stream. This embodiment uses T_nm to represent the track identifier of the mth video stream transmission track in the nth video transmission connection. For example, Figure 4As shown, this embodiment includes M video streams and N video transmission connections, each video transmission connection includes M video stream transmission tracks, the mth video stream is scheduled to the mth video stream transmission track in the nth video transmission connection, that is, the mth video stream occupies the video stream transmission track with the track identification T_nm, and the mth video stream is scheduled to the mth video stream transmission track in the kth video transmission connection, that is, the mth video stream occupies the video stream transmission track with the track identification T_km.

[0067] Among them, the video stream identifier corresponds to the track identifier, indicating that there is a corresponding relationship between the video stream corresponding to the video stream identifier and the video stream transmission track corresponding to the track identifier. For example, the video stream identifier corresponds to the track identifier, and the video stream identifier and the track identifier may be the same.

[0068] Specifically, for any video stream among multiple video streams, the sender obtains the network status parameters of each video transmission connection among multiple video transmission connections, and schedules the any video stream to at least one target video transmission connection among the multiple video transmission connections according to the network status parameters, at least one of which includes one or more than one. Exemplarily, if among multiple video transmission connections, the video transmission connection with the best current network transmission performance indicated by the network status parameters can meet the network transmission requirements of any video stream transmission, then the video transmission connection with the best current network transmission performance is used as the target video transmission connection for transmitting the video stream; if the video transmission connection with the best current network transmission performance cannot meet the network transmission requirements of any video stream, then some video frames are split from the video stream, and the video transmission connection with the best current network transmission performance is used as the first target video transmission connection for transmitting the video stream, and some video frames are scheduled to the first target video transmission connection, and according to the network status parameters of each video transmission connection in the multiple video transmission connections, the next target video transmission connection is continued to be determined from other video transmission connections in the multiple video transmission connections except the first target video transmission connection, and video frames in the video stream except some video frames are scheduled to the next target video transmission connection, and multiple target video transmission connections are used to jointly schedule the same video stream.

[0069] Step 306: Determine a target video stream transmission track represented by a track identifier corresponding to the video stream identifier in at least one target video transmission connection.

[0070] If the number of the determined target video transmission connections is one, the sender uses the video stream transmission track represented by the track identifier corresponding to the video stream identifier in the determined target video transmission connection as the target video stream transmission track for transmitting the video stream to be transmitted. If the number of the determined target video transmission connections is multiple, the video stream transmission track represented by the track identifier corresponding to the video stream identifier in each determined target video transmission connection is used as the target video stream transmission track for transmitting the video stream to be transmitted.

[0071] For example, the video stream identifier of the video stream is 1, and if the target video transmission connection corresponding to video stream 1 is only video transmission connection 1, the video stream transmission track represented by the track identifier T_11 corresponding to video stream identifier 1 in video transmission connection 1 is used as the target video stream transmission track. If the target video transmission connection corresponding to video stream 1 includes video transmission connection 1 and video transmission connection 2, the video stream transmission track represented by the track identifier T_11 corresponding to video stream identifier 1 in video transmission connection 1 and the video stream transmission track represented by the track identifier T_21 corresponding to video stream identifier 1 in video transmission connection 2 are used as the target video stream transmission tracks, that is, video stream transmission track T_11 and video stream transmission track T_21 are used as the target video stream transmission tracks for transmitting video stream 1.

[0072] Specifically, the transmitter uses, according to the video stream identifier of the video stream, the video stream transmission track represented by the track identifier corresponding to the video stream identifier in each determined target video transmission connection as the target video stream transmission track for transmitting the video stream to be transmitted.

[0073] Step 308: Transmit the video stream to the receiving end through the target video stream transmission track.

[0074] If there is one target video stream transmission track, the sender will schedule each video frame to the target video stream transmission track in turn according to the timestamp of the video frame in the video stream. If there are multiple target video stream transmission tracks, the video stream is split into multiple partial video frames, and the multiple partial video frames are respectively scheduled to multiple target video stream transmission tracks; for any partial video frame in the multiple partial video frames, each video frame in the partial video frame is scheduled to the target video stream transmission track in turn according to the timestamp of the video frame in the partial video frame.

[0075] For example, a video stream includes video frames 1 to 100. If there is one target video stream transmission track, the video frames are scheduled to the target video stream transmission track in the order of video frames 1 to 100. If there are two target video stream transmission tracks, the video stream is split into two partial video frames. According to the timestamps of the video frames, each video frame in the first partial video frame is scheduled to the first target video stream transmission track in sequence, and according to the timestamps of the video frames, each video frame in the second partial video frame is scheduled to the second target video stream transmission track in sequence.

[0076] Specifically, if there is one target video stream transmission track, the sender schedules each video frame to the target video stream transmission track in turn according to the timestamp of the video frame in the video stream. If there are multiple target video stream transmission tracks, the sender splits the video stream into multiple partial video frames and schedules the multiple partial video frames to multiple target video stream transmission tracks respectively.

[0077] In the above-mentioned video stream transmission method, multiple video transmission connections are established according to different network transmission paths between the sending end and the receiving end, multiple video stream transmission tracks are divided in each video transmission connection, and a track identifier is assigned to each video stream transmission track, and a video stream identifier is assigned to each video stream to be transmitted, and according to the network status parameters of each video transmission connection in the multiple video transmission connections, at least one target video transmission connection for transmitting the video stream is determined from the multiple video transmission connections. That is, the video stream can be dynamically scheduled to the multiple video transmission connections according to the network status parameters, and the reliability of the transmission process of each video stream can be improved by making full use of multi-path dynamic transmission, and the same video stream can be transmitted to the receiving end across multiple video transmission connections, which can improve the network bandwidth utilization.

[0078] In one embodiment, the network status parameter includes a network delay and a recommended sending rate; wherein, for any one of the multiple video streams, determining at least one target video transmission connection for transmitting the video stream from the multiple video transmission connections according to the network status parameter of each video transmission connection in the multiple video transmission connections comprises the following steps:

[0079] First, determine the video transmission connection with the lowest network delay among multiple video transmission connections.

[0080] Among them, network delay is an indicator used to evaluate the network performance of video transmission connections. Network delay refers to the time required for a message to travel from the sender to the receiver. The protocol stack of the video transmission connection includes a network perception module and a congestion control module; the network perception module is responsible for collecting network status parameters such as network delay and packet loss rate of the video transmission connection; the congestion control module is responsible for calculating the recommended sending rate of the video transmission connection.

[0081] To ensure the transmission efficiency of the video stream, this embodiment uses the video transmission connection with the smallest network delay as the video transmission connection with the best current network transmission performance, sorts multiple video transmission connections in order of network delay from small to large, and starts with the video transmission connection with the lowest network delay to operate in sequence to determine at least one target video transmission connection for transmitting the video stream.

[0082] Specifically, the sending end collects the network delays in multiple video transmission connections through the network perception module on the protocol stack of the video transmission connection, sorts the multiple video transmission connections in order of network delay from small to large, and determines the video transmission connection with the lowest network delay among the multiple video transmission connections.

[0083] Second, for any one of the multiple video streams, the preset encoding rate of the video stream is used as the initial residual encoding rate. If the residual encoding rate of the video stream is less than or equal to the recommended sending rate of the video transmission connection with the lowest network delay, the video transmission connection with the lowest network delay is used as a target video transmission connection for transmitting the video stream.

[0084] The preset encoding rate refers to the bitrate speed that ensures the video stream can be restored. The remaining encoding rate refers to the bitrate speed that ensures the remaining video frames can be restored after the video stream is split into some video frames and dispatched to the target video transmission connection. The recommended sending rate refers to the amount of data transmitted by the video transmission connection per unit time.

[0085] When the video stream starts to be scheduled, the preset encoding rate of the video stream is used as the initial remaining encoding rate. If the remaining encoding rate of the video stream is less than or equal to the recommended sending rate of the video transmission connection with the lowest network delay, it means that the video transmission connection with the lowest network delay meets the network parameters required for video stream transmission.

[0086] For example, Figure 5 The scheduling process diagram of video stream i is shown in Fig. Figure 5 , the scheduling process of the i-th video stream executed by the transmitter includes the following steps: Step 501, the transmitter determines the video transmission connection j with the lowest network delay among multiple video transmission connections; Step 502, the transmitter uses the preset coding rate Ri of the video stream i as the initial remaining coding rate Rsi; Step 503, obtains the recommended sending rate Pj of the video transmission connection j with the lowest network delay; If Rsi≤Pj, then execute step 504, and use the video transmission connection j as the target video transmission connection for transmitting the video stream. If Rsi>Pj, then execute step 509, and use the video transmission connection j as the first target video transmission connection for transmitting the video stream i.

[0087] In this embodiment, the video transmission connection with the smallest network delay is used as the video transmission connection with the best current network transmission performance, and the network delay is ordered from the video transmission connection with the lowest network delay, and the operations are performed in sequence, to determine at least one target video transmission connection for transmitting the video stream. The video transmission connection with the smallest network delay is used as the target video transmission connection, which can ensure the best video delay effect and improve the transmission efficiency of the video stream. At the same time, when the remaining coding rate of the video stream is less than or equal to the recommended sending rate of the video transmission connection with the lowest network delay, the video transmission connection with the lowest network delay is used as the target video transmission connection for transmitting the video stream; the above determination of whether the video transmission connection with the lowest network delay can be used as the target video transmission connection based on the remaining coding rate of the video stream and the recommended sending rate of the video transmission connection can improve the accuracy of video stream scheduling.

[0088] In some embodiments, after using the video transmission connection with the lowest network delay as the target video transmission connection for transmitting the video stream, the method further includes:

[0089] The recommended sending rate of the target video transmission connection is updated according to the recommended sending rate of the target video transmission connection and the remaining encoding rate of the video stream; the updated recommended sending rate is used to participate in the path scheduling of the next video stream in the multiple video streams.

[0090] The difference between the recommended sending rate of the target video transmission connection and the remaining encoding rate of the video stream is used as the updated recommended sending rate of the video transmission connection, that is, the updated recommended sending rate can be expressed as: Pj = Pj-Rsi. The updated recommended sending rate is used to participate in the path scheduling of the next video stream in the multi-channel video stream.

[0091] Specifically, for any video stream among multiple video streams, the sending end loads the video stream to be transmitted, calculates the preset encoding rate of the video stream, and uses the preset encoding rate of any video stream as the initial residual encoding rate; if the residual encoding rate of the video stream is less than or equal to the recommended sending rate of the video transmission connection with the lowest network delay, the sending end uses the video transmission connection with the lowest network delay as the target video transmission connection for transmitting the video stream, and uses the difference between the recommended sending rate of the target video transmission connection and the residual encoding rate of the video stream as the updated recommended sending rate of the video transmission connection.

[0092] For example, refer to Figure 5After the sending end executes step 504, it also executes the following steps: step 505, schedules video stream i to video transmission connection j; step 506, updates the recommended sending rate of video transmission connection j, Pj=Pj-Rsi; if video stream i is the last video stream, execute step 507 to end video stream scheduling; if video stream i is not the last video stream, execute step 508, load the next video stream, i=i+1, and return to execute step 501.

[0093] In this embodiment, after the video transmission connection with the lowest network delay is used as the target video transmission connection for transmitting the video stream, the recommended sending rate of the target video transmission connection is updated. The updated recommended sending rate is used to participate in the path scheduling of the next video stream in the multiple video streams, providing the latest reference data for the path scheduling of the next video stream, which can improve the accuracy of the path scheduling of the next video stream.

[0094] In one embodiment, the method further comprises:

[0095] 1. If the remaining encoding rate of the video stream is greater than the recommended sending rate of the video transmission connection with the lowest network delay, the video transmission connection with the lowest network delay is used as a target video transmission connection for transmitting the video stream.

[0096] Among them, in order to ensure the transmission efficiency of the video stream, this embodiment sorts multiple video transmission connections in order from small to large network delay, and operates them in sequence starting from the video transmission connection with the lowest network delay. If the remaining encoding rate of the video stream is greater than the recommended sending rate of the video transmission connection with the lowest network delay, it means that the video transmission connection with the lowest network delay does not meet the network parameters required for video stream transmission. At this time, the video transmission connection with the lowest network delay is used as the first target video transmission connection for transmitting the video stream, and the next video transmission connection is operated, that is, it is determined whether the video transmission connection with the second smallest network delay meets the network parameters required for video stream transmission.

[0097] 2. Determine a split factor according to a preset encoding rate and a recommended sending rate of a target video transmission connection, and split some video frames from the video stream according to the split factor.

[0098] Among them, when the remaining encoding rate of the video stream is greater than the recommended sending rate of the video transmission connection with the lowest network delay, the video transmission connection with the lowest network delay is used as the first target video transmission connection for transmitting the video stream, and some video frames are split from the video stream according to the splitting factor, and some video frames are scheduled to the first target video transmission connection, and the remaining video streams are scheduled to other target video transmission connections, so that the same video stream can be transmitted to the receiving end across multiple video transmission connections, which can improve network bandwidth utilization.

[0099] The split factor represents the ratio of the number of video frames extracted from the video stream. In the process of splitting the video stream into multiple partial video frames, according to the split factor determined by the preset encoding rate and the recommended sending rate of the target video transmission connection, the video frames of the ratio represented by the split factor are extracted from the video stream as partial video frames. The method for extracting the partial video frames may be to extract a preset number of video frames from the video stream according to the equal interval extraction method. For example, the video stream includes video frames 1-100, and the split factor is 0.2, then one frame is extracted every 5 frames, and 20 video frames are extracted from the video stream as partial video frames.

[0100] In some embodiments, determining a split factor according to a preset encoding rate and a recommended sending rate of a target video transmission connection, and splitting a portion of video frames from a video stream according to the split factor comprises the following steps:

[0101] Obtain a preset splitting parameter, and calculate a splitting factor according to the splitting parameter, a preset encoding rate, and a recommended sending rate of a target video transmission connection; split the video stream into multiple video frames according to the splitting factor, and each video frame is used as a split partial video frame.

[0102] Among them, the preset splitting parameter is the ratio of the rate increase of the encoder encoding the split stream compared to the encoding of the unsplit stream. For example, it can be a video stream splitting rate increase factor. The video stream splitting rate increase factor is recorded as alpha, and the specific value of alpha can be an empirical value.

[0103] The splitting factor is determined according to a mapping function including variables such as the splitting parameter, the preset coding rate, and the recommended sending rate of the target video transmission connection, and the splitting factor is recorded as ratio. For example, the mapping function may be ratio=Pj / (Ri*(1+alpha)), where Pj is the recommended sending rate of the target video transmission connection, Ri is the preset coding rate, and alpha is the splitting parameter.

[0104] In the process of splitting the video stream, the number of video frames to be extracted is determined according to the splitting factor, and video frames are extracted from the video stream according to the number of video frames to be extracted, wherein sampling is performed from unsampled video frames in the video stream.

[0105] Specifically, the sender calculates the splitting factor according to a mapping function between the splitting parameter, the preset encoding rate, and the recommended sending rate of the target video transmission connection, wherein the mapping function may be ratio=Pj / (Ri*(1+alpha)), wherein ratio is the splitting factor, Pj is the recommended sending rate of the target video transmission connection, Ri is the preset encoding rate, and alpha is the splitting parameter. The sender determines the number of video frames to be extracted according to the splitting factor, and extracts video frames from the video stream according to the number of video frames to be extracted as the split partial video frames.

[0106] In this embodiment, a preset number of video frames are extracted from the video stream according to an equal-interval extraction method as partial video frames. The equal-interval extraction method can extract video frames evenly at intervals, so that the traffic on each video transmission connection is evenly distributed.

[0107] 3. According to the preset coding rate, the remaining coding rate and the splitting factor, the remaining coding rate of the video stream is updated; and according to the updated remaining coding rate and the recommended sending rates of other video transmission connections except the target video transmission connection in the multiple video transmission connections, at least one target video transmission connection for transmitting the video stream is continued to be determined from the other video transmission connections.

[0108] Among them, since the preset encoding rate of the video stream is used as the initial residual encoding rate, and a partial video frame after the video stream is split is scheduled to the first target video transmission connection, the residual encoding rate of the video stream will change. Before continuing to determine at least one target video transmission connection for transmitting the video stream from other video transmission connections, the residual encoding rate of the video stream needs to be updated.

[0109] In some embodiments, updating the remaining coding rate of the video stream according to the preset coding rate, the remaining coding rate and the splitting factor comprises the following steps:

[0110] According to the preset coding rate and the splitting factor, the scheduled rate of the video stream is calculated; the difference between the remaining coding rate and the scheduled rate is calculated; and the calculated difference is used as the updated remaining coding rate of the video stream.

[0111] The scheduled rate refers to the bitstream speed corresponding to the portion of video frames scheduled to the target video transmission connection. In this embodiment, the scheduled rate is equal to the product of the preset coding rate and the split factor. The preset coding rate of this embodiment is denoted as Ri, the split factor is denoted as ratio, the scheduled rate is denoted as Ri*ratio, and the remaining coding rate is denoted as Rsi. The updated remaining coding rate can be expressed as Rsi=Rsi-Ri*ratio.

[0112] In some embodiments, according to the updated remaining coding rate and the recommended sending rates of other video transmission connections except the target video transmission connection in the plurality of video transmission connections, continuing to determine at least one target video transmission connection for transmitting the video stream from the other video transmission connections specifically includes the following steps:

[0113] Using the next video transmission connection of the target video transmission connection as the current video transmission connection;

[0114] If the updated remaining encoding rate of the video stream is less than or equal to the recommended sending rate of the current video transmission connection, the current video transmission connection is used as the next target video transmission connection for transmitting the video stream.

[0115] If the updated residual coding rate of the video stream is greater than the recommended sending rate of the current video transmission connection, the current video transmission connection is used as the next target video transmission connection for transmitting the video stream, and the splitting factor is determined according to the preset coding rate and the recommended sending rate of the target video transmission connection, and some video frames are split from the video stream according to the splitting factor; the residual coding rate of the video stream is updated according to the preset coding rate, the residual coding rate and the splitting factor; if the residual coding rate of the updated video stream is less than or equal to zero, the next video stream is loaded, and the step of determining the video transmission connection with the lowest network delay among multiple video transmission connections is returned and continued; if the residual coding rate of the updated video stream is greater than zero, the step of taking the next video transmission connection of the target video transmission connection as the current video transmission connection is returned and continued until the residual coding rate of the updated video stream is less than or equal to zero.

[0116] For example, Figure 5 As shown, if Rsii>Pj, execute step 509, and use video transmission connection j as the first target video transmission connection for transmitting video stream i; step 510, determine the splitting factor ratio, ratio=Pj / (Ri*(1+alpha)); step 511, split some video frames from video stream i according to the splitting factor; step 512, schedule some video frames to video transmission connection j; step 513, update the remaining coding rate Rsi of video stream i, Rsi=Rsi-Ri*ratio; if the remaining coding rate Rsi of video stream i is zero, execute step 508, load the next video stream, i=i+1; if the remaining coding rate Rsi of video stream i is not equal to zero, execute step 514, load the next video transmission connection, j=j+1, and return to execute step 503 until the video stream scheduling is completed.

[0117] Among them, this embodiment sorts multiple video transmission connections in order from small to large network delay, and operates in sequence starting from the video transmission connection with the lowest network delay, that is, the next video transmission connection of the first target video transmission connection is the video transmission connection with the second-to-last smallest network delay.

[0118] Among them, since the recommended sending rates of different video transmission connections may be the same or different, for multiple target video transmission connections, the splitting factors determined according to the preset encoding rate and the recommended sending rate of the target video transmission connection may be the same or different, and the number of video frames of the partial video frames extracted during each scheduling may be the same or different, that is, the video stream is split into multiple video frames according to the splitting factor, and the number of video frames in each video frame may be the same or different.

[0119] In this embodiment, when the remaining encoding rate of the video stream is greater than the recommended sending rate of the video transmission connection with the lowest network delay, the video transmission connection with the lowest network delay is used as the first target video transmission connection for transmitting the video stream. At the same time, the splitting factor is determined according to the preset encoding rate and the recommended sending rate of the target video transmission connection, and some video frames are split from the video stream according to the splitting factor. The some video frames are scheduled to the first target video transmission connection, and the remaining video stream is scheduled to at least one other target video transmission connection, thereby enabling the same video stream to be transmitted to the receiving end across multiple video transmission connections, thereby improving network bandwidth utilization.

[0120] In some embodiments, if the current video transmission connection is the last video transmission connection and the remaining coding rate of the updated video stream is greater than zero, the preset coding rate of the video stream is updated based on the video stream rate reduction factor, the preset coding rate of the updated video stream is used as the initial remaining coding rate, and the step of determining the video transmission connection with the lowest network delay among multiple video transmission connections is returned to execute, and the execution is continued until the remaining coding rate of the updated video stream is equal to zero.

[0121] Among them, if the current video transmission connection is the last video transmission connection, and the remaining encoding rate of the updated video stream is greater than zero, it indicates that the scheduling process cannot schedule all video streams to the video transmission connection within the current video stream scheduling period. At this time, it is necessary to adjust the parameters required for scheduling, such as increasing the network status parameters, or reducing the preset encoding rate of the video stream. Compared with increasing the network status parameters, the method of reducing the preset encoding rate of the video stream is less expensive and more effective. This embodiment adopts the method of reducing the preset encoding rate of the video stream to adjust the scheduling process of the video stream.

[0122] Among them, the amount of decrease in the preset coding rate of the video stream can be determined according to the video stream rate decrease factor. The video stream rate decrease factor refers to the compromise decrease gradient of the preset coding rate. For example, if the preset coding rate cannot meet the scheduling, the preset coding rate can be reduced and tried again. The video stream rate reduction factor can be preset, such as 0.8, which is a ratio of 20% each time. The video stream rate reduction factor is denoted as beta, and the updated preset coding rate is identified as Ri*beta. After updating the preset coding rate of the video stream, the video stream scheduling algorithm is re-executed again, that is, the relationship between the initial remaining coding rate of the video stream and the recommended sending rate of the video transmission connection with the lowest network delay is re-compared, and the corresponding scheduling strategy is executed according to the comparison result.

[0123] In this embodiment, when the current video transmission connection is the last video transmission connection and the remaining encoding rate of the updated video stream is greater than zero, the scheduling process of the video stream is adjusted by reducing the preset encoding rate of the video stream, so that all video streams can be scheduled to the video transmission connection based on the preset encoding rate of the updated video stream, thereby ensuring the accuracy of video stream scheduling while reducing costs.

[0124] In one of the embodiments, determining a target video stream transmission track represented by a track identifier corresponding to a video stream identifier in at least one target video transmission connection comprises the following steps:

[0125] The video stream transmission track represented by the track identifier corresponding to the video stream identifier in each determined target video transmission connection is used as the target video stream transmission track for transmitting the video stream to be transmitted.

[0126] If there is only one target video transmission connection, the video stream transmission track represented by the track identifier corresponding to the video stream identifier in the target video transmission connection is used as the target video stream transmission track for transmitting the video stream to be transmitted. For example, the video stream identifier of the mth video stream is m, and the track identifier corresponding to the video stream identifier m is the mth video stream transmission track, then the mth video stream transmission track in the target video transmission connection is used as the target video stream transmission track for transmitting the video stream to be transmitted.

[0127] If there are multiple target video transmission connections, the video stream transmission track represented by the track identifier corresponding to the video stream identifier in each target video transmission connection is used as the target video stream transmission track for transmitting the video stream to be transmitted. Figure 6 The schematic diagram of video stream splitting transmission is as follows: Figure 6As shown, the video stream identifier of the mth video stream is m, the track identifier corresponding to the video stream identifier m is the mth video stream transmission track, and the target video transmission connection corresponding to the mth video stream includes video transmission connection 1 and video transmission connection 2. The video stream transmission track represented by the track identifier T_1m corresponding to the video stream identifier m in the video transmission connection 1 and the video stream transmission track represented by the track identifier T_2m corresponding to the video stream identifier m in the video transmission connection 2 are used as the target video stream transmission tracks for transmitting the video stream to be transmitted.

[0128] In this embodiment, by using the video stream transmission track represented by the track identifier corresponding to the video stream identifier in the target video transmission connection as the target video stream transmission track for transmitting the video stream to be transmitted, the receiving end can easily distinguish the received video stream.

[0129] In one embodiment, when there are multiple target video transmission connections, there are also multiple target video stream transmission tracks for transmitting the video stream to be transmitted. In this embodiment, transmitting the video stream to the receiving end through the target video stream transmission track includes:

[0130] The split part of the video frames is transmitted through the video stream transmission track represented by the track identifier corresponding to the video stream identifier in the target video transmission connection; the video frames other than the part of the video frames in the video stream are transmitted through the video stream transmission track represented by the track identifier corresponding to the video stream identifier in the target video transmission connection that is further determined.

[0131] Wherein, in the case of multiple target video transmission connections, some video frames are split from the video stream according to the splitting factor, and the some video frames are scheduled to the video stream transmission track represented by the track identifier corresponding to the video stream identifier in the first target video transmission connection; in the next round of determining the target video transmission connection, some video frames are split from the video stream again according to the recalculated splitting factor from the video stream removed from the video stream, and the re-extracted some video frames are scheduled to the video stream transmission track represented by the track identifier corresponding to the video stream identifier in the next target video transmission connection. Repeat the process of multiple rounds of splitting and scheduling until all video frames of the video stream are scheduled to multiple target video transmission connections.

[0132] For example, the target video transmission connection corresponding to the mth video stream includes video transmission connection 1 and video transmission connection 2. Partial video frame 1 is extracted from the mth video stream according to the splitting factor, and the partial video frame 1 is scheduled to video transmission connection 1, and the partial video frame 1 is transmitted through the video stream transmission track represented by the track identifier T_1m in video transmission connection 1; after extracting partial video frame 1, partial video frame 2 is extracted from the remaining video frames in the mth video stream again according to the recalculated splitting factor, and the partial video frame 2 is scheduled to video transmission connection 2, and the partial video frame 2 is transmitted through the video stream transmission track represented by the track identifier T_2m in video transmission connection 2.

[0133] In this embodiment, when there are multiple target video transmission connections, the split part of the video frames is transmitted through the video stream transmission track represented by the track identifier corresponding to the video stream identifier in the target video transmission connection; and the video frames other than some video frames in the video stream are transmitted through the video stream transmission track represented by the track identifier corresponding to the video stream identifier in the target video transmission connection that is further determined. The above method dynamically schedules the same video stream to multiple video transmission connections, making full use of multi-path dynamic transmission to improve the reliability of each video stream transmission process, and the same video stream can be transmitted to the receiving end across multiple video transmission connections, which can improve network bandwidth utilization.

[0134] In one of the embodiments, the target video transmission connections determined for the same video stream in adjacent scheduling cycles are different. When switching from the target video transmission connection determined in the current scheduling cycle to the target video transmission connection determined in the next scheduling cycle, the video stream has frame loss, resulting in a problem of video stream switching jamming. For example, in one scheduling cycle, the target video transmission connection corresponding to the determined video stream i is video transmission connection 1, and in the next scheduling cycle, the target video transmission connection corresponding to the determined video stream i is video transmission connection 2 and video transmission connection 3. When switching from video transmission connection 1 of the current scheduling cycle to video transmission connection 2 and video transmission connection 3 of the next cycle, if the video stream loses frames, it will cause jamming and unsmooth switching during the switching process. Therefore, in order to solve the above problem, this embodiment, for any video stream in multiple video streams, determines at least one target video transmission connection for transmitting the video stream from multiple video transmission connections according to the network status parameters of each video transmission connection in the multiple video transmission connections, including:

[0135] In each scheduling cycle, for any video stream among multiple video streams, at least one target video transmission connection for transmitting the video stream in the current scheduling cycle is determined from the multiple video transmission connections according to the network status parameters of each video transmission connection in the multiple video transmission connections; if the target video transmission connection determined in the current scheduling cycle is different from the target video transmission connection determined in the previous scheduling cycle, the first video frame in the video stream to be scheduled to the target video transmission connection determined in the current scheduling cycle is synchronously transmitted to the receiving end through the target video transmission connection determined in the current scheduling cycle and the target video transmission connection determined in the previous scheduling cycle.

[0136] Among them, the scheduling cycle refers to the cycle for scheduling multiple video streams according to the same network status parameters. For example, one scheduling cycle (about 100ms) needs to schedule M video streams, and it needs to be scheduled multiple times, that is, for each video stream, the scheduling time is about 1-2ms. The number of schedulings in one scheduling cycle may be M times, or more than M times. In the process of scheduling multiple video streams, the same network status parameters are used, and the network status parameters obtained in the next scheduling cycle may change. For example, including M video streams, network status parameter 1 is obtained at time t1. For any of the M video streams, the M video streams are scheduled according to the network status parameter 1 at time t1, and the updated network status parameter 2 is obtained at time t1+T. The M video streams are scheduled according to the network status parameter 2 at time t1+T. Among them, T represents the scheduling cycle, which is about 100ms.

[0137] The target video transmission connection determined in the current scheduling cycle is different from the target video transmission connection determined in the previous scheduling cycle, indicating that the target video transmission connection determined in the current scheduling cycle is not exactly the same as the target video transmission connection determined in the previous scheduling cycle, that is, the target video transmission connection determined in the current scheduling cycle is partially the same as or completely different from the target video transmission connection determined in the previous scheduling cycle.

[0138] In this embodiment, based on the network status parameters of each video transmission connection in the multiple video transmission connections, at least one target video transmission connection for transmitting the video stream in the current scheduling period is determined from the multiple video transmission connections. The method steps are the same as those in the above embodiment and will not be repeated here.

[0139] The first video frame in the video stream to be scheduled to the target video transmission connection determined by the current scheduling period is determined according to the video frame to be transmitted corresponding to the switching time. For example, the time corresponding to the switching to the current scheduling period is t1, and the video frame scheduled at t1 is used as the first video frame in the video stream to be scheduled to the target video transmission connection determined by the current scheduling period.

[0140] In this embodiment, the target video transmission connection determined in the previous scheduling cycle refers to the last target video transmission connection determined in the previous scheduling cycle. For example, the target video transmission connection determined in the previous scheduling cycle includes video transmission connection 2 and video transmission connection 3, and the corresponding target video transmission connection in the current scheduling cycle is video transmission connection 1, then the last target video transmission connection determined in the previous scheduling cycle is video transmission connection 3, and the first video frame is scheduled to be transmitted on video transmission connection 3 and video transmission connection 1.

[0141] For example, the target video transmission connection corresponding to video stream i in the previous scheduling cycle is video transmission connection 1. When the 100th video frame is transmitted to video transmission connection 1, the current scheduling cycle is entered. In the current scheduling cycle, the corresponding target video transmission connections are determined to be video transmission connections 2 and 3, that is, the 100th video frame needs to be transmitted on video transmission connection 2. At this time, the 100th video frame needs to be transmitted simultaneously on video transmission connection 1 and video transmission connection 2.

[0142] For example, the target video transmission connection corresponding to video stream i in the previous scheduling cycle is video transmission connection 1. When the 100th video frame is transmitted to video transmission connection 1, the current scheduling cycle is entered. In the current scheduling cycle, the corresponding target video transmission connection is determined to be video transmission connection 2, that is, the 100th video frame needs to be transmitted on video transmission connection 2. At this time, the 100th video frame needs to be transmitted on video transmission connection 1 and video transmission connection 2 at the same time.

[0143] For example, the target video transmission connections corresponding to video stream i in the previous scheduling cycle are video transmission connection 1 and video transmission connection 2. When the 100th video frame is transmitted to video transmission connection 2, the current scheduling cycle is entered. In the current scheduling cycle, the corresponding target video transmission connection is determined to be video transmission connection 3, that is, the 100th video frame needs to be transmitted on video transmission connection 3. At this time, the 100th video frame needs to be transmitted simultaneously on video transmission connection 1 and video transmission connection 3.

[0144] Specifically, when the target video transmission connections determined for the same video stream in adjacent scheduling cycles are different, when switching from the target video transmission connection determined in the previous scheduling cycle to the target video transmission connection determined in the current scheduling cycle, the sending end uses the video frame to be transmitted corresponding to the switching moment as the first video frame in the video stream to be scheduled to the target video transmission connection determined in the current scheduling cycle, and transmits the first video frame to the receiving end via both the target video transmission connection determined in the current scheduling cycle and the target video transmission connection determined in the previous scheduling cycle.

[0145] In this embodiment, the target video transmission connections determined in adjacent scheduling cycles for the same video stream are different, and when switching from the target video transmission connection determined in the previous scheduling cycle to the target video transmission connection determined in the current scheduling cycle, the video frame to be transmitted corresponding to the switching moment is used as the first video frame in the video stream to be scheduled to the target video transmission connection determined in the current scheduling cycle, and the first video frame is transmitted to the receiving end simultaneously through the target video transmission connection determined in the current scheduling cycle and the target video transmission connection determined in the previous scheduling cycle, thereby avoiding frame loss, ensuring that the switching of video streams will be smoother, reducing the possibility of frame loss, and improving the fault tolerance rate of switching.

[0146] In a detailed embodiment, a video stream transmission method specifically includes the following steps:

[0147] 1. Establish multiple video transmission connections according to different network transmission paths between the sending end and the receiving end; the multiple video transmission connections are used to transmit multiple video streams generated locally, each video transmission connection includes multiple video stream transmission tracks, and each video stream transmission track is used to transmit one of the multiple video streams.

[0148] 2. Determine the video transmission connection with the lowest network delay among multiple video transmission connections.

[0149] 3. For any video stream among the multiple video streams, the preset encoding rate of the video stream is used as the initial residual encoding rate. If the residual encoding rate of the video stream is less than or equal to the recommended sending rate of the video transmission connection with the lowest network delay, step 4 is executed; if the residual encoding rate of the video stream is greater than the recommended sending rate of the video transmission connection with the lowest network delay, step 8 is executed.

[0150] 4. Use the video transmission connection with the lowest network delay as the target video transmission connection for transmitting the video stream.

[0151] 5. Update the recommended sending rate of the target video transmission connection according to the recommended sending rate of the target video transmission connection and the remaining encoding rate of the video stream; the updated recommended sending rate is used to participate in the path scheduling of the next video stream in the multi-channel video stream.

[0152] 6. Using the video stream transmission track represented by the track identifier corresponding to the video stream identifier in the determined target video transmission connection as the target video stream transmission track for transmitting the video stream to be transmitted.

[0153] 7. Transmit the video stream to the receiving end through the target video stream transmission track, and end the video stream transmission.

[0154] 8. Use the video transmission connection with the lowest network delay as the first target video transmission connection for transmitting the video stream.

[0155] 9. Obtain preset splitting parameters, and calculate a splitting factor according to the splitting parameters, a preset encoding rate, and a recommended sending rate of a target video transmission connection; split the video stream into multiple video frames according to the splitting factor, with each video frame being a split partial video frame.

[0156] 10. Calculate the scheduled rate of the video stream according to the preset coding rate and the splitting factor; calculate the difference between the remaining coding rate and the scheduled rate; and use the calculated difference as the updated remaining coding rate of the video stream.

[0157] 11. Continue to determine at least one target video transmission connection for transmitting the video stream from the other video transmission connections according to the updated remaining coding rate and the recommended sending rates of the other video transmission connections except the target video transmission connection among the multiple video transmission connections.

[0158] 12. Using the video stream transmission track represented by the track identifier corresponding to the video stream identifier in each determined target video transmission connection as the target video stream transmission track for transmitting the video stream to be transmitted.

[0159] 13. Using the video stream transmission track represented by the track identifier corresponding to the video stream identifier in each determined target video transmission connection as the target video stream transmission track for transmitting the video stream to be transmitted.

[0160] 14. Transmit the split part of the video frames through the video stream transmission track represented by the track identifier corresponding to the video stream identifier in the first target video transmission connection; transmit the video frames other than the part of the video frames in the video stream through the video stream transmission track represented by the track identifier corresponding to the video stream identifier in the continuously determined target video transmission connection, and end the video stream transmission.

[0161] In this embodiment, multiple video transmission connections are established according to different network transmission paths between the sending end and the receiving end, multiple video stream transmission tracks are divided in each video transmission connection, and a track identifier is assigned to each video stream transmission track, and a video stream identifier is assigned to each video stream to be transmitted, and according to the network status parameters of each video transmission connection in the multiple video transmission connections, at least one target video transmission connection for transmitting the video stream is determined from the multiple video transmission connections. That is, the video stream can be dynamically scheduled to the multiple video transmission connections according to the network status parameters, and the reliability of the transmission process of each video stream can be improved by making full use of multi-path dynamic transmission, and the same video stream can be transmitted to the receiving end across multiple video transmission connections, which can improve the network bandwidth utilization.

[0162] In one embodiment, this embodiment also provides a video stream transmission method, which can be applied to Figure 1In the application environment shown. Among them, the receiving end 104 establishes multiple video transmission connections according to different network transmission paths between the sending end 102 and the receiving end 104; the multiple video transmission connections are used to receive multiple video streams generated by the sending end, each video transmission connection includes multiple video stream transmission tracks, and each video stream transmission track is used to receive one of the multiple video streams; the receiving end 104 receives multiple video streams transmitted on each video transmission connection; the video stream has a video stream identifier, and the video stream identifier corresponds to the track identifier of one of the multiple video stream transmission tracks; when a video stream with the same video stream identifier is received through multiple different video transmission connections, the receiving end 104 merges the video streams with the same video stream identifier to obtain one of the multiple video streams. Among them, the sending end 102 can be, but is not limited to, various desktop computers, laptops, smart phones, tablet computers, Internet of Things devices and portable wearable devices, and the Internet of Things devices can be video gateways, etc. Portable wearable devices can be smart watches, smart bracelets, head-mounted devices, etc. The receiving end 104 can be a video server or a server cluster composed of multiple servers.

[0163] In one embodiment, Figure 7 As shown, a video stream transmission method is provided, and the method is applied to Figure 1 The receiving end in FIG. 1 is taken as an example to illustrate, and the following steps are included:

[0164] Step 702, establish multiple video transmission connections according to different network transmission paths between the sending end and the receiving end; the multiple video transmission connections are used to receive multiple video streams generated by the sending end, each video transmission connection includes multiple video stream transmission tracks, and each video stream transmission track is used to receive one of the multiple video streams.

[0165] The video transmission connection refers to a video channel for transmitting video streams on the network transmission path between the sender and the receiver. Each video transmission connection is established on a different network transmission path, and one video transmission connection can be established on one network transmission path. Specifically, it is established based on a local IP address and a network communication port of the sender, and a local IP address and a network communication port of the receiver; a local IP address and a network communication port can determine a network address. For example, Figure 4 is a schematic diagram of the video transmission connection between the transmitter and the receiver, such as Figure 4As shown, the sending end has N network cards and the receiving end has one network card. Then, according to the channel between each network card of the sending end and the network card of the receiving end, N network transmission paths are established, and a video transmission connection is established on each network transmission path. For example, according to the channel between network card 1 of the sending end and the network card of the receiving end, network transmission path 1 is established, and a video transmission connection 1 is established on network transmission path 1. For example, according to the channel between network card N of the sending end and the network card of the receiving end, network transmission path N is established, and a video transmission connection N is established on network transmission path N.

[0166] The video stream transmission track is a logical division of data transmission on the video transmission connection to obtain a logically isolated video transmission data channel. Each video transmission connection includes multiple video stream transmission tracks. The number of video stream transmission tracks can be set according to actual needs. In this embodiment, the number of video stream transmission tracks is set to be the same as the number of multiple video streams. That is, if the sending end has a total of M video streams, each video transmission connection includes M video stream transmission tracks. For example, Figure 4 As shown, M video stream transmission tracks are divided on each network transmission path, and a track identifier is assigned to each video stream transmission track. In this embodiment, T_nm is used to represent the track identifier of the m-th video stream transmission track in the n-th video transmission connection. Taking network transmission path 1 as an example, network transmission path 1 is divided into M video stream transmission tracks, and the track identifier of the first video stream transmission track is recorded as T_11, and the track identifier of the m-th video stream transmission track is recorded as T_1m. Taking network transmission path N as an example, network transmission path N is divided into M video stream transmission tracks, and the track identifier of the first video stream transmission track is recorded as T_n1, and the track identifier of the m-th video stream transmission track is recorded as T_nm.

[0167] Video streams can be video streams from multiple different video sources such as cameras, screen capture, video files, etc. Multi-channel video streams are multi-channel video streams generated by the same sending end. For example, the sending end includes multiple cameras, and the multiple video streams captured by the multiple cameras are transmitted to the receiving end through multiple video transmission connections. The sending end generates the original data of the video stream, and packets it according to the RTP protocol, adds the video stream identifier during packetization, and then packets it according to the UDP protocol and encapsulates it, using the same local IP address and network communication port number, encapsulating it into a UDP message, and then sending the video stream out. The embodiment of the present application adopts the protocol stack of real-time video transmission itself, which has its own anti-packet loss capabilities, and no additional aggregation transmission overhead.

[0168] In some embodiments, the multi-channel video streams can be multi-channel video streams collected in a remote real-time control scenario. For example, the sending end is a terminal and the receiving end is a video server. The terminal transmits the video streams collected by multiple cameras to the video server through multiple video transmission connections. The video server displays the multi-channel video streams on a monitoring platform. On-site staff issue control instructions to the terminal based on the video images displayed on the monitoring platform. The terminal executes actions according to the control instructions, thereby realizing the remote real-time control function.

[0169] In some embodiments, the multi-channel video streams may be multi-channel video streams collected in a real-time vehicle monitoring scenario. For example, the sending end is a vehicle controller, the receiving end is a server, and multiple cameras are set at different positions of the vehicle. The vehicle controller obtains the video streams collected by the multiple cameras and transmits the multi-channel video streams to the server through multiple video transmission connections. The server displays the multi-channel video streams on the vehicle monitoring platform, thereby realizing real-time monitoring of the surrounding environment of the scene where the vehicle is located, which can be applied to a vehicle navigation system or a vehicle automatic driving system.

[0170] Specifically, the receiving end establishes multiple video transmission connections based on the multiple local IP addresses and network communication ports of the sending end, as well as the multiple local IPs and network communication ports of the receiving end, and logically divides the data transmission on each video transmission connection to obtain multiple logically isolated video stream transmission tracks.

[0171] Step 704: receiving multiple video streams transmitted on each video transmission connection; the video stream has a video stream identifier, and the video stream identifier corresponds to a track identifier of one of the multiple video stream transmission tracks.

[0172] Among them, each video stream includes a video stream identifier for distinguishing different video streams. In order to facilitate the receiving end to distinguish the received video streams, this embodiment sets each video stream transmission track to include a track identifier, and the track identifier is used to identify the video stream transmitted on different video stream transmission tracks in the same video transmission connection, and the video stream identifier corresponds to the track identifier of one of the multiple video stream transmission tracks. When scheduling the video stream, the mth video stream is scheduled on the video stream transmission track whose track identifier in at least one video transmission connection corresponds to the video stream identifier of the mth video stream. This embodiment uses T_nm to represent the track identifier of the mth video stream transmission track in the nth video transmission connection. For example, Figure 4 As shown, this embodiment includes M video streams and N video transmission connections, each video transmission connection includes M video stream transmission tracks, and the mth video stream is scheduled to the mth video stream transmission track in the nth video transmission connection, that is, the mth video stream occupies the video stream transmission track with the track identification T_nm.

[0173] Among them, the video stream identifier corresponds to the track identifier, indicating that there is a corresponding relationship between the video stream corresponding to the video stream identifier and the video stream transmission track corresponding to the track identifier. For example, the video stream identifier corresponds to the track identifier, and the video stream identifier and the track identifier may be the same.

[0174] Step 706: When video streams with the same video stream identifier are received through multiple different video transmission connections, the video streams with the same video stream identifier are merged to obtain one of the multiple video streams.

[0175] The merging of video streams with the same video stream identifier specifically refers to merging multiple video streams into one video stream according to the timestamps of the video streams to obtain one of the multiple video streams.

[0176] For example, the video stream identifiers received by multiple different video transmission connections include video stream 1, video stream 2, ..., video stream m, and the video stream 1 with the multiple video stream identifiers 1 is merged into the first video stream in the multiple video streams.

[0177] In some embodiments, if the sending end transmits the same video stream as a whole through only one video transmission connection, the receiving end receives a video stream with a different identifier from other video streams. In this case, there is no need to merge the video stream. If the sending end splits the same video stream into multiple parts and transmits the same video stream through multiple target video transmission connections, each target video transmission connection transmits partial video frames split from the same video stream, and the receiving end receives multiple partial video frames with the same video stream identifier. In this case, the partial video frames with the same video stream identifier are merged to obtain a video stream with the same video stream identifier.

[0178] For example, if the sender transmits video stream 1 to the receiver only through video transmission connection 1, the receiver can only receive one video stream 1. In this case, there is no need to merge video stream 1. If the sender transmits video stream 1 to the receiver through video transmission connection 1 and video transmission connection 2, the receiver can receive two video streams 1. In this case, the two video streams 1 need to be merged to obtain the video stream 1 sent by the sender.

[0179] Specifically, the receiving end parses the received video streams to obtain video stream identifiers of video streams transmitted on multiple video transmission connections, and merges at least one video stream with the same video stream identifier into a video stream represented by the video stream identifier in the multiple video streams.

[0180] In this embodiment, multiple video transmission connections are established according to different network transmission paths between the sending end and the receiving end, multiple video stream transmission tracks are divided in each video transmission connection, and a track identifier is assigned to each video stream transmission track, and a video stream identifier is assigned to each video stream to be transmitted. The sending end dynamically schedules the video stream to multiple video transmission connections, making full use of multi-path dynamic transmission to improve the reliability of the transmission process of each video stream, and the same video stream can be transmitted to the receiving end across multiple video transmission connections, which can improve network bandwidth utilization; the receiving end merges the video streams received from multiple different video transmission connections according to the video stream identifier to obtain one of the multiple video streams, and can quickly identify the video streams belonging to the same video stream, thereby improving the video stream identification efficiency of the receiving end.

[0181] In one embodiment, in order to synchronize the output of video streams with the same video stream identifier transmitted on multiple different video transmission connections, a video stream buffer is set in the receiving end to cache the video streams with the same video stream identifier transmitted on multiple different video transmission connections. Specifically, when video streams with the same video stream identifier are received through multiple different video transmission connections, the video streams with the same video stream identifier are merged to obtain one of the multiple video streams, including:

[0182] When video streams with the same video stream identifier are received through multiple different video transmission connections, a corresponding video stream buffer is established for the video stream identifier; the received video streams with the same video stream identifier are stored in the video stream buffer; the video streams in the video stream buffer are sorted according to the timestamps of the video frames and then merged to obtain one of the multiple video streams.

[0183] The video stream buffer refers to a temporary storage space allocated in the memory of the receiving end, which is used to temporarily store received video streams with the same video stream identifier and sort the video streams with the same video stream identifier according to the timestamps of the video streams.

[0184] For example, the receiving end receives 3 video streams with the same video stream identifier, stores the 3 video streams with the same video stream identifier in a video stream buffer, sorts the video frames in the 3 video streams according to the timestamps of each video frame, and merges the sorted video frames to obtain a video stream with the same video stream identifier.

[0185] Specifically, when video streams with the same video stream identifier are received through multiple different video transmission connections, the receiving end allocates temporary storage space in the local memory as a video stream buffer, and stores the received video streams with the same video stream identifier in the video stream buffer; the receiving end sorts the video frames in the video stream according to the timestamp of each video frame in the video stream, and merges the sorted video frames to obtain a video stream with the same video stream identifier.

[0186] In this embodiment, a video stream buffer is set in the receiving end to cache video streams with the same video stream identifier transmitted on multiple different video transmission connections, so that the video streams with the same video stream identifier transmitted on multiple different video transmission connections can be output synchronously.

[0187] In some embodiments, in order to ensure that all video streams with the same video stream identifier can be received and the merged video stream can be quickly output, the present embodiment sets a buffering time in the video stream buffer. Specifically, before the video streams in the video stream buffer are sorted according to the timestamps of the video frames and then merged to obtain one of the multiple video streams, the method further includes:

[0188] Obtain the buffering durations of the multiple target video stream transmission tracks that receive the video stream with the same video stream identifier; determine the buffering duration threshold of the video stream buffer according to the buffering durations of the multiple target video stream transmission tracks; when the buffering duration of the video stream buffer is greater than or equal to the buffering duration threshold, execute the step of sorting the video streams in the video stream buffer according to the timestamps of the video frames and then merging them to obtain one of the multiple video streams, and clear the buffering duration of the video stream buffer.

[0189] Among them, target video stream transmission channels that output video streams with the same video stream identifier on multiple different video transmission connections are determined, and the buffering time of each target video stream transmission channel is determined through a network perception module on the protocol stack of the video transmission connection.

[0190] Among them, clearing the buffering time of the video stream buffer means distributing the merged video stream in the video stream buffer to subsequent modules, and setting the buffering time of the video stream buffer to zero. For example, the merged video stream in the video stream buffer is distributed to the upper-layer application running on the receiving end, and the upper-layer application needs to render and display the video, etc., or it can be forwarded to other terminals, etc.

[0191] In some embodiments, determining a buffering duration threshold of a video stream buffer according to respective buffering durations of a plurality of target video stream transmission tracks comprises the following steps:

[0192] Determine the different video transmission connections to which multiple target video stream transmission tracks respectively belong; for each target video stream transmission track, calculate the sum of the buffering time of the target video stream transmission track and the network delay of the corresponding video transmission connection; determine the buffering time threshold of the video stream buffer according to the maximum and minimum values ​​of the sum of the cache time and the network delay calculated for each target video stream transmission track.

[0193] Among them, the target video stream transmission track refers to multiple video stream transmission tracks that transmit video streams with the same video stream identifier. When receiving each target video stream transmission track, the receiving end can determine the video transmission connection to which each target video stream transmission track belongs according to the track identifier. For example, the track identifier of the target video stream transmission track is T_12. According to the track identifier T_12, it can be determined that the target video stream transmission track belongs to the second video stream transmission track of video transmission connection 1. Therefore, it can be determined that the video transmission connection to which the target video stream transmission track belongs is video transmission connection 1.

[0194] For a video stream with video stream identifier j, video stream j occupies multiple target video stream transmission tracks when it is sent, and the corresponding track identifier is T_nj. In this embodiment, the buffering time of the target video stream transmission track is recorded as TB_nj, TB_nj represents the jth video stream transmission track in the nth video transmission connection, and the network delay of the video transmission connection to which the target video stream transmission track belongs is recorded as Rtt_n, which represents the network delay of the nth video transmission connection. Calculate the sum of the buffering time of each target video stream transmission track and the network delay of the video transmission connection to which it belongs, take the maximum and minimum values ​​of the sum of the cache time and network delay calculated for each target video stream transmission track, and determine the buffering time threshold of the video stream buffer according to the following formula:

[0195] max(Rtt_n+TB_nj)–min(Rtt_n+TB_nj)+TB_0;

[0196] Among them, TB_0 is the buffer protection margin.

[0197] For example, for video stream 1, the corresponding target video stream transmission tracks are T_11 and T_21, the buffering time TB_11 of the target video stream transmission track T_11 is 10ms, and the network delay RTT1 of the video transmission connection to which the target video stream transmission track T_11 belongs is 30ms; the buffering time TB_21 of the target video stream transmission track T_21 is 15ms, and the network delay RTT2 of the video transmission connection to which the target video stream transmission track T_21 belongs is 40ms, then RTT1+TB_11=40ms and RTT2+TB_21=55ms are calculated, wherein the maximum value of the sum of the cache time and the network delay calculated for each target video stream transmission track is 55ms, and the minimum value of the sum of the cache time and the network delay calculated for each target video stream transmission track is 40ms, and the buffer protection margin TB_0 is taken as 5ms, then the buffering time threshold of the video stream buffer is: 55ms-40ms+5ms=20ms.

[0198] Specifically, the receiving end determines a target video stream transmission channel that outputs a video stream with the same video stream identifier on multiple different video transmission connections, determines the buffering time of each target video stream transmission channel, and for each target video stream transmission track, calculates the sum of the buffering time of each target video stream transmission track and the network delay of the video transmission connection to which it belongs, takes the maximum and minimum values ​​of the sum of the buffering time and the network delay calculated for each target video stream transmission track, determines the difference between the maximum and the minimum, and uses the sum of the difference and the buffer protection margin as the buffering time threshold of the video stream buffer; the receiving end records the buffering time of the video stream stored in the video stream buffer, and when the buffering time of the video stream buffer is greater than or equal to the buffering time threshold, executes the step of sorting the video streams in the video stream buffer according to the timestamps of the video frames and then merging them to obtain one of the multiple video streams, distributes the merged video streams in the video stream buffer to subsequent modules, and sets the buffering time of the video stream buffer to zero.

[0199] In this embodiment, the buffering time of each target video stream transmission track and the sum of the network delay of the corresponding video transmission connection are taken; the maximum and minimum values ​​of the sum of the cache time and the network delay calculated for each target video stream transmission track are taken, and the buffering time threshold of the video stream buffer is determined according to the maximum and minimum values; based on the above method for determining the buffering time threshold, it is ensured that all video streams with the same video stream identifier can be received, and the merged video stream can be quickly output at the same time.

[0200] In a detailed embodiment, a video stream transmission method specifically includes the following steps:

[0201] 1. Establish multiple video transmission connections according to different network transmission paths between the sending end and the receiving end; the multiple video transmission connections are used to receive multiple video streams generated by the sending end, each video transmission connection includes multiple video stream transmission tracks, and each video stream transmission track is used to receive one of the multiple video streams.

[0202] 2. Receive multiple video streams transmitted on each video transmission connection; the video stream has a video stream identifier, and the video stream identifier corresponds to a track identifier of one of the multiple video stream transmission tracks.

[0203] 3. When video streams with the same video stream identifier are received through multiple different video transmission connections, a corresponding video stream buffer is established for the video stream identifier; and the received video streams with the same video stream identifier are stored in the video stream buffer.

[0204] 4. Obtain the buffering duration of each of the multiple target video stream transmission tracks that receive the video stream with the same video stream identifier.

[0205] 5. Determine different video transmission connections to which the multiple target video stream transmission tracks respectively belong.

[0206] 6. For each target video stream transmission track, calculate the sum of the buffering time of the target video stream transmission track and the network delay of the corresponding video transmission connection.

[0207] 7. Determine a buffering time threshold of the video stream buffer according to the maximum and minimum values ​​of the sum of the cache time and the network delay calculated for each target video stream transmission track.

[0208] 8. When the buffering time of the video stream buffer is greater than or equal to the buffering time threshold, the video streams in the video stream buffer are sorted according to the timestamps of the video frames and then merged to obtain one of the multiple video streams, and the buffering time of the video stream buffer is cleared.

[0209] In this embodiment, the receiving end merges the video streams received from multiple different video transmission connections according to the video stream identifier to obtain one of the multiple video streams, so that the video streams belonging to the same video stream can be quickly identified, thereby improving the video stream identification efficiency of the receiving end; a video stream buffer is established to cache video streams with the same video stream identifier, so that the video streams with the same video stream identifier transmitted on multiple different video transmission connections can maintain synchronous output.

[0210] The following is a specific example to introduce the video stream transmission method provided by the embodiment of the present application, as follows:

[0211] Assume that there are two 30fps video streams S1 and S2. The sender has two network cards A and B, and the receiver has one network card C. The networks of A and B can both connect to the network of C. Thus, there are two network channels from the sender to the receiver, A->C and B->C, corresponding to two video transmission connections, namely video transmission connection 1 and video transmission connection 2. Video transmission connection 1 is established on network channel 1 (A->C), and video transmission connection 2 is established on network channel 2 (B->C).

[0212] Video transmission connection 1 establishes video stream transmission track T_11 and video stream transmission track T1_2 for two video streams, and video transmission connection 2 establishes video stream transmission track T_21 and video stream transmission track T_22 for two video streams.

[0213] Assume that the preset encoding rate of video stream S1 is 2 Mbps, the preset encoding rate of video stream S2 is 2 Mbps, the RTT of video transmission connection 1 is 30 ms, the recommended sending rate is 3 Mbps, the network measurement and congestion control of video transmission connection 2, the feedback RTT is 40 ms, and the recommended sending rate is 3 Mbps;

[0214] According to the video stream dynamic scheduling algorithm, video transmission connection 1 is the video transmission connection with the lowest RTT and is operated first;

[0215] Schedule video stream S1 to video transmission connection 1, and update P1 to 1Mbps; since video stream S2 exceeds the remaining recommended bit rate of video transmission connection 1, split video stream S2. Assuming alpha is 0.2, place the video frames of video stream S2 with a split factor ratio of 1 / 2.4 on video transmission connection 1; and place the remaining video frames of video stream S2 on video transmission connection 2;

[0216] After scheduling, video stream S1 corresponds to video stream transmission track T_11, and video stream S2 corresponds to video stream transmission track T_12 and video stream transmission track T_22. The sender transmits video stream S1 to the receiver through video stream transmission track T_11, and transmits video stream S2 to the receiver through video stream transmission track T_12 and video stream transmission track T_22.

[0217] The receiving end receives video data from different video stream transmission tracks, distributes the data transmitted on the video stream transmission track T_11 to the video stream S1, and distributes the data transmitted on the video stream transmission track T_12 and the video stream transmission track T_22 to the video stream S2.

[0218] Since the video stream S2 uses multiple video stream transmission tracks, a buffer is established for the video stream S2 data during distribution.

[0219] Assuming that the buffering time of video stream transmission track T_22 is 10ms, the buffering time of T_12 is 12ms, and TB0 is 5ms, the buffering time of the video stream buffer is: max(40+10,30+12)–min(40+10,30+12)+5ms=13ms, that is, the video stream data output starts 13ms after the first video frame arrives at the buffer.

[0220] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0221] Based on the same inventive concept, the embodiment of the present application further provides a video stream transmission device for implementing the video stream transmission method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more video stream transmission device embodiments provided below can refer to the limitations of the video stream transmission method above, and will not be repeated here.

[0222] In one embodiment, Figure 8 As shown, a video stream transmission device is provided, including: a first establishment module 801, a scheduling module 802 and a sending module 803, wherein:

[0223] The first establishing module 801 is used to establish multiple video transmission connections according to different network transmission paths between the sending end and the receiving end; the multiple video transmission connections are used to transmit multiple video streams generated locally, each video transmission connection includes multiple video stream transmission tracks, and each video stream transmission track is used to transmit one of the multiple video streams;

[0224] The scheduling module 802 is used to determine, for any one of the multiple video streams, at least one target video transmission connection for transmitting the video stream from the multiple video transmission connections according to the network state parameter of each video transmission connection in the multiple video transmission connections; the video stream has a video stream identifier, and the video stream identifier corresponds to a track identifier of one of the multiple video stream transmission tracks; determine a target video stream transmission track represented by a track identifier corresponding to the video stream identifier in the at least one target video transmission connection;

[0225] The sending module 803 is used to transmit the video stream to the receiving end through the target video stream transmission track.

[0226] In one of the embodiments, the network status parameters include network latency and a recommended sending rate;

[0227] The scheduling module 802 is also used to determine the video transmission connection with the lowest network delay among multiple video transmission connections; for any video stream among the multiple video streams, the preset encoding rate of the video stream is used as the initial residual encoding rate. If the residual encoding rate of the video stream is less than or equal to the recommended sending rate of the video transmission connection with the lowest network delay, the video transmission connection with the lowest network delay is used as the target video transmission connection for transmitting the video stream.

[0228] In one of the embodiments, the scheduling module 802 is also used to update the recommended sending rate of the target video transmission connection based on the recommended sending rate of the target video transmission connection and the remaining encoding rate of the video stream; the updated recommended sending rate is used to participate in the path scheduling of the next video stream in the multiple video streams.

[0229] In one of the embodiments, the scheduling module 802 is further used to use the video transmission connection with the lowest network delay as a target video transmission connection for transmitting the video stream if the remaining coding rate of the video stream is greater than the recommended sending rate of the video transmission connection with the lowest network delay, and determine a splitting factor according to a preset coding rate and the recommended sending rate of the target video transmission connection, and split some video frames from the video stream according to the splitting factor; update the remaining coding rate of the video stream according to the preset coding rate, the remaining coding rate and the splitting factor, and continue to determine at least one target video transmission connection for transmitting the video stream from the other video transmission connections according to the updated remaining coding rate and the recommended sending rates of other video transmission connections among the multiple video transmission connections except the target video transmission connection.

[0230] In one embodiment, the scheduling module 802 is further configured to use the video stream transmission track indicated by the track identifier corresponding to the video stream identifier in each determined target video transmission connection as the target video stream transmission track for transmitting the video stream.

[0231] In one of the embodiments, the sending module 803 is also used to transmit the split partial video frames through the video stream transmission track represented by the track identifier corresponding to the video stream identifier in the target video transmission connection; and transmit video frames other than partial video frames in the video stream through the video stream transmission track represented by the track identifier corresponding to the video stream identifier in the target video transmission connection that is further determined.

[0232] In one of the embodiments, the scheduling module 802 is also used to obtain preset splitting parameters, calculate the splitting factor according to the splitting parameters, the preset encoding rate and the recommended sending rate of the target video transmission connection; split the video stream into multiple video frames according to the splitting factor, and each video frame is used as a split partial video frame.

[0233] In one of the embodiments, the scheduling module 802 is further used to calculate the scheduled rate of the video stream based on the preset coding rate and the splitting factor; calculate the difference between the remaining coding rate and the scheduled rate; and use the calculated difference as the updated remaining coding rate of the video stream.

[0234] In one of the embodiments, the sending module 803 is also used to determine, in each scheduling period, for any video stream among the multiple video streams, from the multiple video transmission connections, at least one target video transmission connection for transmitting the video stream in the current scheduling period according to the network status parameters of each video transmission connection in the multiple video transmission connections; if the target video transmission connection determined in the current scheduling period is different from the target video transmission connection determined in the previous scheduling period, the first video frame in the video stream to be scheduled to the target video transmission connection determined in the current scheduling period is synchronously transmitted to the receiving end through the target video transmission connection determined in the current scheduling period and the target video transmission connection determined in the previous scheduling period.

[0235] Each module in the above video stream transmission device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0236] Based on the same inventive concept, the embodiment of the present application further provides a video stream transmission device for implementing the video stream transmission method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more video stream transmission device embodiments provided below can refer to the limitations of the video stream transmission method above, and will not be repeated here.

[0237] In one embodiment, Fig. 9As shown, a video stream transmission device is provided, including a second establishing module 901, a receiving module 902 and a merging module 903, wherein:

[0238] The second establishing module 901 is used to establish multiple video transmission connections according to different network transmission paths between the sending end and the receiving end; the multiple video transmission connections are used to receive multiple video streams generated by the sending end, each video transmission connection includes multiple video stream transmission tracks, and each video stream transmission track is used to receive one of the multiple video streams;

[0239] The receiving module 902 is used to receive multiple video streams transmitted on each video transmission connection; the video stream has a video stream identifier, and the video stream identifier corresponds to a track identifier of one of the multiple video stream transmission tracks;

[0240] The merging module 903 is used to merge the video streams with the same video stream identifier when the video streams with the same video stream identifier are received through multiple different video transmission connections to obtain one of the multiple video streams.

[0241] In one of the embodiments, the merging module 903 is also used to establish a corresponding video stream buffer for the video stream identifier when a video stream with the same video stream identifier is received through multiple different video transmission connections; store the received video stream with the same video stream identifier in the video stream buffer; sort the video streams in the video stream buffer according to the timestamps of the video frames and then merge them to obtain one of the multiple video streams.

[0242] In one of the embodiments, the merging module 903 is also used to obtain the buffering duration of each of the multiple target video stream transmission tracks that receive the video stream with the same video stream identifier; determine the buffering duration threshold of the video stream buffer according to the buffering duration of each of the multiple target video stream transmission tracks; when the buffering duration of the video stream buffer is greater than or equal to the buffering duration threshold, execute the step of sorting the video streams in the video stream buffer according to the timestamps of the video frames and then merging them to obtain one of the multiple video streams, and clear the buffering duration of the video stream buffer.

[0243] In one of the embodiments, the merging module 903 is also used to determine the different video transmission connections to which each of the multiple target video stream transmission tracks belongs; for each target video stream transmission track, calculate the sum of the buffering time of the target video stream transmission track and the network delay of the corresponding video transmission connection; determine the buffering time threshold of the video stream buffer based on the maximum and minimum values ​​of the sum of the cache time and the network delay calculated for each target video stream transmission track.

[0244] In one embodiment, a computer device is provided, which may be Figure 2 The server shown in FIG. 1 may have an internal structure as shown in FIG. Fig.10 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store video stream data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by a processor, a video stream transmission method is implemented, for example, multiple video transmission connections are established according to different network transmission paths between a sending end and a receiving end; the multiple video transmission connections are used to transmit multiple video streams generated locally, each video transmission connection includes multiple video stream transmission tracks, and each video stream transmission track is used to transmit one of the multiple video streams; for any video stream in the multiple video streams, according to the network status parameters of each video transmission connection in the multiple video transmission connections, at least one target video transmission connection for transmitting the video stream is determined from the multiple video transmission connections; the video stream has a video stream identifier, and the video stream identifier corresponds to the track identifier of one of the multiple video stream transmission tracks; a target video stream transmission track represented by a track identifier corresponding to the video stream identifier in at least one target video transmission connection is determined; and the video stream is transmitted to the receiving end through the target video stream transmission track.

[0245] In one embodiment, a computer device is provided, which may be Figure 2 The terminal shown in FIG. 1 may have an internal structure as shown in FIG. Fig.11As shown. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. The processor, the memory and the input / output interface are connected via a system bus, and the communication interface, the display unit and the input device are connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by a processor, a video stream transmission method is implemented, for example, multiple video transmission connections are established according to different network transmission paths between a sending end and a receiving end; the multiple video transmission connections are used to receive multiple video streams generated by a sending end, each video transmission connection includes multiple video stream transmission tracks, and each video stream transmission track is used to receive one of the multiple video streams; multiple video streams transmitted on each video transmission connection are received; the video stream has a video stream identifier, and the video stream identifier corresponds to a track identifier of one of the multiple video stream transmission tracks; when video streams with the same video stream identifier are received through multiple different video transmission connections, the video streams with the same video stream identifier are merged to obtain one of the multiple video streams.

[0246] The display unit of the computer device is used to form a visually visible image, and can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device can be a touch layer covered on the display screen, or a button, trackball or touchpad set on the computer device casing, or an external keyboard, touchpad or mouse, etc.

[0247] Those skilled in the art will understand that Fig.10 and Fig.11 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0248] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, each step of the video stream transmission method provided in the embodiment of the present application is implemented.

[0249] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, each step of the video stream transmission method provided in the embodiment of the present application is implemented.

[0250] In one embodiment, a computer program product is provided, including a computer program, which, when executed by a processor, implements the various steps of the video stream transmission method provided in the embodiment of the present application.

[0251] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.

[0252] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0253] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0254] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A video stream transmission method, characterized in that: The method comprises: Establish multiple video transmission connections according to different network transmission paths between the sending end and the receiving end; the multiple video transmission connections are used to transmit multiple video streams generated locally, each of the video transmission connections includes multiple video stream transmission tracks, and each of the video stream transmission tracks is used to transmit one of the multiple video streams; For any one of the multiple video streams, determining at least one target video transmission connection for transmitting the video stream from the multiple video transmission connections according to a network status parameter of each video transmission connection in the multiple video transmission connections; the video stream has a video stream identifier, and the video stream identifier corresponds to a track identifier of one of the multiple video stream transmission tracks; Determine a target video stream transmission track represented by a track identifier corresponding to the video stream identifier in the at least one target video transmission connection; The video stream is transmitted to the receiving end through the target video stream transmission track.

2. The method according to claim 1, characterized in that The network status parameters include network delay and recommended sending rate; The method of determining, for any one of the multiple video streams, at least one target video transmission connection for transmitting the video stream from the multiple video transmission connections according to a network status parameter of each video transmission connection in the multiple video transmission connections, comprises: Determine a video transmission connection with the lowest network delay among the multiple video transmission connections; For any one of the multiple video streams, the preset encoding rate of the video stream is used as the initial residual encoding rate. If the residual encoding rate of the video stream is less than or equal to the recommended sending rate of the video transmission connection with the lowest network delay, the video transmission connection with the lowest network delay is used as the target video transmission connection for transmitting the video stream.

3. The method according to claim 2, characterized in that The method further comprises: The recommended sending rate of the target video transmission connection is updated according to the recommended sending rate of the target video transmission connection and the remaining encoding rate of the video stream; the updated recommended sending rate is used to participate in the path scheduling of the next video stream in the multiple video streams.

4. The method according to claim 2, characterized in that: The method further comprises: If the remaining encoding rate of the video stream is greater than the recommended sending rate of the video transmission connection with the lowest network delay, using the video transmission connection with the lowest network delay as a target video transmission connection for transmitting the video stream, and Determine a split factor according to the preset encoding rate and a recommended sending rate of the target video transmission connection, and split some video frames from the video stream according to the split factor; The remaining coding rate of the video stream is updated according to the preset coding rate, the remaining coding rate and the splitting factor; and at least one target video transmission connection for transmitting the video stream is continued to be determined from the other video transmission connections according to the updated remaining coding rate and the recommended sending rates of other video transmission connections among the multiple video transmission connections except the target video transmission connection.

5. The method according to claim 4, characterized in that The determining of a target video stream transmission track represented by a track identifier corresponding to the video stream identifier in the at least one target video transmission connection comprises: The video stream transmission track represented by the track identifier corresponding to the video stream identifier in each determined target video transmission connection is used as the target video stream transmission track for transmitting the video stream.

6. The method according to claim 4, characterized in that The step of transmitting the video stream to the receiving end through the target video stream transmission track includes: Transmitting the split part of the video frames through the video stream transmission track represented by the track identifier corresponding to the video stream identifier in the target video transmission connection; The video frames other than the partial video frames in the video stream are transmitted by continuing to determine the video stream transmission track represented by the track identifier corresponding to the video stream identifier in the target video transmission connection.

7. The method according to claim 4, characterized in that The step of determining a splitting factor according to the preset encoding rate and a recommended sending rate of the target video transmission connection, and splitting a portion of video frames from the video stream according to the splitting factor, comprises: Obtaining a preset splitting parameter, and calculating a splitting factor according to the splitting parameter, the preset encoding rate, and a recommended sending rate of the target video transmission connection; The video stream is split into a plurality of video frames according to the splitting factor, and each video frame serves as the split partial video frame.

8. The method according to claim 4, characterized in that The updating of the remaining coding rate of the video stream according to the preset coding rate, the remaining coding rate and the splitting factor comprises: Calculating a scheduled rate of the video stream according to the preset encoding rate and the splitting factor; Calculating a difference between the remaining coding rate and the scheduled rate; The calculated difference is used as the updated residual coding rate of the video stream.

9. The method according to any one of claims 1 to 8, characterized in that: The method of determining, for any one of the multiple video streams, at least one target video transmission connection for transmitting the video stream from the multiple video transmission connections according to a network status parameter of each video transmission connection in the multiple video transmission connections, comprises: In each scheduling period, for any one of the multiple video streams, determining, from the multiple video transmission connections, at least one target video transmission connection for transmitting the video stream in the current scheduling period according to a network state parameter of each video transmission connection in the multiple video transmission connections; The method further comprises: If the target video transmission connection determined by the current scheduling cycle is different from the target video transmission connection determined by the previous scheduling cycle, the first video frame in the video stream to be scheduled to the target video transmission connection determined by the current scheduling cycle is synchronously transmitted to the receiving end through the target video transmission connection determined by the current scheduling cycle and the target video transmission connection determined by the previous scheduling cycle.

10. A video stream transmission method, characterized in that: The method comprises: Establish multiple video transmission connections according to different network transmission paths between the sending end and the receiving end; the multiple video transmission connections are used to receive multiple video streams generated by the sending end, each of the video transmission connections includes multiple video stream transmission tracks, and each of the video stream transmission tracks is used to receive one of the multiple video streams; Receiving multiple video streams transmitted on each of the video transmission connections; the video streams have video stream identifiers, and the video stream identifiers correspond to track identifiers of one of the multiple video stream transmission tracks; When video streams with the same video stream identifier are received through multiple different video transmission connections, the video streams with the same video stream identifier are merged to obtain one of the multiple video streams.

11. The method according to claim 10, characterized in that When video streams with the same video stream identifier are received through multiple different video transmission connections, the video streams with the same video stream identifier are merged to obtain one of the multiple video streams, including: When a video stream having the same video stream identifier is received through a plurality of different video transmission connections, a corresponding video stream buffer is established for the video stream identifier; The received video streams with the same video stream identifier are stored in the video stream buffer; The video streams in the video stream buffer are sorted according to the timestamps of the video frames and then merged to obtain one of the multiple video streams.

12. The method according to claim 11, characterized in that Before sorting the video streams in the video stream buffer according to the timestamps of the video frames and merging them to obtain one of the multiple video streams, the method further includes: Obtaining the buffering duration of each of the plurality of target video stream transmission tracks receiving the video stream having the same video stream identifier; Determining a buffering time threshold of the video stream buffer according to the buffering time of each of the plurality of target video stream transmission tracks; When the buffering duration of the video stream buffer is greater than or equal to the buffering duration threshold, the step of sorting the video streams in the video stream buffer according to the timestamps of the video frames and then merging them to obtain one of the multiple video streams is performed, and the buffering duration of the video stream buffer is cleared.

13. The method according to claim 12, characterized in that The step of determining the buffering duration threshold of the video stream buffer according to the buffering durations of the plurality of target video stream transmission tracks includes: Determine different video transmission connections to which the multiple target video stream transmission tracks respectively belong; For each target video stream transmission track, calculate the sum of the buffering time of the target video stream transmission track and the network delay of the corresponding video transmission connection; The buffering time threshold of the video stream buffer is determined according to the maximum value and the minimum value of the sum of the buffering time and the network delay calculated for each target video stream transmission track.

14. A video stream transmission device, characterized in that: The device comprises: A first establishing module is used to establish multiple video transmission connections according to different network transmission paths between a sending end and a receiving end; the multiple video transmission connections are used to transmit multiple video streams generated locally, each of the video transmission connections includes multiple video stream transmission tracks, and each of the video stream transmission tracks is used to transmit one of the multiple video streams; A scheduling module, configured to determine, for any one of the multiple video streams, at least one target video transmission connection for transmitting the video stream from the multiple video transmission connections according to a network status parameter of each video transmission connection in the multiple video transmission connections; the video stream has a video stream identifier, and the video stream identifier corresponds to a track identifier of one of the multiple video stream transmission tracks; and determine a target video stream transmission track represented by a track identifier corresponding to the video stream identifier in the at least one target video transmission connection; The sending module is used to transmit the video stream to the receiving end through the target video stream transmission track.

15. A video stream transmission device, characterized in that: The device comprises: A second establishing module is used to establish multiple video transmission connections according to different network transmission paths between the sending end and the receiving end; the multiple video transmission connections are used to receive multiple video streams generated by the sending end, each of the video transmission connections includes multiple video stream transmission tracks, and each of the video stream transmission tracks is used to receive one of the multiple video streams; A receiving module, configured to receive multiple video streams transmitted on each of the video transmission connections; the video streams have video stream identifiers, and the video stream identifiers correspond to track identifiers of one of the multiple video stream transmission tracks; The merging module is used to merge the video streams with the same video stream identifier when receiving the video streams with the same video stream identifier through multiple different video transmission connections to obtain one of the multiple video streams.

16. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 13 are implemented.

17. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 13 are implemented.

18. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 13 are implemented.

Images