[0036] In order to make the object, technical solution and advantages of the present invention clearer, the present invention will be further described in detail below in conjunction with the accompanying drawings.
[0037] see figure 1 , introducing the structure of the multi-connection-based real-time streaming media transmission terminal of the present invention: the terminal is built on the basis of a traditional single-connection terminal, including all functional modules of a single-connection terminal, and a signaling control unit and a media transmission unit are added, here only Introduces new modules. in:
[0038] (1) Signaling control unit: responsible for completing the negotiation and establishment of multi-connection links: first exchange signaling messages with network entities, and complete various signaling interaction functions including user registration and establishing signaling channels; then establish signaling After the channel, the media transmission unit is triggered to start the sending and receiving process of the media stream. The unit is equipped with: a multi-connection real-time monitoring module and a multi-connection negotiation and establishment module. The functions of the two constituent modules are as follows:
[0039] Multi-connection real-time monitoring module: responsible for detecting the status of the network itself throughout the communication process, and sending the monitoring results to the multi-connection negotiation and establishment module for negotiation and connection establishment with the communication peer. As long as the network status changes, it will immediately notify the multi-connection negotiation and establishment module to adjust the network connection between the communication parties: if it finds that there is a new available connection, it will also notify the multi-connection negotiation and establishment module to send a connection establishment request to the communication peer: the request will be The new available connection is added to the existing session; if it is found that a self-used connection is broken, the multi-connection negotiation and establishment module is notified to send a connection deletion request to the communication peer: request to remove the broken connection from the existing session Delete, and send this information to the communication peer.
[0040] Multi-connection negotiation and establishment module: responsible for sending a signaling negotiation request to the communication peer to complete the session negotiation function: before the communication starts, this module receives the own network information provided by the multi-connection real-time monitoring module, and adds the own network information to the session In the establishment request message, it is sent to the communication peer to request establishment of a multi-connection session. During the communication process, the module receives its own network change information provided by the multi-connection real-time monitoring module at any time, and sends a corresponding link establishment request or link deletion request according to the type of information: The end sends a connection establishment request: request to add the new available connection to the existing session; if the information is that a connection in use by itself is broken, send a connection deletion request to the communication peer: request to delete the connection from the existing session Break the connection and send this information to the communication peer. This module also receives the information that a connection in use by itself has become invalid from the QoS quality evaluation module at any time, and then sends a request to the communication peer to delete the connection. At the same time, the module continuously receives requests sent by the communication peer, and takes corresponding processing according to different situations: if it is a connection addition request, add the new available connection to the existing session and reply to the peer end; if it is a connection deletion request , delete the connection from the existing session and reply to the peer. During the communication process, the module also exchanges information with the confluence module of the media transmission unit, and uses the stream control transmission protocol RTCP (Stream Control Transmission Protocol) data packets to collect statistics from the confluence module, including data packet transmission volume, packet loss rate, and delay. sent to the communication peer in the form of
[0041] (2) Media transmission unit: responsible for completing the sending and receiving and transmission functions of media streams: after the signaling control unit completes session establishment and establishes a signaling channel, it starts the media transmission unit to perform media interaction operations. Including: control the media acquisition device to collect the original media stream, obtain the original media stream from the media acquisition device and encode it, and then send it to the communication peer; obtain the media encoding stream from the communication peer and decode it, and then decode the decoded media stream Submit to the terminal for playback. The unit is equipped with: sequentially connected QoS quality evaluation module, traffic distribution and scheduling module, confluence module and data buffer module. The functions of the four modules are described as follows:
[0042] QoS quality evaluation module: responsible for real-time statistics and calculation of the network status and throughput capacity of each connection based on the information carried by the received RTCP data packets, including the data packet transmission volume, packet loss rate and delay of each connection, and judge the quality of each connection. Whether the status is idle, normal, congested or invalid, so as to provide decision-making basis for the traffic allocation and scheduling module, make full use of the effective bandwidth of each connection and achieve load balancing.
[0043] Because the terminal of the present invention is transparent to the transport layer, and uses the flow control transmission protocol RTCP as a statistical tool for streaming media data packet transmission information, the sending end uses the RTCP receiving report RR (Receiver Report) data packet fed back by the receiving end. Information statistics periodical throughput capability. The information of the RR data packet includes: the ratio of the RTP data packet lost by the sending end after sending the previous RR packet, the maximum sequence number of the data packet received by the sending end after sending the previous RR packet, and the number used to receive these data packets Time; the bandwidth quality of the connection is calculated according to the relationship proportional to the effective sending volume of the streaming media data packets and inversely proportional to the packet loss rate and transmission time.
[0044] Traffic scheduling allocation module: used to complete three functions: first, determine the priority of the size of the sending window of each connection according to the bandwidth quality of each connection provided by the QoS quality evaluation module; second, complete the selection of the traffic sending path, and also It is to select in real time which connection to send the streaming media data to according to the traffic allocation strategy; third, to send a certain proportion of redundant streaming media data packets on each available connection according to the set strategy.
[0045] Convergence module: responsible for receiving the streaming media data sent by the communication peer, and counting the sending volume, packet loss rate and delay of each connection streaming media data, and then reordering the data streams on each connection according to the global serial number , remove redundant streaming media data packets, and then submit them to the media playback module for its use. Because the terminal of the present invention adopts a double-layer serial number encoding scheme: each data packet has a global serial number and a local serial number respectively, and the global serial number is used for the media playback module to remove redundant data packets and reorder data packets and playback, the local serial number is for the confluence module to count the sending volume, packet loss rate, delay and other information of the data packets of each connection, and send the statistical information to the communication peer in the form of RTCP packets for the communication peer QoS The quality assessment module statistically calculates the quality of each connection, and then executes traffic allocation and scheduling decisions for the traffic allocation and scheduling module.
[0046] Data buffer module: including a global data buffer and multiple connection data buffers that store each connection separately. The global data buffer is used for data packet reordering and playback, and each connection data buffer is used for the confluence module to count the connection quality Information Use.
[0047] see figure 2 , introduce the specific operation steps of the data transmission method of the real-time streaming media transmission terminal of the present invention based on multi-connection technology:
[0048] Step 1. Before communication starts, the multi-connection real-time monitoring module detects its own network status, and sends the monitoring results to the multi-connection negotiation and establishment module for use in negotiation and connection establishment with the communication peer.
[0049] Step 2, the multi-connection negotiation and establishment module and the communication peer follow the standard session initiation process and request to establish a session with the peer, exchange their available network interface information, and establish multiple optional connections; then according to the settings including network conditions and user needs The policy selects appropriate multiple connections among the optional connections to establish a multi-connection session to transmit data. And during the communication process, the multi-connection real-time monitoring module notifies the multi-connection negotiation and establishment module to adjust the connection between the communication parties in real time according to the change of each connection: that is, to add and delete connections in the session in real time.
[0050] In step 2, taking the SIP multi-connection session as an example, the work of the multi-connection negotiation and establishment module is based on the extended message of the multimedia session description protocol SDP (Session Description Protocol). SDP is a Session Description Protocol that provides a multimedia session description for purposes such as session notification, session invitation, and other forms of multimedia session initiation. Its effective text carrier includes the media that constitute the session and information (address) about receiving the media, etc. By extending the SDP message, the mpip field and the mpport field added in the SDP message are used to respectively carry information about multiple network cards and multiple available ports of the terminal. Before the session initiator initiates a session, the multi-connection real-time monitoring module automatically obtains local network information and available port information, and when sending a session request, fills in the information of multiple network cards and multiple available ports into the SDP carried in the session establishment Invite request The mpip and mpport fields. After receiving the session establishment request, the receiving end parses the SDP message and constructs a reply message. There are four different situations:
[0051] (A) When both the sending end and the receiving end have multiple network cards, in the SDP message carried by the Invite request, the mpip and mpport fields respectively carry the IP addresses of multiple available networks and multiple available port information of the session initiator. After receiving the Invite request, the receiving end parses out the multiple available networks carried by mpip and the multiple available ports carried by mpport, and matches them with its own available networks one by one. Each pair of available networks will be based on the number of available ports Establish multiple connections, multiple networks and multiple connections established belong to the same session, and when replying with 200 OK, inform the session initiator of the matching result, and notify the session initiator to establish a connection with the same strategy, so that in a Multiple connections are established in a session.
[0052] (B) When the sending end has multiple access network cards and the receiving end has only one network card, in the SDP message carried by the Invite request, the mpip and mpport fields respectively carry the IP addresses of multiple available networks and multiple available port information of the session initiator . After receiving the Invite request, the receiving end parses out the multiple available networks carried by mpip and the multiple available ports carried by mpport. Since the receiving end detects that it has only one available network, this network of the receiving end will be connected to each network of the initiator. Each of the available networks establishes a connection. The above-mentioned multiple connections established belong to the same session, and when the 200 OK is replied, the session initiator is notified of the matching result, and the session initiator is notified to establish a connection with the same strategy, thereby establishing multiple connections in one session.
[0053](C) When the sending end has only one access network card and the receiving end has multiple access network cards, in the SDP message carried by the Invite request, the mpip and mpport fields respectively carry the IP address of the only available network of the session initiator and the multi-network address of the session initiator. Available port information. After receiving the Invite request, the receiving end parses out the only available network carried by mpip and the multiple available ports carried by mpport. At this time, each available network of the receiving end establishes a connection with the unique available network of the initiator. The above-mentioned multiple connections established belong to the same session, and when the 200 OK is replied, the session initiator is notified of the matching result, and the session initiator is notified to establish a connection with the same strategy, thereby establishing multiple connections in one session.
[0054] (D) When both the sending end and the receiving end have only one access network card, in the SDP message carried by the Invite request, the mpip and mpport fields respectively carry the IP address of the only available network of the session initiator and the information of multiple available ports of the session initiator. After receiving the Invite request, the receiving end parses out the only available network carried by mpip and the multiple available ports carried by mpport. At this time, the receiving end will establish multiple connections between the pair of networks according to the number of available ports. All connections belong to the same session, and when replying with 200 OK, the matching result is returned to the session initiator, and the session initiator is notified to establish a connection with the same strategy, thereby establishing multiple connections in one session.
[0055] Step 3, the QoS quality evaluation module carries real-time statistics and calculates the network status and throughput capacity of each connection according to the information carried by the RTCP data packet sent by the opposite end, including the data packet transmission volume, packet loss rate and time delay of each connection, and sets Determine the working status of each connection.
[0056] In step 3, the operation of the QoS quality evaluation module to judge the state of each connection includes the following contents: (see image 3 )
[0057] (31) Preparation stage: According to the principle that it is proportional to the effective sending volume of streaming media data packets and inversely proportional to the packet loss rate and transmission time, the present invention sets three connection quality thresholds: TH1, TH2, TH3, and correspondingly Set four different states of connection quality: idle, normal, congested and invalid; define less than TH1 as idle state, greater than TH1 but less than TH2 as normal state, greater than TH2 but less than TH3 as congested state, greater than TH3 is defined as a failure state.
[0058] (32) The QoS quality evaluation module is based on the RTCP data packet statistical data transmission status fed back periodically by the communication peer: extract information including the data transmission volume, packet loss rate, and delay of each connection from the feedback information, and calculate the data transmission rate of each connection. Quality: compare the calculated quality of each connection with the three set thresholds TH1, TH2, and TH3, and set the working status of each connection.
[0059] (33) Store and record the judgment results (see image 3 ).
[0060] Step 4: The traffic allocation and scheduling module initializes the size of the sending window of each connection according to the QoS quality evaluation result, and dynamically adjusts the size of the sending window of each connection according to the QoS quality evaluation result, and then divides the streaming media data according to the traffic scheduling strategy and redundancy strategy , to distribute the data to each connection for transmission.
[0061] The basis of traffic scheduling is the adjustment result of the sending window size, so that a frame of video data is allocated to each connection according to the ratio of the connection sending window size to all connections; the specific method of traffic scheduling is:
[0062] When new video stream data arrives, first determine whether the size of each sending window has been updated;
[0063] If there is no update, the corresponding data is divided according to the original proportion to each connection for transmission respectively;
[0064] If there is an update, the corresponding data is divided according to the new ratio and distributed to each connection for transmission.
[0065] In step 4, according to each connection status set by the QoS quality evaluation module, the traffic distribution scheduling module adjusts the adjustment operation of the sending window size of each connection including the following operations (see Figure 4 ):
[0066] In the initial stage of communication, because there is not enough information available to evaluate the connection QoS quality, the QoS quality evaluation module evaluates the initial state of each connection as an idle state at the beginning of the streaming media transmission stage, and the traffic scheduling and allocation module evaluates the connection QoS quality according to the connection QoS When initializing the traffic allocation strategy, the priority of each connection is set to the same priority, and the sending window of each connection is increased by imitating the slow start method of the Transmission Control Protocol TCP (Transmission Control Protocol);
[0067] If the connection status is still idle, gradually increase the sending window size;
[0068] If the connection status is normal, keep the transmission rate unchanged;
[0069] If the connection status is congested, theoretically, the sending window of the congested connection should be reduced to avoid further congestion; but in practical applications, because updating the traffic sending window immediately will cause excessive swings in the window size, it is chosen to set a The interval time for updating the window size. After the interval time arrives, if it is judged that the connection is still in a congested state, the sending window will be reduced accordingly;
[0070] If the connection state is invalid, stop sending the connection, and notify the multi-connection negotiation and control module to delete the invalid connection from the existing session.
[0071] Step 5, receive the RTCP packet sent by the peer, and then return to the operation of step 3.
[0072] The invention adopts a double-layer serial number coding scheme, and each data packet has a global serial number and a local serial number. In addition, the terminal not only has a global video data buffer area, each video sending and receiving thread also has a data buffer area, connects to the QoS evaluation module to count the packet loss status of each thread, and notifies the traffic distribution scheduling module to adjust each The sending and receiving ratio of the thread, etc.
[0073] Step 6. During the communication process, if the QoS quality evaluation module calculates that a certain connection of itself is in an invalid state, it notifies the multi-connection negotiation and establishment module to send a connection deletion request to the communication peer: request to remove the broken connection from the existing session Delete, and send this information to the communication peer.
[0074] Step 7, during the communication process, the multi-connection negotiation and establishment module receives a connection change request sent by the communication peer end because it detects a change in its own network status, and performs corresponding follow-up processing.
[0075] Step 8, during the communication process, the confluence module receives the streaming media data sent by the communication peer, and when reordering the data streams on each connection according to the global sequence number, removes redundant streaming media data packets, and then submits them to the media Playback module; at the same time, it counts the sending volume, packet loss rate and delay information of the streaming media data of each connection, and sends the statistical information to the communication peer in the form of RTCP packets.
[0076] The present invention has carried out many implementation tests, and the result of the test section is successful, and the purpose of the invention has been realized.
