A method for RTP stream multiplexing and packet loss retransmission in a network monitoring scenario

Abstract

The application provides a method for RTP stream multiplexing and packet loss retransmission in a network monitoring scene, comprising the following steps: RTP stream multiplexing and separation: a server aggregates multiple single-channel video streams to form an aggregated data stream and sends the aggregated data stream to a player client; when the server receives the multiple single-channel video streams, the server allocates an ID to each single-channel video stream; automatic request for retransmission object packet accumulation processing and packet loss detection; automatic request for retransmission object packet loss and retransmission processing; automatic request for retransmission object processing in the case of no packet loss; server processing of packet loss request: when the server receives a packet loss request from the client, the server extracts the missing packet sequence number contained in the request and searches for the corresponding packet in the cache. The application uses the packet loss retransmission technology to improve the integrity of the network video monitoring picture and the smoothness of the playback when the network fluctuates.

Description

Technical Field

[0001] This invention relates to the field of multimedia communication technology, and more particularly to a method for RTP stream multiplexing and packet loss retransmission in a network monitoring scenario. Background Technology

[0002] RTP, or Real-Time Transport Protocol, is a network transport protocol used to transmit audio and video over the Internet. It provides end-to-end transmission services with real-time characteristics, including information such as timestamps, sequence numbers, and payload formats to support real-time multimedia streaming. However, when using the UDP-based RTP protocol for video data streaming, there is no packet loss retransmission mechanism.

[0003] To address the issue of network packet loss, patent CN202011384070.3 employs a packet loss retransmission method. This patent determines whether packet loss has occurred by detecting whether the sequence numbers of the first RTP packet and the RTP packet corresponding to the currently playing audio / video data are consecutive. If packet loss exists, a packet loss request is sent to the server based on the sequence number of the detected lost RTP packet to be retransmitted. The method provided in this patent promptly triggers the lookup of the lost packet sequence number in the cache queue when reading RTP packets, thereby increasing the hit rate of the packet loss request within the server's window and thus ensuring a relatively high probability of obtaining the lost packet data, guaranteeing the quality of audio / video playback.

[0004] CN202111176877.2 achieves reliable video frame transmission through RTP header extension. It extends the RTP protocol itself by expanding the RTP header, ensuring compatibility with video devices. Furthermore, it employs packet loss retransmission based on the RTP header extension, achieving reliable video frame transmission with minimal bandwidth and a low network packet loss rate, effectively guaranteeing video quality.

[0005] Typically, one RTP session corresponds to only one media stream. Aggregating multiple RTP data streams into a single RTP data stream facilitates the transmission of multiple video feeds. Current patents introducing packet loss retransmission mechanisms for RTP treat an RTP stream as a single audio / video stream, without considering the scenario of a single RTP stream transmitting multiple media streams. If this single RTP data stream contains multiple audio / video streams, and the round-trip time (RTT) is too long, packet loss in one or more audio / video streams cannot be processed in a timely manner. In this situation, if real-time audio / video streaming is prioritized, it can easily lead to stuttering and choppy playback; conversely, if smooth playback is prioritized, it can introduce significant latency. Summary of the Invention

[0006] In view of this, the purpose of this invention is to propose a method for RTP stream multi-path aggregation and packet loss retransmission in network monitoring scenarios, so as to solve the technical problem of not being able to process packet loss in a timely manner when an RTP data stream contains multiple audio and video streams, resulting in stuttering.

[0007] The technical means employed in this invention are as follows:

[0008] A method for RTP stream multi-path aggregation and packet loss retransmission in a network monitoring scenario includes the following steps:

[0009] RTP stream aggregation and splitting: The server aggregates multiple individual video streams to form an aggregated data stream, which is then sent to the player client. When the server receives multiple individual video streams, it assigns an ID to each stream. The server inserts the ID of each individual video stream into a data packet and sends it to the player client. The player client splits the aggregated data stream based on the ID of each data packet. After receiving the data packet, the player client obtains its ID to determine the video stream it belongs to. If it is a new data stream, the client creates an automatic retransmission request object and passes the data packet to that object. If it is an existing data stream, the client passes the data packet to the corresponding object.

[0010] Automatic retransmission object's packet accumulation processing and packet loss detection: After receiving a new data packet, the automatic retransmission object determines whether there is packet loss or packet accumulation in the current buffer queue. If there is packet accumulation, the extra data packets are added to the bitstream recovery queue and deleted from the buffer queue. While removing the extra data packets from the buffer queue, it checks whether there is packet loss between the data packets. If packet loss occurs, the sequence number of the lost packet is recorded and sent to the server. When packet loss is detected or there is no packet accumulation, the packet loss and retransmission processing steps of the automatic retransmission object are executed.

[0011] Automatic retransmission request handling for packet loss and retransmission: Check if the received data packet is a retransmission data packet from the server. If it is a retransmission data packet, insert the retransmission data packet into the corresponding position in the buffer queue and remove it from the packet loss sequence number queue. If it is not a retransmission data packet and the waiting time has not expired, continue to wait for the next data packet. If the timeout occurs, stop waiting for retransmission, directly send the first data packet in the buffer queue to the bitstream recovery queue, and clear the corresponding packet loss sequence number.

[0012] Handling of Automatic Retransmission Objects in Case of No Packet Loss: If there is no packet loss in the current data stream, add the data packets to the cache queue of the Automatic Retransmission Object and sort them to prevent misjudgment due to out-of-order data packets; when the number of data packets in the cache queue is greater than or equal to the predetermined minimum value, they will be processed in the data packet accumulation and packet loss detection steps of the Automatic Retransmission Object in the next loop.

[0013] Server handling of packet loss requests: When the server receives a packet loss request from the client, the server extracts the sequence number of the lost data packet contained in the request and looks up the corresponding data packet in the cache; if the corresponding data packet is found, the server marks the corresponding data packet as a retransmission packet and sends it to the client.

[0014] Furthermore, the specific structure of the RTP data packet is as follows:

[0015] Version number V indicates the protocol version; padding bit P indicates whether there is padding at the end of the packet; extension bit X indicates whether an extended header is included; contributor source count CC indicates the number of CSRC fields; marker bit M is used to mark special packets; payload type PT indicates the type of payload carried in the packet; sequence number identifies the order of the packets; timestamp is used to identify the sampling time of the media data; synchronization source identifier SSRC uniquely identifies the source of the data stream in the same session; contributor source identifier CSRC is used to identify the contributor source of the media stream generated by mixing or synthesis; custom field data stream identifier ID, extended to distinguish different data streams; payload indicates audio and video data; padding is used to align packets to a specific block size.

[0016] Furthermore, the multiplexing of RTP streams specifically includes the following steps:

[0017] S11. Wait to receive RTP data packets. After receiving the data packets, execute S12.

[0018] S12. Obtain the SSRC of the RTP packet, determine whether it is a new RTP session, if yes, execute S13, otherwise execute S14;

[0019] S13. Find the ID corresponding to the SSRC of this RTP session and execute S15;

[0020] S14. Assign an ID to the new data stream and ensure that this ID does not conflict with other IDs. Execute S15.

[0021] S15. Divide the 32-bit unsigned integer ID into four 8-bit unsigned numbers from high to low, write them into the first four bytes of the send buffer array, and execute S16.

[0022] S16. Copy the data from the received data packet to the fourth byte of the send buffer array, and then execute S17 starting from the fifth byte.

[0023] S17. Set the timestamp and flag of the data packet to be sent to be the same as those of the received data packet, send the data packet to the player client, and execute S11.

[0024] Furthermore, the automatic retransmission request object is responsible for packet loss detection and retransmission request. The automatic retransmission request object includes a data packet buffer queue, a packet loss sequence number queue, and a stream recovery queue. The data packet buffer queue is used to store data packets. The packet loss sequence number queue is used to record the sequence number of lost data packets. The stream recovery queue is used to restore fragmented H.265 data into the original H.265 stream.

[0025] Furthermore, the multiplexing of RTP streams includes the following steps:

[0026] S110. Initialize RTP session parameters and establish an RTP session;

[0027] S120, Waiting to receive data packets from the server;

[0028] S130. When a data packet is received, read the ID of the data stream to which the data packet belongs, and determine whether it is a new data stream. If it is a new data stream, execute S140; otherwise, execute S150.

[0029] S140. Based on the ID of the data packet, create a corresponding automatic request return object for this data stream to detect whether the data stream has lost packets and to request retransmission from the server when packet loss occurs.

[0030] S150. Pass the data packet to the automatic request return object, so that the object can detect whether packet loss has occurred and implement the packet loss retransmission function.

[0031] Furthermore, the data packet accumulation processing and packet loss detection of the automatic retransmission request object, the packet loss and retransmission processing of the automatic retransmission request object, and the processing of the automatic retransmission request object in the case of no packet loss specifically include the following steps:

[0032] S160. Determine whether the packet loss sequence number queue of the automatic request return object is empty and whether the cache queue is greater than or equal to the minimum value; if the data stream corresponding to this object has not experienced packet loss and the data packets in the cache queue are greater than the minimum value, execute S170; otherwise, if there is not enough data in the cache queue or packet loss has occurred in the current data stream, execute S180.

[0033] S170. Determine whether the sequence numbers of the first two data packets in the buffer queue are consecutive. If they are consecutive, it means that no packet loss has occurred between the two data packets. Execute S171. Otherwise, it means that packet loss has occurred. Execute S172.

[0034] S171. Add the first data packet to the video stream recovery queue and clear it from the buffer queue to restore the original video data for decoding;

[0035] S172. Record the sequence number of the missing data packet in the two data packets and add the sequence number of the missing data packet to the lost sequence number queue;

[0036] S173. Send the missing data packet sequence number to the server, execute S120, and continue receiving data packets;

[0037] S180. Determine if the packet loss sequence number queue of the current data stream is empty. If it is empty, it means that no packet loss has been found in the current data stream. Execute S190; otherwise, execute S200.

[0038] S190. No packet loss was detected in the current data stream. The data packets were directly added to the cache queue of the automatic request return object and sorted.

[0039] S200: Determine whether the received data packet is a retransmitted data packet. If yes, execute S201; otherwise, execute S210.

[0040] S201. Add the retransmitted data packet to the corresponding position in the buffer queue and clear the sequence number of the data packet from the lost packet sequence number queue;

[0041] S210. If the received data packet is not a retransmitted data packet, then add the data packet to the buffer queue;

[0042] S220. Determine whether the timeout for waiting to retransmit the data packet has expired. The timeout period is set according to the parameters of the video stream. If there is no timeout, execute S120 and continue to wait to receive the data packet; otherwise, execute S230.

[0043] S230. Add the first data packet in the buffer queue to the stream recovery queue and clear the lost packet sequence number queue.

[0044] Furthermore, the specific steps for the server to handle packet loss requests include:

[0045] Establish a TCP connection with the player client program;

[0046] Waiting to receive a TCP message containing the packet loss sequence number;

[0047] Search for this data packet in the buffer queue. If it is found, proceed to the next step; otherwise, proceed to the previous step.

[0048] All found data packets are sent to the client again, with the data packet type PT set to 103 to indicate that the data packets are being resent.

[0049] The present invention also provides a storage medium comprising a stored program, wherein, when the program is executed, it performs the method for RTP stream multiplexing and packet loss retransmission under any of the above-mentioned network monitoring scenarios.

[0050] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the method for RTP stream multiplexing and packet loss retransmission under any of the above-described network monitoring scenarios through the computer program.

[0051] Compared with the prior art, the present invention has the following advantages:

[0052] This invention improves the integrity and smoothness of network video surveillance footage during network fluctuations by using packet loss retransmission technology.

[0053] This invention facilitates the forwarding and processing of multiple video streams by aggregating them into a single stream.

[0054] This invention decomposes the aggregated video stream and performs packet loss retransmission, which facilitates the management of the packet loss retransmission waiting time for each stream, is compatible with video streams of different frame rates, and avoids mutual interference between video streams when the network fluctuates. Attached Figure Description

[0055] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0056] Figure 1 This is a flowchart of the RTP aggregation stream decomposition, packet loss detection, and retransmission request of the present invention.

[0057] Figure 2 This is a flowchart of the RTP stream aggregation process of the present invention.

[0058] Figure 3 This is a flowchart of the server responding to a packet loss request according to the present invention.

[0059] Figure 4 This is the RTP data packet header extension structure of the present invention. Detailed Implementation

[0060] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0061] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0062] This invention relates to a network video surveillance system, which consists of three parts: a video transmitter, a server, and a player client. There can be multiple video transmitters, which send video stream data to the server. The server program receives this data and forwards it to the player client. This patent pertains to network communication between the server and the player client.

[0063] like Figure 1-3 As shown, this invention provides a method for RTP stream multi-path aggregation and packet loss retransmission in a network monitoring scenario, comprising the following steps:

[0064] (1) Multiplexing and decoupling of RTP streams

[0065] The aggregation task is executed by the server program. When the server receives multiple single video streams, it assigns an ID to each stream. The server inserts the corresponding ID into the data packet and sends it to the player client. The client separates the aggregated data streams based on the ID of each data packet. After receiving a data packet, the client first obtains its ID to determine the video stream it belongs to. If it is a new data stream, the client creates an automatic retransmission request object and passes the data packet to that object. If it is an existing data stream, the client passes the data packet to the corresponding object.

[0066] (2) Data packet accumulation processing and packet loss detection for objects that automatically request retransmission

[0067] Upon receiving a new data packet, the automatic retransmission object first checks if there is packet loss or packet accumulation in the current buffer queue. If packet accumulation exists, the extra packets are added to the stream recovery queue and removed from the buffer queue. While removing the extra packets from the buffer queue, it also checks for packet loss. If packet loss occurs, the sequence number of the lost packet is recorded and sent to the server. If packet loss is detected or there is no packet accumulation, the next step is to process the received data packet.

[0068] (3) Handling packet loss and retransmission of objects that automatically request retransmission

[0069] The program first determines if there is packet loss in the current data stream. If there is packet loss, the program checks if the received data packet is a retransmission data packet from the server. If it is a retransmission data packet, the program inserts it into the corresponding position in the buffer queue and removes it from the packet loss sequence number queue; if not, and the waiting time has not expired, it continues to wait for the next data packet; if the timeout occurs, the program will no longer wait for retransmission, directly send the first data packet in the buffer queue to the bitstream recovery queue, and clear the corresponding packet loss sequence number.

[0070] (4) Handling of automatic retransmission requests in the absence of packet loss

[0071] If no packet loss occurs in the current data stream, the program will add the data packets to the cache queue of the automatic retransmission object and sort them to prevent misjudgment due to out-of-order data packets. When the number of data packets in the cache queue is greater than or equal to the predetermined minimum value, they will be processed in step (2) in the next loop.

[0072] (5) Server handles packet loss requests

[0073] When the server receives a packet loss request from a client, it extracts the sequence number of the lost data packet from the request and searches for the corresponding data packet in its cache. If the corresponding data packet is found, the server marks it as a retransmission packet and sends it to the client.

[0074] Figure 4The diagram shows the RTP data packet structure provided in this embodiment of the invention, including: Version Number (V, 2 bits), representing the protocol version; Padding Bit (P, 1 bit), indicating whether there is padding at the end of the data packet; Extension Bit (X, 1 bit), indicating whether an extension header is included; Contributor Count (CC, 4 bits), representing the number of CSRC fields; Mark Bit (M, 1 bit), used to mark special data packets; Payload Type (PT, 7 bits), indicating the type of payload carried in the data packet; Sequence Number (16 bits), identifying the order of the data packets; Timestamp (32 bits), used to identify the sampling time of the media data; Synchronization Source Identifier (SSRC, 32 bits), uniquely identifying the source of the data stream in the same session; Contributor Source Identifier (CSRC, 32 bits / each, the number is specified by CC), used to identify the contributor source of the media stream generated by mixing or synthesis; Custom Field Data Stream Identifier (ID, 32 bits), extended to distinguish different data streams; Payload, representing audio and video data; Padding, used to align the data packet to a specific block size.

[0075] This invention adds a 32-bit ID before the payload to distinguish which data stream a packet belongs to, based on the original RTP packet structure. Different PT values ​​are used to distinguish between normal packets and retransmitted packets; 98 represents H.265 encoded video stream packets, and 103 represents retransmitted packets. This invention uses the consecutive sequence numbers of adjacent packets to determine if packet loss has occurred.

[0076] Figure 2 The diagram shown is an RTP stream aggregation flowchart provided by an embodiment of the present invention. The key steps in the flowchart will be described in detail below:

[0077] S1: Wait to receive RTP data packets, and execute S2 after receiving the data packets;

[0078] S2: Obtain the SSRC of the RTP packet, determine whether it is a new RTP session, if so, execute S3, otherwise execute S4;

[0079] S3: Find the ID corresponding to the SSRC of this RTP session, and execute S5;

[0080] S4: Assign an ID to the new data stream and ensure that this ID does not conflict with other IDs, then execute S5;

[0081] S5: Divide the 32-bit unsigned integer ID into four 8-bit unsigned numbers from high to low, write them into the first four bytes of the send buffer array, and then execute S6;

[0082] S6: Copy the data from the received data packet to the fourth byte of the send buffer array, that is, starting from the fifth byte, and then execute S7;

[0083] S7: Set the timestamp and flag bits of the data packet to be sent to be the same as those of the received data packet, send the data packet to the player client, and execute S1;

[0084] In the player client program, an automatic request return object is defined to detect whether packet loss occurs in each of the decomposed data streams. If packet loss occurs, a retransmission request with the sequence number of the lost data packets is automatically sent to the server via the TCP protocol.

[0085] The Automatic Repeat Request (ATR) object is responsible for packet loss detection and retransmission requests. Internally, it contains a packet buffer queue, a lost packet sequence number queue, and a stream recovery queue. The packet buffer queue stores data packets for easy packet loss detection and to prevent out-of-order delivery; the lost packet sequence number queue records the sequence numbers of lost data packets; and the stream recovery queue restores fragmented H.265 data into the original H.265 stream.

[0086] Figure 1 The diagram shows a flowchart of the RTP aggregation stream decomposition, packet loss detection, and retransmission request implementation of this invention. The key steps in the flowchart are explained in detail below:

[0087] S110: Initialize RTP session parameters and establish an RTP session;

[0088] S120: Waiting to receive data packets from the server;

[0089] S130: When a data packet is received, read the ID of the data stream to which the data packet belongs, and determine whether it is a new data stream. If it is a new data stream, execute S140; otherwise, execute S150.

[0090] S140: Based on the data packet ID, create a corresponding automatic request return object for this data stream to detect whether packet loss has occurred in this data stream, and to request retransmission from the server when packet loss occurs;

[0091] S150: Pass the data packet to the automatic request return object, so that the object can detect whether packet loss has occurred and implement the packet loss retransmission function;

[0092] S160: Determine if the packet loss sequence number queue of the automatically requested return object is empty and if the cache queue is greater than or equal to the minimum value. If no packet loss has occurred in the data stream corresponding to this object, and the number of data packets in the cache queue is greater than the minimum value, it indicates that there is data accumulation in the current data stream. In this case, the excess data should be processed, and S170 should be executed. Otherwise, either the data in the cache queue is insufficient or packet loss has occurred in the current data stream, and S180 should be executed.

[0093] S170: Determine whether the sequence numbers of the first two data packets in the buffer queue are consecutive. If they are consecutive, it means that no packet loss has occurred between the two data packets, and execute S171. Otherwise, it means that packet loss has occurred, and execute S172.

[0094] S171: Add the first data packet to the video stream recovery queue and clear it from the buffer queue to restore the original video data for decoding;

[0095] S172: Record the sequence number of the missing data packet in the two data packets and add the sequence number of the missing data packet to the lost sequence number queue;

[0096] S173: Send the missing data packet sequence number to the server, execute S120, and continue receiving data packets;

[0097] S180: Determine if the packet loss sequence number queue of the current data stream is empty. If it is empty, it means that no packet loss has been found in the current data stream. Execute S190; otherwise, execute S200.

[0098] S190: No packet loss was detected in the current data stream. The data packets were directly added to the cache queue of the automatic request return object and sorted.

[0099] S200: Determine whether the received data packet is a retransmitted data packet. If yes, execute S201; otherwise, execute S210.

[0100] S201: Add the retransmitted data packet to the corresponding position in the buffer queue and clear the sequence number of the data packet from the lost sequence number queue;

[0101] S210: If the received data packet is not a retransmitted data packet, then add the data packet to the buffer queue;

[0102] S220: Determine whether the timeout period for waiting to retransmit data packets has expired. The timeout period is set according to the parameters of the video stream. This invention uses a timeout period to ensure smooth video playback. That is, the reciprocal of the video frame rate. If there is no timeout, execute S120 and continue to wait for receiving data packets; otherwise, execute S230.

[0103] S230: The retransmitted data packet may have been lost again or the retransmitted data packet has not yet arrived, causing the waiting time to expire. We cannot continue to wait for the retransmitted data packet. Add the first data packet in the buffer queue to the bitstream recovery queue and clear the lost packet sequence number queue.

[0104] Figure 3 This is a flowchart of a server responding to a packet loss request according to an embodiment of the present invention. The key steps in the flowchart will be described in detail below:

[0105] Establish TCP connection with the player client program

[0106] Waiting to receive a TCP message containing the packet loss sequence number;

[0107] Search for this data packet in the buffer queue. If it is found, proceed to the next step; otherwise, proceed to the previous step.

[0108] All found data packets are sent to the client again, with the data packet type PT set to 103 to indicate that the data packets are being resent.

[0109] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for RTP stream multi-path aggregation and packet loss retransmission in a network monitoring scenario, characterized in that, Includes the following steps: RTP stream aggregation and splitting: The server aggregates multiple single video streams to form an aggregated data stream and sends it to the player client; when the server receives multiple single video streams, the server assigns an ID to each single video stream; the server inserts the ID of each single video stream into the data packet and sends it to the player client. The player client separates the aggregated data stream based on the ID of each data packet; After receiving a data packet, the player client obtains its ID to determine the video stream to which it belongs. If it is a new data stream, the client will create an automatic retransmission request object and pass the data packet to that object. If it is an existing data stream, the data packet will be passed to the corresponding object. Automatic retransmission request object's packet accumulation processing and packet loss detection: After receiving a new data packet, the automatic retransmission request object determines whether there is packet loss or packet accumulation in the current buffer queue. If there is packet accumulation, the extra data packets are added to the bitstream recovery queue and deleted from the buffer queue. While removing the extra data packets from the buffer queue, it checks whether there is packet loss between the data packets. If packet loss occurs, the sequence number of the lost packet is recorded and sent to the server. When packet loss is detected or there is no packet accumulation, the packet loss and retransmission processing steps of the automatic retransmission request object are executed. Automatic retransmission request handling for packet loss and retransmission: Check if the received data packet is a retransmission data packet from the server. If it is a retransmission data packet, insert the retransmission data packet into the corresponding position in the buffer queue and remove it from the packet loss sequence number queue. If it is not a retransmission data packet and the waiting time has not expired, continue to wait for the next data packet. If the timeout occurs, stop waiting for retransmission, directly send the first data packet in the buffer queue to the bitstream recovery queue, and clear the corresponding packet loss sequence number. Handling of Automatic Retransmission Objects in Case of No Packet Loss: If there is no packet loss in the current data stream, add the data packets to the cache queue of the Automatic Retransmission Object and sort them to prevent misjudgment due to out-of-order data packets; when the number of data packets in the cache queue is greater than or equal to the predetermined minimum value, they will be processed in the data packet accumulation and packet loss detection steps of the Automatic Retransmission Object in the next loop. Server handling of packet loss requests: When the server receives a packet loss request from the client, the server extracts the sequence number of the lost data packet contained in the request and looks up the corresponding data packet in the cache; if the corresponding data packet is found, the server marks the corresponding data packet as a retransmission packet and sends it to the client.

2. The method for RTP stream multi-path aggregation and packet loss retransmission in a network monitoring scenario according to claim 1, characterized in that, The specific structure of the data packet includes: Version number V indicates the protocol version; padding bit P indicates whether there is padding at the end of the packet; extension bit X indicates whether an extended header is included; contributor source count CC indicates the number of CSRC fields; marker bit M is used to mark special packets; payload type PT indicates the type of payload carried in the packet; sequence number identifies the order of the packets; timestamp is used to identify the sampling time of the media data; synchronization source identifier SSRC uniquely identifies the source of the data stream in the same session; contributor source identifier CSRC is used to identify the contributor source of the media stream generated by mixing or synthesis; custom field data stream identifier ID, extended to distinguish different data streams; payload indicates audio and video data; padding is used to align packets to a specific block size.

3. The method for RTP stream multi-path aggregation and packet loss retransmission in a network monitoring scenario according to claim 2, characterized in that, RTP stream multi-path aggregation specifically includes the following steps: S11. Wait to receive RTP data packets. After receiving the data packets, execute S12. S12. Obtain the SSRC of the RTP packet, determine whether it is a new RTP session, if yes, execute S13, otherwise execute S14; S13. Find the ID corresponding to the SSRC of this RTP session and execute S15; S14. Assign an ID to the new data stream and ensure that this ID does not conflict with other IDs. Execute S15. S15. Divide the 32-bit unsigned integer ID into four 8-bit unsigned numbers from high to low, write them into the first four bytes of the send buffer array, and execute S16. S16. Copy the data from the received data packet to the fourth byte of the send buffer array, and then execute S17 starting from the fifth byte. S17. Set the timestamp and flag of the data packet to be sent to be the same as those of the received data packet, send the data packet to the player client, and execute S11.

4. The method for RTP stream multi-path aggregation and packet loss retransmission in a network monitoring scenario according to claim 1, characterized in that, The automatic retransmission request object is responsible for packet loss detection and retransmission requests. The automatic retransmission request object includes a data packet buffer queue, a packet loss sequence number queue, and a bitstream recovery queue. The data packet buffer queue is used to store data packets. The packet loss sequence number queue is used to record the sequence number of lost data packets. The bitstream recovery queue is used to restore fragmented H.265 data into the original H.265 bitstream.

5. The method for RTP stream multi-path aggregation and packet loss retransmission in a network monitoring scenario according to claim 1, characterized in that, RTP stream demultiplexing includes the following steps: S110. Initialize RTP session parameters and establish an RTP session; S120, Waiting to receive data packets from the server; S130. When a data packet is received, read the ID of the data stream to which the data packet belongs, and determine whether it is a new data stream. If it is a new data stream, execute S140; otherwise, execute S150. S140. Based on the ID of the data packet, create a corresponding automatic request return object for this data stream to detect whether the data stream has lost packets and to request retransmission from the server when packet loss occurs. S150. Pass the data packet to the automatic request return object, so that the object can detect whether packet loss has occurred and implement the packet loss retransmission function.

6. The method for RTP stream multi-path aggregation and packet loss retransmission in a network monitoring scenario according to claim 5, characterized in that, The data packet accumulation processing and packet loss detection for automatically requesting retransmission objects, the packet loss and retransmission processing for automatically requesting retransmission objects, and the processing of automatically requesting retransmission objects in the absence of packet loss specifically include the following steps: S160. Determine whether the packet loss sequence number queue of the automatic request return object is empty and whether the cache queue is greater than or equal to the minimum value; if the data stream corresponding to this object has not experienced packet loss and the data packets in the cache queue are greater than the minimum value, execute S170; otherwise, if there is not enough data in the cache queue or packet loss has occurred in the current data stream, execute S180. S170. Determine whether the sequence numbers of the first two data packets in the buffer queue are consecutive. If they are consecutive, it means that no packet loss has occurred between the two data packets. Execute S171. Otherwise, it means that packet loss has occurred. Execute S172. S171. Add the first data packet to the video stream recovery queue and clear it from the buffer queue to restore the original video data for decoding; S172. Record the sequence number of the missing data packet in the two data packets and add the sequence number of the missing data packet to the lost sequence number queue; S173. Send the missing data packet sequence number to the server, execute S120, and continue receiving data packets; S180. Determine if the packet loss sequence number queue of the current data stream is empty. If it is empty, it means that no packet loss has been found in the current data stream. Execute S190; otherwise, execute S200. S190. No packet loss was found in the current data stream. The data packets were directly added to the cache queue of the automatic request return object and sorted. S200. Determine whether the received data packet is a retransmitted data packet. If yes, execute S201; otherwise, execute S210. S201. Add the retransmitted data packet to the corresponding position in the buffer queue, and clear the sequence number of the data packet from the lost packet sequence number queue; S210. If the received data packet is not a retransmitted data packet, then add the data packet to the buffer queue; S220: Determine whether the timeout for waiting to retransmit the data packet has expired. The timeout period is set according to the parameters of the video stream. If there is no timeout, execute S120 and continue to wait to receive the data packet; otherwise, execute S230. S230. Add the first data packet in the buffer queue to the stream recovery queue and clear the lost packet sequence number queue.

7. The method for RTP stream multi-path aggregation and packet loss retransmission in a network monitoring scenario according to claim 1, characterized in that, The specific steps for a server to handle a lost packet request include: Establish a TCP connection with the player client program; Waiting to receive a TCP message containing the packet loss sequence number; Search for this data packet in the buffer queue. If it is found, proceed to the next step; otherwise, proceed to the previous step. All found data packets are sent to the client again, with the payload type PT set to 103 to represent retransmitted data packets.

8. A storage medium, characterized in that, The storage medium includes a stored program, wherein when the program is executed, it performs the method for RTP stream multiplexing and packet loss retransmission in a network monitoring scenario as described in any one of claims 1 to 7.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, The processor executes the method for RTP stream aggregation and packet loss retransmission in a network monitoring scenario as described in any one of claims 1 to 7 through the computer program.

Images