Weak network transmission optimization method and system based on video monitoring networking protocol

Abstract

The invention discloses a weak network transmission optimization method and system for a video monitoring networking protocol, and the method comprises the steps: carrying out the registration and communication of each entity device with an access server through G1, expanding an SDP in a national standard pull stream message through the server, adding an FEC data type, carrying out the interaction between a browser end and the access server through HTTP1, and transmitting a pull stream instruction, the access server notifies the media server to operate through HTTP2, the equipment collects, encodes and sends video data according to an instruction, the data is transmitted to the media server through S1 and S2, the processed data is forwarded to the browser to be played through S3, the system supports FEC error correction and NACK retransmission, the media server can repair data missing, and after watching is finished, all parts stop related operation according to the instruction. The optimized GB28181 video is transmitted through a UDP (User Datagram Protocol), under the condition that the network packet loss rate is 20%, a small amount of time delay is sacrificed, the video can be smoothly played without a blurred screen, and the media server can also effectively manage and process the received data, for example, timeout data is timely cleaned, missing packet sequences are repaired and the like.

Description

Technical Field

[0001] The present invention relates to the field of weak network transmission, and in particular to a weak network transmission optimization system and method based on GB28181 protocol. Background Art

[0002] With the development of the Internet and the country's increased investment in security and other aspects, communication products based on the GB28181 protocol have been widely used. Security cameras and surveillance cameras can be seen everywhere, and the staff of public security, procuratorial, judicial, health, railway and other departments basically wear GB28181 law enforcement recorders.

[0003] In the prior art, the main purpose of current communication products based on GB28181 is basically real-time audio and video transmission, with a large amount of data transmission and high requirements for network quality. At present, the RTP real-time transmission method is mainly used for video transmission. The traditional RTP / RTCP method can provide good flow and congestion control, but it does not handle packet loss in weak network conditions well, and it is easy to cause screen distortion, mosaics and other phenomena, affecting the viewing experience. Although the TCP / RTP / AVP method can be used to solve the problems of screen distortion, mosaics and other problems caused by packet loss, it will increase the delay and may cause video freezes.

[0004] Therefore, there is an urgent need for a transmission solution based on GB28181 that is optimized for weak networks, balances the problems of packet loss and delay, and contributes to current security work. Summary of the invention

[0005] The purpose of the present invention is to provide a weak network transmission optimization system and method based on the GB28181 protocol to solve the problem that (1) packet loss is not handled well in a weak network, which easily leads to a distorted screen; and (2) the TCP / RTP / AVP method is used to solve the packet loss problem, which increases the delay and causes the video to freeze.

[0006] To achieve the above object, the present invention provides the following technical solutions:

[0007] A weak network transmission optimization method based on a video surveillance networking protocol is characterized in that: each entity device registers with an access server through G1 and communicates using a video surveillance networking protocol, the access server extends the SDP in a national standard stream pulling message and adds an FEC data type, the browser interacts with the access server through HTTP1, sends a stream pulling instruction, the access server notifies the media server of the operation through HTTP2, the entity device collects, encodes and sends video data according to the instruction, the data is transmitted to the media server through S1 and S2, and is forwarded to the browser for playback through S3 after processing; the G1 is a national standard signaling stream, S1, S2, and S3 are all RTP media streams, S1 and S2 are transmitted using UDP, and S3 is transmitted using TCP.

[0008] A weak network transmission optimization method based on a video surveillance networking protocol specifically includes the following steps:

[0009] S1: Each physical device registers with the access server through G1, and each physical device and the access server use the video surveillance networking protocol to communicate with each other;

[0010] S2: The access server receives the physical device registration message through G1, searches the physical device information from the database and verifies the registration parameters, and returns the result to the physical device through G1;

[0011] S3: The browser communicates with the access server through HTTP1. After the connection is successful, the browser will first query the currently registered devices through HTTP1 and display them. For the subsequently registered devices, the access server will report them to the browser through HTTP1 for display;

[0012] S4: The browser sends the streaming instruction to the access server via HTTP1;

[0013] S5: The access server notifies the media server to open the data transmission and reception ports through HTTP2, and sends the media server receiving stream address and media parameters to the physical device;

[0014] S6: The physical device receives the stream pull message and replies the media parameters to the access server;

[0015] S7: After receiving the reply from the physical device, the access server sends the streaming address of the media server to the browser via HTTP1, and then sends an "ACK" message to the physical device, indicating that the physical device starts streaming;

[0016] S8: After receiving the media server streaming address replied by the access server, the browser actively connects to the media server streaming address and starts receiving and playing the stream;

[0017] S9: After the physical device receives the "ACK" message sent by the server, the physical device starts to collect camera video at the preset bit rate and frame rate, and encodes and puts it into the video frame queue;

[0018] S10: The physical device packages the video frames in the video frame queue into RTP data packets, generates FEC data packets, and puts them into the sending queue and the FEC data queue respectively;

[0019] S11: The physical device sends the original video RTP packets in the sending queue to the media server through S1, and adds the sent RTP packets to the sent queue, saves them for a period of time, and regularly clears the timeout data;

[0020] S12: The physical device sends the FEC data in the FEC data queue to the media server through S2;

[0021] S13: The media server puts the original video RTP and FEC data received from S1 and S2 into the RTP data queue and FEC data queue respectively;

[0022] S14: the media server now forwards the waiting timeout data and the data with complete spliced ​​frames at the head of the RTP data queue to the browser through S3;

[0023] S15: The browser sends the media data received from S3 to the player for decoding and playback.

[0024] The method for optimizing weak network transmission based on a video surveillance networking protocol comprises the following steps, after step S15:

[0025] S16: The media server finds the missing packet sequence in the current RTP data queue, tries to repair it through the FEC data queue, and inserts the repaired data into the RTP data queue according to the packet sequence; if the missing sequence is too large or the FEC data itself is missing, resulting in a repair failure, a NACK signal is sent to the physical device through S1 to notify the physical device to resend a copy of the missing original data;

[0026] S17: After receiving the NACK signal sent by the media server, the physical device finds the data to be resent from the sent queue and resends it to the media server through S1.

[0027] The method for optimizing weak network transmission based on the video surveillance networking protocol further includes the following steps:

[0028] S1: After watching, the browser sends an end instruction to the access server;

[0029] S2: After receiving the end instruction, the access server forwards it to the physical device and notifies the media server to close the receiving and forwarding ports through HTTP2;

[0030] S3: The physical device receives the end instruction, exits the streaming state, and stops streaming.

[0031] The physical device mentioned refers to the law enforcement recorder device.

[0032] The G1 is a national standard signaling stream, S1, S2, and S3 are all RTP media streams, S1 and S2 are transmitted using UDP, and S3 is transmitted using TCP.

[0033] The DB1 and DB2 are database data streams, and HTTP1 and HTTP2 are HTTP protocol data streams.

[0034] The described weak network transmission optimization method based on the video surveillance networking protocol, the access server expands the SDP (Session Description Protocol) in the national standard streaming message "INVITE", and the rtpmap (RTP parameter mapping table) adds the flexfec / 90000 type to the original video parameters PS / 90000, H264 / 90000, MPEG4 / 90000, and H265 / 90000, indicating FEC (Forward Error Correction) data with a clock frequency of 90000.

[0035] The method for optimizing weak network transmission based on a video surveillance networking protocol, the media server receives, repairs, records and forwards media data; the browser provides a human-computer interaction interface and a platform management window for adding, deleting, modifying and checking device information; and the database stores terminal device information.

[0036] The method for optimizing weak network transmission based on a video surveillance networking protocol, the device extends the GB28181 protocol SDP, and newly adds NACK and FEC in the RTP stream.

[0037] A weak network transmission optimization system based on a video surveillance networking protocol runs the weak network transmission optimization method based on a video surveillance networking protocol to optimize weak network transmission.

[0038] In order to solve the packet loss problem caused by media transmission from the device to the media server, since the actual network packet loss is uncontrollable, software simulation is needed during testing to simulate packet loss. Microsoft provided a network simulation tool (Network Emulator Toolkit) in the early days. First, find a computer that can connect to the server and has sufficient bandwidth, then configure the simulation tool parameters on this computer and set the random packet loss rate to 15%. Finally, let this computer provide a hotspot for the device to connect, and the packet loss rate between the device and the media server can be stabilized at 15±1%;

[0039] After comparative testing, in the case of 15% packet loss rate:

[0040] (1) In the traditional RTP / RTCP mode, the picture cannot be decoded due to packet loss;

[0041] (2) In TCP / RTP / RTCP mode, the video can be played, but the initial delay is as high as 7 seconds, which increases over time and causes serious lag;

[0042] (3) With the design of the present invention, the picture is smooth and the delay is only 2 seconds. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 It is a schematic diagram of a weak network transmission optimization system based on a video surveillance networking protocol in an embodiment of the present invention.

[0044] Figure 2 It is a flow chart of a weak network transmission optimization method based on a video surveillance networking protocol in an embodiment of the present invention. DETAILED DESCRIPTION

[0045] See also Figure 1 .

[0046] A weak network transmission optimization system based on a video surveillance networking protocol, comprising:

[0047] a. Access the server and perform GB28181 session management;

[0048] b. Media server, which forwards and records media data; connects with access server, physical device and browser for signal transmission;

[0049] c. The browser interacts with the access server and pulls the stream;

[0050] d. Database, storing terminal information and recorded file information; connected with access server and media server for signal transmission;

[0051] e. Physical equipment, supports video surveillance networking protocol, is connected to the access server signal transmission, and supports FEC (Forward Error Correction) and NACK (Negative Acknowledgment) retransmission functions.

[0052] G1 is the national standard signaling stream, S1, S2, and S3 are all RTP media streams, S1 and S2 use UDP transmission, and S3 uses TCP transmission.

[0053] DB1 and DB2 are database data streams, and HTTP1 and HTTP2 are HTTP protocol data streams.

[0054] The system communicates based on GB28181 protocol.

[0055] A weak network transmission optimization method based on a video surveillance networking protocol comprises the following steps:

[0056] S1: When the system is running, each GB28181 physical device registers to the GB28181 access server through G1 and performs corresponding message communication.

[0057] S2: The GB28181 access server receives the GB28181 entity registration message through G1, searches for entity device information from the database (DB1) and verifies registration parameters, and returns the result to the entity device through G1.

[0058] S3: The browser communicates with the GB28181 server via HTTP1. After the connection is successful, the browser will query the currently registered devices via HTTP1 and display them. For the devices registered subsequently, the GB28181 server will report them to the browser via HTTP1 for display.

[0059] In actual scenarios, the device end mainly uses the mobile network for communication, which is easily interfered or shielded by other signals to cause packet loss. The present invention is mainly used to solve the packet loss problem caused by media transmission from the device to the media server.

[0060] Since the actual network packet loss is uncontrollable, software simulation is needed to simulate packet loss during testing. Microsoft provided a network simulation tool (Network Emulator Toolkit) in the early days. First, find a computer that can connect to the server and has sufficient bandwidth, then configure the simulation tool parameters on this computer and set the random packet loss rate to 15%. Finally, let this computer provide a hotspot for the device to connect, and the packet loss rate between the device and the media server can be stabilized at 15±1%.

[0061] S4: The browser sends the streaming instruction to the GB28181 access server through HTTP1.

[0062] S5: The GB28181 access server notifies the media server to open the data transmission and reception ports through HTTP2, and sends the media server receiving stream address and media parameters to the device (G1).

[0063] S6: The device receives the stream pull message and replies the media parameters to the GB28181 access server.

[0064] S7: After receiving the device's reply, the GB28181 access server sends the media server's streaming address to the browser via HTTP1, and then sends an "ACK" message to the device, indicating that the device has started streaming.

[0065] S8: After receiving the media server streaming address replied by the GB28181 access server, the browser actively connects to the media server streaming address and starts to receive and play the stream.

[0066] S9: After the device receives the "ACK" message sent by the server, it starts to collect the camera video at the preset bit rate and frame rate, and encodes it into the video frame queue.

[0067] S10: The device packages the video frames in the video frame queue into RTP data packets, generates FEC data packets, and puts them into the sending queue and the FEC data queue respectively.

[0068] S11: The device sends the original video RTP packets in the sending queue to the media server through S1, and adds the sent RTP packets to the sent queue, saves them for a period of time, and regularly clears the timed data.

[0069] S12: The device sends the FEC data in the FEC data queue to the media server through S2.

[0070] S13: The media server puts the original video RTP and FEC data received from S1 and S2 into the RTP data queue and the FEC data queue respectively.

[0071] S14: The media server now forwards the waiting timeout data and the data with complete spliced ​​frames at the head of the RTP data queue to the browser through S3.

[0072] S15: The browser sends the media data received from S3 to the player for decoding and playback.

[0073] S16: The media server finds the missing packet sequence in the current RTP data queue, tries to repair it through the FEC data queue, and inserts the repaired data into the RTP data queue in packet sequence. If there are too many missing sequences or the FEC data itself is missing, resulting in a repair failure, a NACK signal is sent to the device through S1 to notify the device to resend a copy of the missing original data.

[0074] S17: After receiving the NACK signal sent by the media server, the device finds the data to be resent from the sent queue and resends it to the media server through S1.

[0075] S18: When the viewing is finished, the browser can send an end command to the GB28181 access server

[0076] S19: After receiving the end instruction, the GB28181 access server forwards it to the device through G1, and notifies the media server to close the receiving and forwarding ports through HTTP2.

[0077] S20: The device receives the end instruction, exits the streaming state, and stops streaming.

[0078] After comparative testing, in the case of 15% packet loss rate:

[0079] (1) In the traditional RTP / RTCP mode, the picture cannot be decoded due to packet loss.

[0080] (2) In TCP / RTP / RTCP mode, the video can be played, but the initial delay is as high as 7 seconds, which increases over time and causes serious lag.

[0081] (3) With the design of the present invention, the picture is smooth and the delay is only 2 seconds.

[0082] The working principle of the present invention is as follows: each GB28181 entity device registers with the GB28181 access server through G1 (national standard signaling flow) and communicates messages. The access server expands the SDP in the national standard stream pulling message "INVITE" and adds a new FEC data type. After receiving the entity device registration message, the access server searches for device information from the database DB1 and verifies the registration parameters, and then returns the result through G1. The browser side connects to the access server through HTTP1 to query the registered devices. Subsequent newly registered devices will also be reported to the browser side. The browser side sends a stream pulling instruction, and the access server notifies the media server to open the data sending and receiving port through HTTP2. After receiving the stream pulling message, the device replies with the media parameters, and the access server sends the stream address sent by the media server. An "ACK" message is sent to the browser and the device to start streaming. After receiving the "ACK", the device starts to collect video and encode it into a queue, generates RTP data packets and FEC data packets, and sends them to the media server through S1 and S2 respectively. After receiving the data, the media server puts them into the corresponding queues respectively, and forwards part of the data in the RTP data queue to the browser through S3. The browser decodes and plays. If there is a missing RTP data queue, try to repair it with FEC data. If the repair fails, send a NACK signal to the device to request retransmission. When the viewing ends, the browser sends an end instruction, the access server forwards it to the device and notifies the media server to close the port, and the device stops streaming. The entire process realizes video acquisition, transmission, playback and control through the interaction of multiple protocols and data streams.

[0083] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The above embodiments and descriptions only describe the principles of the present invention. The present invention may be subject to various changes and improvements without departing from the spirit and scope of the present invention. These changes and improvements fall within the scope of the present invention. The scope of protection claimed by the present invention is defined by the attached claims and their equivalents.

Claims

1. A weak network transmission optimization method based on a video surveillance networking protocol, characterized in that: Each entity device registers with the access server through G1 and communicates using the video surveillance networking protocol. The access server expands the SDP in the national standard streaming message and adds the FEC data type. The browser interacts with the access server through HTTP1 and sends streaming instructions. The access server notifies the media server of the operation through HTTP2. The entity device collects, encodes and sends video data according to the instructions. The data is transmitted to the media server through S1 and S2, and is forwarded to the browser for playback through S3 after processing. The G1 is the national standard signaling stream, S1, S2, and S3 are all RTP media streams, S1 and S2 are transmitted using UDP, and S3 is transmitted using TCP.

2. According to claim 1, a weak network transmission optimization method based on a video surveillance networking protocol is characterized in that: Specifically include the following steps: S1: Each physical device registers with the access server through G1, and each physical device and the access server use the video surveillance networking protocol to communicate with each other; S2: The access server receives the physical device registration message through G1, searches the physical device information from the database and verifies the registration parameters, and returns the result to the physical device through G1; S3: The browser communicates with the access server through HTTP1. After the connection is successful, the browser will first query the currently registered devices through HTTP1 and display them. For the subsequently registered devices, the access server will report them to the browser through HTTP1 for display; S4: The browser sends the streaming instruction to the access server via HTTP1; S5: The access server notifies the media server to open the data transmission and reception ports through HTTP2, and sends the media server receiving stream address and media parameters to the physical device; S6: The physical device receives the stream pull message and replies the media parameters to the access server; S7: After receiving the reply from the physical device, the access server sends the streaming address of the media server to the browser via HTTP1, and then sends an "ACK" message to the physical device, indicating that the physical device starts streaming. S8: After receiving the media server streaming address replied by the access server, the browser actively connects to the media server streaming address and starts receiving and playing the stream; S9: After the physical device receives the "ACK" message sent by the server, the physical device starts to collect camera video at the preset bit rate and frame rate, and encodes and puts it into the video frame queue; S10: The physical device packages the video frames in the video frame queue into RTP data packets, generates FEC data packets, and puts them into the sending queue and the FEC data queue respectively; S11: The physical device sends the original video RTP packets in the sending queue to the media server through S1, and adds the sent RTP packets to the sent queue, saves them for a period of time, and regularly clears the timeout data; S12: The physical device sends the FEC data in the FEC data queue to the media server through S2; S13: The media server puts the original video RTP and FEC data received from S1 and S2 into the RTP data queue and FEC data queue respectively; S14: the media server now forwards the waiting timeout data and the data with complete spliced ​​frames at the head of the RTP data queue to the browser through S3; S15: The browser sends the media data received from S3 to the player for decoding and playback.

3. According to claim 2, a weak network transmission optimization method based on a video surveillance networking protocol is characterized in that: After step S15, the following steps are performed in sequence: S16: The media server finds the missing packet sequence in the current RTP data queue, tries to repair it through the FEC data queue, and inserts the repaired data into the RTP data queue according to the packet sequence; If there are too many missing sequences or the FEC data itself is missing, resulting in a repair failure, a NACK signal is sent to the physical device through S1 to notify the physical device to resend a copy of the missing original data; S17: After receiving the NACK signal sent by the media server, the physical device finds the data to be resent from the sent queue and resends it to the media server through S1.

4. According to claim 2, a weak network transmission optimization method based on a video surveillance networking protocol is characterized in that: The following steps are also included: S1: After watching, the browser sends an end instruction to the access server; S2: After receiving the end instruction, the access server forwards it to the physical device and notifies the media server to close the receiving and forwarding ports through HTTP2; S3: The physical device receives the end instruction, exits the streaming state, and stops streaming.

5. A weak network transmission optimization method based on a video surveillance networking protocol according to claim 1 or 2, characterized in that: The physical device mentioned refers to the law enforcement recorder device.

6. The method for optimizing weak network transmission based on video surveillance networking protocol according to claim 2 is characterized in that: The DB1 and DB2 are database data streams, and HTTP1 and HTTP2 are HTTP protocol data streams.

7. A weak network transmission optimization method based on a video surveillance networking protocol according to claim 1 or 2, characterized in that: The access server extends the session description protocol SDP in the national standard streaming message "INVITE", and the RTP parameter mapping table rtpmap adds the flexfec / 90000 type to the original video parameters PS / 90000, H264 / 90000, MPEG4 / 90000, and H265 / 90000, indicating forward error correction FEC data with a clock frequency of 90000.

8. A weak network transmission optimization method based on a video surveillance networking protocol according to claim 1 or 2, characterized in that: The media server receives, repairs, records and forwards media data; the browser provides a human-computer interaction interface and a platform management window for adding, deleting, modifying and checking device information; The database stores terminal device information.

9. The method for optimizing weak network transmission based on video surveillance networking protocol according to claim 1, characterized in that: The video surveillance networking protocol refers to the GB28181 communication protocol, and the physical device extends the GB28181 protocol SDP and adds NACK and FEC to the RTP stream.

10. A weak network transmission optimization system based on video surveillance networking protocol, characterized in that: Run the weak network transmission optimization method based on the video surveillance networking protocol according to any one of claims 1-9 to optimize weak network transmission.

Images