Non-perceptual RTP audio and video stream analysis and reduction method and system and medium

Abstract

The invention provides a non-perceptual RTP audio and video stream analysis and reduction method and system and a medium, and the method comprises the steps: accessing a monitoring system, and mirroring flow data in the monitoring system to a system network card to recognize and filter out target data, the target data comprising signaling protocol data and video stream protocol data, the monitoring system comprises a monitoring device and a storage device. Analyzing the signaling protocol data to correspondingly obtain configuration information, session control information and a video coding parameter set of a video stream; analyzing the video stream protocol data according to the configuration information and extracting field information, and dividing the video stream protocol data into video data and audio data according to the field information; restoring the audio data, restoring the video data, and synthesizing and synchronizing the restored audio data and video data to obtain a target video file; according to the invention, accurate format identification of the video stream and audio and video entity file restoration are realized, and any interference to an existing monitoring system is avoided.

Description

Technical Field

[0001] The present invention belongs to the field of monitoring technology, and relates to a monitoring data restoration method, in particular to an imperceptible RTP audio and video stream parsing and restoration method, system and medium. Background Art

[0002] The monitoring system is one of the most widely used systems in the security system. The mainstream construction site monitoring is to use handheld video communication equipment to achieve video monitoring. From the earliest analog monitoring, to the digital monitoring in recent years, to the current network video monitoring, the monitoring technology has changed a lot. Today, when IP technology is gradually unifying the world, it is necessary to re-understand the development history of the video monitoring system. From a technical perspective, the development of the video monitoring system is divided into the first generation of analog video monitoring system (CCTV), to the second generation of digital video monitoring system (DVR) based on "PC+Multimedia Card", to the third generation of video monitoring system (IPVS) based entirely on IP network.

[0003] In a network video surveillance system, external devices need to obtain data transmitted by network surveillance in real time. However, in the prior art, due to the lack of accurate recognition of video streams, video streams are prone to interfere with the surveillance system when obtaining surveillance video data, especially in some scenarios where highly concealed information acquisition is required, which has a greater impact and cannot meet the use requirements. Summary of the invention

[0004] The purpose of the present invention is to provide a method, system and medium for analyzing and restoring RTP audio and video streams without perception, so as to solve the problems in the above-mentioned background technology.

[0005] In a first aspect, the present invention provides a method for parsing and restoring RTP audio and video streams without perception, the method comprising: connecting an access module to a monitoring system, and mirroring the traffic data in the monitoring system to a network card connected to the access module, so as to identify and filter out target data, the target data comprising signaling protocol data and video stream protocol data, the monitoring system comprising a monitoring device and a storage device; parsing the signaling protocol data to obtain configuration information, session control information and a video encoding parameter set of the corresponding video stream; parsing the video stream protocol data and extracting field information according to the configuration information, and dividing the video stream protocol data into video data and audio data according to the field information; restoring the audio data, restoring the video data, synthesizing and synchronizing the restored audio data and video data to obtain a target video file.

[0006] In an implementation of the first aspect, identifying and filtering out target data includes: separating the data link layer in the traffic data to obtain an IP network layer data packet; extracting the source IP, destination IP and protocol number based on the IP network layer data packet and then separating the IP network layer to obtain transport layer data; extracting the source port and destination port of the TCP protocol and the UDP protocol based on the transport layer data and then separating the transport layer to obtain an application layer load.

[0007] In an implementation of the first aspect, parsing the signaling protocol data to obtain corresponding configuration information, session control information and video encoding parameter set of the video stream includes: parsing the RTSP protocol and the SIP protocol to obtain SDP information; extracting the configuration information, session control information and video encoding parameter set of the video stream according to the SDP information.

[0008] In an implementation of the first aspect, the parsing of the video stream protocol data according to the configuration information and extracting field information, and dividing the video stream protocol data into video data and audio data according to the field information, includes: parsing the video stream protocol data according to the configuration information to extract a sequence number, a timestamp, and a load type information; sorting the video stream protocol data according to the sequence number; and dividing the sorted video stream protocol data according to the video encoding parameter set to respectively obtain the corresponding audio data and the video data.

[0009] In an implementation of the first aspect, when there is no signaling information in the video stream protocol data, the video stream protocol data sorted according to the load type information standard definition is differentiated to obtain video data and audio data respectively.

[0010] In an implementation of the first aspect, the restoring of the audio data includes: obtaining RTP load type information of the audio data, identifying the audio format according to the RTP load type value; restoring the audio data to an audio stream according to a decoding method corresponding to the identified audio format; wherein, parsing is stopped for the audio data in a non-standard format.

[0011] In an implementation of the first aspect, the restoring of the video data includes: identifying a NAL unit with coding parameters in the video stream data, and identifying whether it is a parameter set based on the first byte in the NAL unit; after determining that the NAL unit is a parameter set, extracting the coding parameters in the NAL unit; identifying the video coding format of the video data based on the video coding parameter set or the coding parameters; adding start codes to the video coding parameter set in sequence and then splicing them with the video data in sequence, restoring and decoding to obtain a complete video raw stream.

[0012] In an implementation of the first aspect, the synthesizing and synchronizing the restored audio data and the video data includes: in a streaming media session, a sending end device periodically sends an RTCP control packet, wherein the RTCP includes an SR and a RR; parsing the RTCP control packet to extract an RTCP timestamp field; converting the RTCP timestamp field into an NTP timestamp field; and adjusting the playback time of the audio data and the video data through the NTP timestamp field to synchronously synthesize the audio data and the video data.

[0013] In an implementation of the first aspect, the restoring the video data further includes: when it is determined that the video stream data lacks encoding parameters, using a combination of multiple common video encoding parameters to restore and decode the video data until decoding is successful or cannot be decoded.

[0014] In the second aspect, the present invention provides a non-perceptual RTP audio and video stream parsing and restoration system, including: an access module, used to access a monitoring system, and mirror the traffic data in the monitoring system to the system network card to identify and filter out target data, the target data including signaling protocol data and video stream protocol data, and the monitoring system includes a monitoring device and a storage device; a parsing module, used to parse the signaling protocol data to obtain corresponding configuration information, session control information and video encoding parameter set of the video stream; a division module, used to parse the video stream protocol data and extract field information according to the configuration information, and divide the video stream protocol data into video data and audio data according to the field information; a restoration module, used to restore the audio data, restore the video data, synthesize and synchronize the restored audio data and video data to obtain a target video file.

[0015] In a third aspect, the present invention further provides a computer-readable storage medium having a computer program stored thereon, which, when executed, implements the above-mentioned imperceptible RTP audio and video stream parsing and restoration system.

[0016] As described above, the imperceptible RTP audio and video stream parsing and restoration method, system and medium of the present invention have the following features:

[0017] Beneficial effects:

[0018] Compared with the prior art, the present invention can automatically identify the video encoding format (such as H264, H265, etc.) without relying on the encoding information transmitted in the initial session through the network traffic mirroring of the access switch, and extract the video stream in the intranet of the target place. Without actively sending SIP protocol requests or other exploratory operations, the present invention realizes accurate format recognition of the video stream and restoration of audio and video entity files through deep analysis of RTP data. Moreover, this non-perceptual feature ensures that the system is completely transparent to the monitoring party and avoids any interference with the existing monitoring system. It is particularly suitable for scenarios where information acquisition requires high concealment. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 Shown is a flowchart of the imperceptible RTP audio and video stream parsing and restoration method described in an embodiment of the present invention.

[0020] Figure 2 Shown is a structural block diagram of the imperceptible RTP audio and video stream parsing and restoration system described in an embodiment of the present invention. DETAILED DESCRIPTION

[0021] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict.

[0022] It should be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present invention. The illustrations only show components related to the present invention rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed arbitrarily, and the component layout may also be more complicated.

[0023] See also Figure 1 and Figure 2 . The following embodiments of the present invention provide a method, system and medium for parsing and restoring RTP audio and video streams without perception, which can automatically identify the video encoding format (such as H264, H265, etc.) without relying on the encoding information transmitted in the initial session, and extract the video stream in the intranet of the target location. Without actively sending SIP protocol requests or other exploratory operations, the RTP data is deeply parsed to achieve accurate format recognition of the video stream and restoration of audio and video entity files. Moreover, this imperceptible feature ensures that the system is completely transparent to the monitoring party and avoids any interference with the existing monitoring system. It is particularly suitable for scenarios where information acquisition requires high concealment.

[0024] RTP (Real-time Transport Protocol): Real-time transport protocol, used to transmit audio and video data.

[0025] SIP (Session Initiation Protocol): Session Initiation Protocol, a signaling protocol used to initiate, maintain and terminate multimedia communication sessions.

[0026] RTSP (Real-time Streaming Protocol): Real-time streaming protocol, a network control protocol used to control streaming media servers.

[0027] RTCP (Real-time Control Protocol): Real-time control protocol that provides quality feedback and synchronizes audio and video streams.

[0028] SDP (Session Description Protocol): Session Description Protocol, used to exchange audio and video information.

[0029] H.264: A digital video codec standard proposed by the Joint Video Team (JVT), a joint venture of the ITU-T Video Coding Experts Group and the ISO / IEC Moving Picture Experts Group. It has become one of the most commonly used formats for high-precision video coding.

[0030] H.265: A digital video codec standard proposed by the Joint Video Team (JVT-VC), the next generation standard of H.264.

[0031] NAL (Network Abstraction Layer): The unit in which video encoding data (such as H.264) is transmitted over the network.

[0032] SPS (Sequence Parameter Set): Sequence parameter set, a key parameter set in H.264 and H.265 video encoding.

[0033] PPS (Picture Parameter Set): Picture parameter set, a key parameter set in H.264 and H.265 video encoding.

[0034] VPS (Video Parameter Set): Video parameter set, a key parameter set in H.265 video encoding.

[0035] I frame (Intra-coded Frame): Intra-coded frame, usually contains a complete frame of image information.

[0036] Start Code Prefix: The prefix before each NAL, used to separate NAL units.

[0037] Quintuple: five parameters: source IP, destination IP, source port, destination port, and transport layer protocol.

[0038] NVR (Network Video Recorder): A network video recorder, a device used to manage and store video surveillance.

[0039] UDP (User Datagram Protocol): User Datagram Protocol is a datagram mode that provides packet switching computer communication in a group of interconnected computer network environments.

[0040] TCP (Transmission Control Protocol): Transmission Control Protocol is a connection-oriented, reliable, byte stream-based transport layer communication protocol.

[0041] The technical solutions in the embodiments of the present invention will be described in detail below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0042] like Figure 1 As shown, in one embodiment, the present invention provides a method for parsing and restoring an RTP audio and video stream without perception, and the method comprises the following steps:

[0043] S101. Connect an access module to a monitoring system, and mirror the traffic data in the monitoring system to a system network card connected to the access module to identify and filter out target data, wherein the target data includes signaling protocol data and video stream protocol data, and the monitoring system includes a monitoring device and a storage device.

[0044] Specifically, a restoration system is first connected between the monitoring device and the NVR device in the monitoring system to mirror the flow data in the monitoring device to the system network card, so that the restoration system can identify and filter out the target data. After the flow data reaches the system network card, the restoration system parses the flow data as follows.

[0045] In one embodiment, identifying and filtering out target data includes:

[0046] Separating the data link layer in the traffic data to obtain an IP network layer data packet;

[0047] After extracting the source IP, destination IP and protocol number from the IP network layer data packet, the IP network layer is separated to obtain the transport layer data;

[0048] The source port and the destination port of the TCP protocol and the UDP protocol are extracted according to the transport layer data, and then the transport layer is separated to obtain the application layer load.

[0049] Specifically, in the process of restoring the system to parse the traffic data, first separate the data link layer to obtain the IP network layer data packet, extract the source IP, destination IP, and protocol number, and then separate the IP network layer to obtain the transport layer data, and extract the source port and destination port of the TCP and UDP protocols. After obtaining the five-tuple, separate the transport layer to obtain the application layer load. According to the characteristics of the Real-Time Streaming Protocol (RTSP), filter out the TCP protocol and the message with the source or destination port of 554; according to the characteristics of the Session Initiation Protocol (SIP), filter out the UDP protocol and the message with the source or destination port of 5060. According to the characteristics of the Audio and Video Data Control Protocol (RTCP), filter out the UDP protocol, and the message with the first byte of the UDP payload being 0x80 or 0xA0 and the second byte being C8; according to the characteristics of the Audio and Video Data Streaming Protocol (RTP), filter out the UDP stream with the first byte of the UDP payload being 0x80 or A0, and starting from the first packet, the 9th byte to the 12th byte of the UDP payload of multiple packets are the same.

[0050] S102: Parse the signaling protocol data to obtain corresponding configuration information of the video stream, session control information, and a video encoding parameter set.

[0051] In one embodiment, the parsing of the signaling protocol data to obtain configuration information, session control information, and a video encoding parameter set of a video stream includes:

[0052] Parse RTSP protocol and SIP protocol to obtain SDP information;

[0053] The configuration information, session control information and video encoding parameter set of the video stream are extracted according to the SDP information.

[0054] Specifically, the system parses the RTSP protocol and the SIP protocol, parses out the SDP information, extracts the configuration information of the audio and video stream from the SDP information, such as the port of the RTP stream, the RTP load type and sampling rate of the audio and video, and the video encoding parameter set such as SPS, PPS, etc., and performs base64 decoding on the parameter set to obtain its original hexadecimal value, thereby obtaining the original video encoding parameter set.

[0055] S103: Parse the video stream protocol data according to the configuration information and extract field information, and divide the video stream protocol data into video data and audio data according to the field information.

[0056] In one embodiment, parsing the video stream protocol data according to the configuration information and extracting field information, and dividing the video stream protocol data into video data and audio data according to the field information, includes:

[0057] Parsing the video stream protocol data according to the configuration information to extract sequence number, timestamp and payload type information;

[0058] Sorting the video stream protocol data according to the sequence number;

[0059] The sorted video stream protocol data is divided according to the video encoding parameter set to obtain the corresponding audio data and video data respectively.

[0060] After obtaining the configuration information, the video stream protocol data is further parsed according to the configuration information parsed from the signaling, and fields including sequence number, timestamp, and payload type are extracted. According to the sequence number information, the RTP stream is correctly sorted to avoid the impact of disorder on the video stream. The same process is also performed for pure RTP streams without signaling information. Then, the payload is extracted from the video stream protocol data, and the audio and video traffic is distinguished according to the video encoding parameter set parsed above.

[0061] The load type of audio is generally a value less than 34, while the load type value of video is generally greater than or equal to 96, that is, a dynamic type. The audio and video data are processed separately to obtain audio data and video data for subsequent decoding and restoration.

[0062] It should be noted that, when there is no signaling information in the video stream protocol data, the video stream protocol data sorted according to the load type information standard definition is distinguished to obtain video data and audio data respectively.

[0063] S104, restoring the audio data, restoring the video data, synthesizing and synchronizing the restored audio data and video data to obtain a target video file.

[0064] In one embodiment, restoring the audio data includes:

[0065] Acquire RTP load type information of the audio data, and identify the audio format according to the RTP load type value;

[0066] Restoring the audio data to an audio stream according to a decoding method corresponding to the identified audio format;

[0067] Wherein, parsing of the audio data in non-standard format is stopped.

[0068] In this embodiment, since the monitoring equipment generally uses standard audio formats, such as PCMA, PCMU, G722, G729, etc., after obtaining the audio stream data, the restoration system identifies the audio format through the RTP load type value, and restores it to the audio stream according to the standard decoding process of each format. The audio system does not parse the non-standard format, thereby obtaining the original audio data and completing the restoration process of the audio data.

[0069] In one embodiment, restoring the video data includes:

[0070] Identify a NAL unit with encoding parameters in the video stream data, and identify whether it is a parameter set according to the first byte in the NAL unit;

[0071] After determining that the NAL unit is a parameter set, extracting coding parameters in the NAL unit;

[0072] Identify a video encoding format of the video data according to the video encoding parameter set or the encoding parameter;

[0073] After adding start codes to the video encoding parameter set in sequence, the set is spliced ​​with the video data in sequence, and restored and decoded to obtain a complete video raw stream.

[0074] In this embodiment, after obtaining the separated video data, the NAL unit of the encoding parameter is first identified, and whether it is a parameter set is identified by the first byte of the NAL unit. The judgment condition is:

[0075] A. Read the first byte of the NAL unit and take the last 5 bits (from the 3rd to the 7th bit), which is 7 or 8 in decimal.

[0076] B. Read the first byte of the NAL unit and take the middle 6 bits (from bit 1 to bit 6), which is 32, 33, or 34 in decimal.

[0077] As long as one of the above conditions is met, that is, the type of the NAL unit (nal_unit_type) is 7 / 8 / 32 / 33 / 34, it is considered to be a NAL unit with coding parameters, and the parameters in this NAL unit are extracted as coding parameters.

[0078] The video encoding format is then identified based on the video encoding parameter set obtained from the signaling protocol data or the encoding parameters extracted from the video data.

[0079] If the video data does not have a corresponding signaling protocol, for example, the system accesses traffic during monitoring, the encoding format can be determined by the NAL unit type of the encoding parameters:

[0080] If the type of NAL unit nal_unit_type in the video data is 7 or 8, it is determined that the encoding of this video is H.264;

[0081] If the type nal_unit_type of the NAL unit in the video data is 32, 33, or 34, it is determined that the encoding of this video stream is H.265.

[0082] If the video data contains neither a corresponding signaling protocol nor a NAL for transmitting coding information, and only pure video frame data is transmitted (for example, the monitoring device only sends coding information at the beginning of traffic transmission), it is necessary to identify the coding format of the video by identifying data features of video frames such as I frames.

[0083] Specifically, if the first byte of more than 90% of the NAL units in the video data is 0x7c, 0x5c, or 0x3c, the video stream is determined to be encoded as H.264;

[0084] If the first byte of more than 90% of the NAL units in the video data is 0x62 or 0x02, it is determined that the encoding of this video stream is H.265.

[0085] After determining the video encoding format of the video data, the extracted video encoding parameter information and the parsed video data are added with a start code, and spliced ​​in sequence to restore a complete video raw stream.

[0086] For H.264 encoding, the processing flow is as follows:

[0087] When the value of the last 5 bits of the 0th byte of the NAL unit is 28 (0x1C): When the value of the 0th bit of the 1st byte is 1, it means that the frame is the first frame. First, write the start code (0x00000001), then take the first 3 bits of the 0th byte and the last 5 bits of the 1st byte to form a new byte, write the byte value, and then write the remaining bytes starting from the 2nd byte; when the value of the 0th bit of the 1st byte is not 1, it means that the frame is not the first frame, and directly write the remaining bytes starting from the 2nd byte. When the value of the last 5 bits of the 0th byte is not 28 (0x1C), but 1 / 5 / 6 / 7 / 8: First write the start code (0x00000001), and then write the remaining bytes starting from the 0th byte.

[0088] For H.265 encoding, the processing flow is as follows:

[0089] When the value of the middle 6 bits of the 0th byte is 49 (0x31): When the value of the 0th bit of the 2nd byte is 1, it means that the frame is the first frame. First, write the start code (0x00000001), then take the last 6 bits of the 2nd byte and shift it left by one bit, write the byte value, and then write the 0x01 mark, and then write the remaining bytes starting from the 3rd byte; when the value of the 0th bit of the 2nd byte is not 1, it means that the frame is not the first frame, and directly write the remaining bytes starting from the 3rd byte. When the value of the middle 6 bits of the 0th byte is not 49 (0x31): First write the start code (0x00000001), and then write the remaining bytes starting from the 0th byte.

[0090] In the case of missing video encoding parameter sets, the resolution, frame rate, chroma format, and color depth of the video stream are unknown. In practice, monitoring equipment uses default parameters, and the range of each parameter is fixed, such as resolution is generally 1920x1080, 1280x720, etc., and frame rate is generally 24, 25, 50, etc. The system will try to use multiple common video encoding parameter combinations, combine them with video frames, restore the video stream, and decode the video stream until it is successfully decoded or confirmed that it cannot be decoded.

[0091] It should be noted that the correctness of certain parameters does not determine whether the decoder can successfully decode. For example, the frame rate only affects the frame interval and video duration. Incorrect resolution may cause the screen to present a meaningless blurred image, so multiple raw video streams may be successfully decoded.

[0092] If the encoding parameters of the surveillance video are adjusted midway, such as changing the frame rate, resolution or encoding format, the modified data must contain parameter sets such as SPS and PPS, and can be parsed according to the above process.

[0093] It should be noted that when it is determined that the video stream data lacks encoding parameters, a combination of multiple common video encoding parameters is used to restore and decode the video data until decoding is successful or decoding fails.

[0094] In one embodiment, the synthesizing and synchronizing the restored audio data and the video data includes:

[0095] In a streaming media session, the sending end device periodically sends RTCP control packets, and the RTCP includes SR and RR;

[0096] Parsing the RTCP control packet to extract an RTCP timestamp field;

[0097] Converting the RTCP timestamp field to an NTP timestamp field;

[0098] The playing time of the audio data and the video data is adjusted through the NTP timestamp field to synchronize and synthesize the audio data and the video data.

[0099] In a streaming media session, the sending device periodically sends RTCP control packets, which include sender reports (SR) and receiver reports (RR). The restoration system parses these RTCP protocols and extracts the timestamp fields provided by RTCP, including NTP timestamps, RTP timestamps and other fields. It then converts the RTP timestamp into an NTP timestamp and uses these fields to adjust the playback time of the audio and video streams to ensure that the restored video and audio are synchronized.

[0100] After the audio data and video data are restored and synchronized, the restored audio stream and video stream are encapsulated into MP4 or other common media file formats by using the ffmpeg tool for further processing or storage.

[0101] The protection scope of the imperceptible RTP audio and video stream parsing and restoration method described in the embodiment of the present invention is not limited to the execution order of the steps listed in this embodiment. All solutions implemented by adding, reducing, and replacing steps in the prior art based on the principles of the present invention are included in the protection scope of the present invention.

[0102] An embodiment of the present invention further provides a computer-readable storage medium on which a computer program is stored. When the program is executed by an electronic device, the above-mentioned imperceptible RTP audio and video stream parsing and restoration method is implemented.

[0103] A person of ordinary skill in the art can understand that all or part of the steps in the method for implementing the above-mentioned embodiment can be completed by instructing a processor through a program, and the program can be stored in a computer-readable storage medium, and the storage medium is a non-transitory medium, such as a random access memory, a read-only memory, a flash memory, a hard disk, a solid-state hard disk, a magnetic tape, a floppy disk, an optical disc, and any combination thereof. The above-mentioned storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or a data center that includes one or more available media. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a tape), an optical medium (for example, a digital video disc (DVD)), or a semiconductor medium (for example, a solid-state disk (SSD)), etc.

[0104] An embodiment of the present invention also provides an imperceptible RTP audio and video stream parsing and restoration system, which can implement the imperceptible RTP audio and video stream parsing and restoration method described in the present invention. However, the implementation device of the imperceptible RTP audio and video stream parsing and restoration method described in the present invention includes but is not limited to the structure of the imperceptible RTP audio and video stream parsing and restoration system listed in this embodiment. All structural deformations and replacements of the prior art made according to the principles of the present invention are included in the protection scope of the present invention.

[0105] like Figure 2 As shown, in one embodiment, the present invention also provides a non-perceptual RTP audio and video stream parsing and restoration system, including:

[0106] The access module 201 is used to access the monitoring system and mirror the traffic data in the monitoring system to the system network card to identify and filter out target data, wherein the target data includes signaling protocol data and video stream protocol data. The monitoring system includes monitoring equipment and storage equipment.

[0107] The parsing module 202 is used to parse the signaling protocol data to obtain the configuration information of the video stream, the session control information and the video encoding parameter set.

[0108] The division module 203 is used to parse the video stream protocol data according to the configuration information and extract field information, and divide the video stream protocol data into video data and audio data according to the field information.

[0109] The restoration module 204 is used to restore the audio data, restore the video data, synthesize and synchronize the restored audio data and video data to obtain a target video file.

[0110] It should be noted that the structures and principles of the access module 201, the parsing module 202, the division module 203 and the restoration module 204 correspond one-to-one to the steps (steps S101 to S104) in the above-mentioned imperceptible RTP audio and video stream parsing and restoration method. The specific working principle can also be referred to the introduction of the imperceptible RTP audio and video stream parsing and restoration method in the above-mentioned embodiment, so it will not be repeated here.

[0111] In the several embodiments provided by the present invention, it should be understood that the disclosed system, device or method can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of modules / units is only a logical function division, and there may be other division methods in actual implementation, such as multiple modules or units can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or modules or units, which can be electrical, mechanical or other forms.

[0112] The modules / units described as separate components may or may not be physically separated, and the components displayed as modules / units may or may not be physical modules, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules / units may be selected according to actual needs to achieve the purpose of the embodiments of the present invention. For example, the functional modules / units in the various embodiments of the present invention may be integrated into one processing module, or each module / unit may exist physically separately, or two or more modules / units may be integrated into one module / unit.

[0113] Those of ordinary skill in the art should further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0114] The descriptions of the processes or structures corresponding to the above-mentioned figures have different emphases. For parts that are not described in detail in a certain process or structure, please refer to the relevant descriptions of other processes or structures.

[0115] The above embodiments are merely illustrative of the principles and effects of the present invention, and are not intended to limit the present invention. Anyone familiar with the art may modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by a person of ordinary skill in the art without departing from the spirit and technical concept disclosed by the present invention shall still be covered by the claims of the present invention.

Claims

1. A method for analyzing and restoring RTP audio and video streams without perception, characterized in that: The method comprises: Connecting an access module to a monitoring system, and mirroring traffic data in the monitoring system to a network card connected to the access module, so as to identify and filter out target data, wherein the target data includes signaling protocol data and video stream protocol data, and the monitoring system includes a monitoring device and a storage device; Parsing the signaling protocol data to obtain configuration information of the video stream, session control information and a video encoding parameter set; Parsing the video stream protocol data and extracting field information according to the configuration information, and dividing the video stream protocol data into video data and audio data according to the field information; The audio data is restored, the video data is restored, and the restored audio data and video data are synthesized and synchronized to obtain a target video file.

2. The method for analyzing and restoring RTP audio and video streams without perception according to claim 1, characterized in that: The method for identifying and filtering target data includes: Separating the data link layer in the traffic data to obtain an IP network layer data packet; After extracting the source IP, destination IP and protocol number from the IP network layer data packet, the IP network layer is separated to obtain the transport layer data; The source port and the destination port of the TCP protocol and the UDP protocol are extracted according to the transport layer data, and then the transport layer is separated to obtain the application layer load.

3. The method for analyzing and restoring the imperceptible RTP audio and video stream according to claim 2, characterized in that: The parsing of the signaling protocol data to obtain configuration information of the video stream, session control information and a video encoding parameter set accordingly includes: Parse RTSP protocol and SIP protocol to obtain SDP information; The configuration information, session control information and video encoding parameter set of the video stream are extracted according to the SDP information.

4. The method for analyzing and restoring the imperceptible RTP audio and video stream according to claim 3, characterized in that: The parsing of the video stream protocol data according to the configuration information and extracting field information, and dividing the video stream protocol data into video data and audio data according to the field information, includes: Parsing the video stream protocol data according to the configuration information to extract sequence number, timestamp and payload type information; Sorting the video stream protocol data according to the sequence number; The sorted video stream protocol data is divided according to the video encoding parameter set to obtain the corresponding audio data and video data respectively.

5. The method for analyzing and restoring the imperceptible RTP audio and video stream according to claim 4, characterized in that: When there is no signaling information in the video stream protocol data, the video stream protocol data sorted according to the load type information standard definition is distinguished to obtain video data and audio data respectively.

6. The method for analyzing and restoring RTP audio and video streams without perception according to claim 1, characterized in that: The restoring of the audio data comprises: Acquire RTP load type information of the audio data, and identify the audio format according to the RTP load type value; Restoring the audio data to an audio stream according to a decoding method corresponding to the identified audio format; Wherein, parsing of the audio data in non-standard format is stopped.

7. The method for analyzing and restoring RTP audio and video streams without perception according to claim 3 is characterized in that: The restoring of the video data includes: Identify a NAL unit with encoding parameters in the video stream data, and identify whether it is a parameter set according to the first byte in the NAL unit; After determining that the NAL unit is a parameter set, extracting coding parameters in the NAL unit; Identify a video encoding format of the video data according to the video encoding parameter set or the encoding parameter; After adding start codes to the video encoding parameter set in sequence, the set is spliced ​​with the video data in sequence, and restored and decoded to obtain a complete video raw stream.

8. The method for analyzing and restoring the imperceptible RTP audio and video stream according to claim 7, characterized in that: The synthesizing and synchronizing the restored audio data and the video data comprises: In a streaming media session, the sending end device periodically sends RTCP control packets, and the RTCP includes SR and RR; Parsing the RTCP control packet to extract an RTCP timestamp field; Converting the RTCP timestamp field to an NTP timestamp field; The playing time of the audio data and the video data is adjusted through the NTP timestamp field to synchronize and synthesize the audio data and the video data.

9. The method for analyzing and restoring the imperceptible RTP audio and video stream according to claim 7, characterized in that: The restoring of the video data further includes: When it is determined that the video stream data lacks encoding parameters, a plurality of common video encoding parameter combinations are used to restore and decode the video data until decoding is successful or decoding fails.

10. A non-perceptual RTP audio and video stream parsing and restoration system, characterized in that: include: An access module, used to access a monitoring system and mirror traffic data in the monitoring system to a network card connected to the access module, so as to identify and filter out target data, wherein the target data includes signaling protocol data and video stream protocol data, and the monitoring system includes a monitoring device and a storage device; A parsing module, used for parsing the signaling protocol data to obtain configuration information of the video stream, session control information and a video encoding parameter set; a division module, configured to parse the video stream protocol data and extract field information according to the configuration information, and divide the video stream protocol data into video data and audio data according to the field information; The restoration module is used to restore the audio data, restore the video data, synthesize and synchronize the restored audio data and video data to obtain a target video file.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed, the imperceptible RTP audio and video stream parsing and restoration system described in any one of claims 1 to 9 is implemented.

Images