Bypass-based VoIP audio and video real-time playback system and processing method

Abstract

The present invention relates to the field of audio and video analysis and auditing, and specifically to a bypass-based VOIP audio and video real-time playback system and a processing method, which comprises a network probe, a DASH codec server and an analysis center server; the network probe is used to obtain bypass acquisition mirror traffic, and parse the audio and video data stream therein to generate audio and video raw streams and communication logs; the DASH codec server is used to decode, resample and re-encode the audio and video raw streams into DASH stream data; the analysis center server is used to receive the DASH stream data and the communication log, so that the client does not need a dedicated playback device to directly perform live broadcast and on-demand playback on the browser side, thereby improving the scalability of the system; at the same time, it can monitor the audio and video communications of key targets in real time, and can play audio and video on the same track.

Description

Technical Field

[0001] The present invention relates to the field of audio and video analysis and auditing, and in particular to a bypass-based VOIP audio and video real-time playback system and a processing method. Background Art

[0002] Currently, DASH (Dynamic Adaptive Streaming over HTTP) is a dynamic adaptive streaming media transmission protocol based on HTTP. Since it is based on HTTP, it can use the existing Web infrastructure, such as content distribution network (CDN) and standard Web servers. The current mainstream browsers all support DASH data streams and can be played directly on the browser side. It is a kind of live broadcast technology and is used in low-time live broadcast scenarios.

[0003] The Chinese patent application number CN201410186321.5 discloses a "VoIP audio and video auditing method based on network monitoring", which provides a method in which a network sniffing device captures network messages from a bypass network and stores them in the cloud, and then selects messages from the cloud and downloads them to a playback device for playback. Among them, VoIP (Voice over Internet Protocol) is a communication technology based on IP voice transmission. This method can only be decoded and played back in a dedicated playback device. If there is a new decoding format extension, all playback devices need to be updated, and existing facilities such as CDN and Web servers cannot be directly used for direct playback on the browser side, which has poor scalability; the key monitored objects are only matched and retrieved through the five-tuple of the data message, and the five-tuple cannot monitor the key targets in the proxy mode. If the target has both audio and video, the current method can only play them separately, and cannot play audio and video on the same track. Summary of the invention

[0004] In view of the above problems, the present invention provides a bypass-based VoIP audio and video real-time playback system and processing method.

[0005] The adopted technical solution is a bypass-based VoIP audio and video real-time playback system, including a network probe, a DASH codec server and an analysis center server;

[0006] The network probe is used to obtain the bypass collection mirror traffic, and parse the audio and video data stream therein to generate audio and video raw streams and communication logs;

[0007] The DASH codec server is used to decode, resample and re-encode the raw audio and video stream into DASH stream data;

[0008] The analysis center server is used to receive DASH stream data and communication logs.

[0009] Optionally, the network probe parses the bypassed VoIP SIP / H323 signaling stream and RTP media stream, and adds different tags to the communicating parties based on the media stream IP and port information negotiated in the signaling stream. When both communicating parties have audio and video data at the same time, the same tag is used for the audio and video in the same direction, and the tag information is inserted into the frame header of each frame of the audio and video raw stream.

[0010] Optionally, the communication log includes the IP address, port number, account number, bidirectional flag, and communication status of the calling and called parties.

[0011] Optionally, after receiving the raw audio and video stream, the DASH codec server selects a decoder for decoding and generates PCM / YUV, resamples the PCM data and re-encodes it into an AAC format frame, and re-encodes the YUV into an H264 format frame.

[0012] Optionally, when there are multi-channel PCM, it is necessary to mix and then resample and re-encode it into AAC format frames. When there is only audio or video, the AAC format frames or H264 format frames are encapsulated into the slice file of DASH streaming data;

[0013] When there are audio and video at the same time, they are aligned to the same track and then encapsulated into the DASH stream data slice file.

[0014] Optionally, the analysis center server stores the received communication log in a database for providing an interface display.

[0015] Optionally, after receiving the DASH streaming data, the analysis center server stores the mpd file and the slice file to the disk path, marks the corresponding HTTP streaming address, and stores it in the database for client streaming and playback.

[0016] The present application also provides a bypass-based VoIP audio and video real-time processing method, which is implemented based on a VoIP audio and video real-time playback system, and is applied to a network probe end. The method includes the following steps:

[0017] Obtain bypass mirror traffic;

[0018] Identify and parse VoIP signaling and media flows in bypass mirrored traffic;

[0019] Generate an association tag for each of the communicating parties and generate a UUID for the association;

[0020] Generate communication logs and send them to the analysis center server;

[0021] Generate audio and video raw streams and send them to the DASH codec server.

[0022] The present application also provides a bypass-based VoIP audio and video real-time processing method, which is implemented based on a VoIP audio and video real-time playback system, and is applied to a DASH codec server. The method includes the following steps:

[0023] Receive audio and video raw streams;

[0024] Look up the hash table based on the UUID and match the decoding context;

[0025] Decode the audio and video raw streams, and resample the audio raw streams and encode them into AAC format, and re-encode the video raw streams into H264 format;

[0026] Align the decoded audio and video streams;

[0027] Generate DASH mpd files and slice files, and send them to the analysis center server.

[0028] The present application also provides a bypass-based VoIP audio and video real-time processing method, which is implemented based on a VoIP audio and video real-time playback system, and is applied to an analysis center server, and the method includes the following steps:

[0029] Receive communication logs and / or DASH streams;

[0030] Determine the type of data received;

[0031] Store communication logs for browser display;

[0032] Store the mpd file and slice files in the DASH stream to the disk directory;

[0033] Create an HTTP streaming address corresponding to the mpd file path of the UUID.

[0034] The benefits of the present invention include:

[0035] 1. It solves the problem that the client can directly broadcast live and play on demand on the browser side without the need for dedicated playback equipment, thus improving the scalability of the system; at the same time, it can monitor the audio and video communications of key targets in real time and play audio and video on the same track.

[0036] 2. It is only necessary to expand a new decoding plug-in on the central end, and there is no need to upgrade the client decoding plug-in, so it is much better than the existing technology in scalability.

[0037] 3. The existing technology solves the problem that when a new decoding format is extended, all playback devices need to be updated, and existing facilities such as CDN and Web servers cannot be used directly for direct playback on the browser side, resulting in poor scalability; the key monitored objects are only matched and retrieved through the five-tuple of the data message, and the five-tuple cannot monitor the key targets in the proxy mode. If the target has both audio and video, the current method can only play them separately, and cannot play audio and video on the same track. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 It is a schematic diagram of the overall framework structure of a bypass-based VoIP audio and video real-time playback system;

[0039] Figure 2 It is a flowchart for network probe processing;

[0040] Figure 3 This is a flow chart of the DASH codec server processing;

[0041] Figure 4 The following is a flow chart of the server processing in the analysis center. DETAILED DESCRIPTION

[0042] The following describes the implementation of the present invention through specific examples. Those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific implementations, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict.

[0043] It should be noted that the illustrations provided in the following embodiments are only used to schematically illustrate the basic concept of the present invention, and thus the drawings only show components related to the present invention rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed arbitrarily, and the component layout may also be more complicated.

[0044] like Figure 1 As shown, a bypass-based VoIP audio and video real-time playback system includes a network probe, a DASH codec server and an analysis center server;

[0045] The network probe is used to obtain the bypass collection mirror traffic, and parse the audio and video data stream therein to generate audio and video raw streams and communication logs;

[0046] The DASH codec server is used to decode, resample and re-encode the raw audio and video stream into DASH stream data;

[0047] The analysis center server is used to receive DASH stream data and communication logs.

[0048] The purpose of this design is to solve the problem of the client directly broadcasting and on-demand on the browser side without a dedicated playback device through the bypass-based VoIP audio and video real-time playback system, thereby improving the scalability of the system; at the same time, it can monitor the audio and video communications of key targets in real time, and can play audio and video on the same track. It only needs to expand the new decoding plug-in on the central side, and there is no need to upgrade the client decoding plug-in, so it is much better than the existing technology in scalability. It solves the problem in the existing technology that when there is a new decoding format extension, all playback devices need to be updated, and existing facilities such as CDN and Web servers cannot be used directly for direct playback on the browser side, resulting in poor scalability; the key monitored objects are only matched and retrieved through the five-tuple of the data message, and the five-tuple cannot monitor the key targets in the proxy mode. If the target has both audio and video, the current method can only play them separately, and cannot play audio and video on the same track.

[0049] At the same time, in this embodiment, the network probe is a system that collects traffic analysis from a bypass, mainly parsing audio and video data streams, parsing VOIP SIP / H323 signaling streams, and RTP media streams, and marking the communicating parties with different tags through the negotiated media stream IP and port information in the signaling stream. If the communicating parties have audio and video data at the same time, the audio and video tags in the same direction are the same, and the signaling stream is extracted to generate a call log. The call log field contains key information such as the IP, port, account, two-way tag, and communication status (communicating / completed) of the caller and the called party.

[0050] In the media stream analysis part, the media stream is parsed into the raw stream of the original audio and video encoding, and the marking information and the parameter information related to the audio and video encoding and decoding are marked on each frame of the raw stream. The call log output by the network probe is sent to the analysis center server for storage and used for front-end display.

[0051] The raw audio and video stream is sent to the DASH encoding service for decoding, resampling, and re-encoding into a DASH stream. If the audio and video tags are the same, they need to be processed on the same track to generate a DASH stream with the same track. The processed DASH stream is sent to the analysis center server for storage, generating an accessible stream pull address, and the search keyword is the tag.

[0052] According to the call log displayed by the front-end browser, you can choose to live broadcast or on-demand according to the call status. Then retrieve the streaming address of the streaming service according to the tag and play the DASH streaming. If it is live broadcast mode, real-time monitoring can be performed, and if it is on-demand, fast forward, pause, jump, etc. can be supported.

[0053] like Figure 2 As shown, this embodiment provides a bypass-based VoIP audio and video real-time processing method. At the network probe end, the method includes the following steps:

[0054] Obtain bypass mirror traffic;

[0055] Identify and parse VoIP signaling and media flows in bypass mirrored traffic;

[0056] Generate an association tag for each of the communicating parties and generate a UUID for the association;

[0057] Generate communication logs and send them to the analysis center server;

[0058] Generate audio and video raw streams and send them to the DASH codec server.

[0059] It should be noted that when the media stream is identified, if the identification is successful, the UUID can be associated and generated, and if the identification is unsuccessful, the media stream is discarded.

[0060] The network probe processing steps are mainly to obtain bypass traffic, identify RTP media streams, parse signaling streams and media streams, generate logs and audio and video stream association tags, generate communication logs, and audio and video raw streams. Each party in the communication generates a tag to associate audio and video data. If the two parties in the communication have audio and video streams, the same tag is used for the audio and video streams in the same direction. A sid is generated for each audio and video stream. The fields of the communication log include key information such as the IP, port, account, bidirectional tag, and communication status (communicating / completed) of the caller and the callee. The frame header of each frame of the audio and video raw stream inserts tag information and parameter information related to audio and video encoding and decoding.

[0061] like Figure 3 As shown, this embodiment provides a bypass-based VoIP audio and video real-time processing method. At the DASH codec server side, the method includes the following steps:

[0062] Receive audio and video raw streams;

[0063] Look up the hash table based on the UUID and match the decoding context;

[0064] Decode the audio and video raw streams, and resample the audio raw streams and encode them into AAC format, and re-encode the video raw streams into H264 format;

[0065] Align the decoded audio and video streams;

[0066] Generate DASH mpd files and slice files, and send them to the analysis center server.

[0067] It should be pointed out that the DASH codec server processing steps mainly receive the raw audio and video stream data sent by the network probe, find the corresponding decoder to decode it into PCM / YUV, resample the PCM data and re-encode it into AAC format frames. If there are multi-channel PCM, it needs to be mixed and then resampled and re-encoded into AAC format frames; re-encode YUV into H264 format frames. If there is only audio or video, the AAC format frame or H264 format frame is encapsulated into the DASH slice file; if there is audio and video, it is aligned to the same track and then encapsulated into the DASH trimming file. Finally, mpd and trimming files are generated.

[0068] It should be added that the codec server parses the complete frame of the audio and video raw stream, and searches the hash table according to the UUID. When the match is successful, the decoding context is decoded. When the match is not successful, it is necessary to find the corresponding decoder according to the frame header parameters, establish the decoding context, and insert the decoding context into the hash table before performing context decoding.

[0069] At the same time, in the audio and video processing part, it is necessary to determine whether the audio stream and video stream exist at the same time. If they exist at the same time, the audio stream and video stream need to be aligned, and then the DASH mpd file and slice file are generated and sent to the analysis center server.

[0070] like Figure 4 As shown, this embodiment provides a bypass-based VoIP audio and video real-time processing method. At the analysis center server side, the method includes the following steps:

[0071] Receive communication logs and / or DASH streams;

[0072] Determine the type of data received;

[0073] Store communication logs for browser display;

[0074] Store the mpd file and slice files in the DASH stream to the disk directory;

[0075] Create an HTTP streaming address corresponding to the mpd file path of the UUID.

[0076] It should be pointed out that the analysis center server stores the process, and after receiving the communication log, the analysis center stores it in the database to provide an interface display; after receiving the DASH stream data, the mpd file and slice file are stored in the disk path, and the corresponding HTTP streaming address is stored in the database for client streaming and playback.

[0077] The browser accesses a certain communication in the communication log, queries the corresponding HTTP streaming address through the tag, and performs live broadcast or on-demand playback through the streaming address. Live broadcast and on-demand playback are distinguished by the communication status field in the communication log. The browser can query and play in real time based on key information in the communication log, such as IP, port, account, etc.

[0078] It should be added that after receiving the communication log and the DASH stream, the analysis center server needs to judge the received data to determine whether it is a communication log or a DASH stream, and process the two differently.

[0079] Finally, it should be noted that the above are only preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, it is still possible for those skilled in the art to modify the technical solutions described in the aforementioned embodiments, or to make equivalent substitutions for some of the technical features therein. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A bypass-based VoIP audio real-time playback system, characterized in that: Includes network probes, DASH codec servers, and analysis center servers; The network probe is used to obtain the bypass collection mirror traffic, and parse the audio and video data stream therein to generate audio and video raw streams and communication logs; The DASH codec server is used to decode, resample and re-encode the raw audio and video stream into DASH stream data; The analysis center server is used to receive DASH stream data and communication logs; the network probe parses the bypassed VoIP SIP / H323 signaling stream and RTP media stream, and based on the media stream IP and port information negotiated in the signaling stream, marks the communicating parties with different tags respectively. When the communicating parties have audio and video data at the same time, the audio and video in the same direction use the same tag, and insert the tag information into the frame header of each frame of the audio and video raw stream; After receiving the raw audio and video stream, the DASH codec server selects a decoder for decoding and generates PCM / YUV, resamples the PCM data and re-encodes it into an AAC format frame, and re-encodes the YUV into an H264 format frame; When there are multi-channel PCM, it is necessary to mix and then resample and re-encode it into AAC format frames. When there is only audio or video, the AAC format frames or H264 format frames are encapsulated into the slice file of DASH stream data; When there are both audio and video, they are aligned to the same track and then encapsulated into DASH stream data slice files; After receiving the DASH stream data, the analysis center server stores the mpd file and the slice file in the disk path, marks the corresponding HTTP streaming address, and stores it in the database for client streaming and playing.

2. According to claim 1, a bypass-based VoIP audio real-time playback system is characterized in that: The communication log contains the IP, port, account, bidirectional flag, and communication status of the calling and called parties.

3. The bypass-based VoIP audio real-time playback system according to claim 1, characterized in that: The analysis center server stores the received communication log in a database for providing an interface display.

4. A bypass-based VoIP audio real-time processing method, implemented based on the VoIP audio real-time playback system described in any one of claims 1 to 3, characterized in that: Applied to the network probe end, the method includes the following steps: Obtain bypass mirror traffic; Identify and parse VoIP signaling and media flows in bypass mirrored traffic; Generate an association tag for each of the communicating parties and generate a UUID for the association; Generate communication logs and send them to the analysis center server; Generate audio and video raw streams and send them to the DASH codec server.

5. A bypass-based VoIP audio real-time processing method, implemented based on the VoIP audio real-time playback system described in any one of claims 1 to 3, characterized in that: Applied to the DASH codec server, the method includes the following steps: Receive audio and video raw streams; Look up the hash table based on the UUID and match the decoding context; Decode the audio and video raw streams, and resample the audio raw streams and encode them into AAC format, and re-encode the video raw streams into H264 format; Align the decoded audio and video streams; Generate DASH mpd files and slice files, and send them to the analysis center server.

6. A bypass-based VoIP audio real-time processing method, implemented based on the VoIP audio real-time playback system described in any one of claims 1 to 3, characterized in that: Applied to the server side of the analysis center, the method includes the following steps: Receive communication logs and / or DASH streams; Determine the type of data received; Store communication logs for browser display; Store the mpd file and slice files in the DASH stream to the disk directory; Create an HTTP streaming address corresponding to the mpd file path of the UUID.

Images