Effective streaming media audio and video recording system

Abstract

The invention discloses an effective streaming media audio and video recording system, and aims to improve audio and video processing efficiency and flexibility. According to the system, audio and video stream characteristics and system resources are dynamically analyzed through the streaming media control module, an encoding and decoding or direct connection processing mode is adopted for intelligent decision, and the problems that the performance burden is increased and the direct packaging function of the coded stream is limited due to traditional decoding and re-encoding are effectively solved. The coding and decoding unit can process complex audio and video data and support advanced functions such as audio mixing synthesis; and the direct connection unit ensures high concurrent processing and efficient file segmentation storage. The system supports a plurality of streaming media protocols such as RTP / RTSP / RTMP / WebRTC and the like, and adapts to various transmission requirements. Through intelligent analysis and flexible processing, the system not only improves the audio and video recording efficiency and quality, but also enhances the information display and file processing capabilities, and provides a comprehensive and efficient streaming media audio and video recording solution for the fields of remote cooperation, online education, security monitoring and the like.

Description

Technical Field

[0001] The present invention relates to RTP transmission technology, RTMP transmission technology, WebRTC technology, TCP / UDP transmission technology, FFmpeg technology, H.264 encoding and decoding technology, Opus encoding and decoding technology, audio and video encapsulation technology and Jitter Buffer technology, and specifically to an effective streaming media recording and video recording system. Background Art

[0002] With the rapid rise of remote office collaboration, online education and training, and security surveillance worldwide, instant messaging methods like audio and video intercoms and video conferencing have become an integral part of our daily lives and work. These applications not only require real-time, efficient audio and video transmission, but also often require the recording of communications for subsequent playback, review, or evidence.

[0003] Traditional server-based recording and video recording methods are unable to meet these requirements. On the one hand, while decoding and re-encoding can perform certain processing on audio and video data, such as mixing and synthesis, it greatly increases the performance burden of the device, especially in high-concurrency scenarios, where the server may face enormous processing pressure. On the other hand, while direct encapsulation of encoded streams can reduce the processing burden on the server, it cannot meet the needs of complex processing such as mixing and synthesis of audio and video data, limiting the functionality and flexibility of recording and video files.

[0004] Furthermore, with the continuous development of technology, streaming media transmission protocols such as RTP, RTSP, RTMP, and WebRTC have become increasingly mature, providing strong support for the real-time transmission of audio and video data. However, traditional recording and video recording methods often fail to fully utilize the advantages of these protocols, resulting in deficiencies in transmission efficiency and data integrity.

[0005] Therefore, developing a streaming media recording and video recording system that can effectively reduce the performance burden of the server and meet the complex processing requirements of audio and video data has become an urgent need in the current technological development. Summary of the Invention

[0006] The purpose of the present invention is to provide an effective streaming media recording and video recording system to improve the efficiency and flexibility of streaming media recording and video recording. By intelligently deciding encoding and decoding or directly processing data streams, it solves the defects of traditional decoding and re-encoding and direct encapsulation of encoded streams, enhances the information display and processing capabilities of recording and video files, and supports multiple streaming media transmission protocols to meet a wide range of application needs.

[0007] To achieve the above objectives, the present invention provides the following technical solutions:

[0008] An effective streaming media recording and video recording system includes:

[0009] The streaming media control module is used to dynamically analyze the data characteristics and system resource usage of audio and video streams, and decide how to process the data stream based on the analysis results;

[0010] The data processing module performs encoding and decoding or direct processing on the data stream according to the decision result of the streaming media control module.

[0011] Further,

[0012] The streaming media control module includes:

[0013] The analysis unit is used to dynamically analyze the data volume, data bit rate resolution, data stability changes, and system CPU and memory information of multiple channel buffers;

[0014] The decision unit establishes a state machine based on the information from the analysis unit to decide whether the current data stream should be recorded or recorded using the codec method or the original bit stream;

[0015] The execution unit sends the data stream to the corresponding unit of the data processing module for processing based on the results of the decision unit.

[0016] Further,

[0017] The decision unit further comprises:

[0018] Whether the data volume of the channel exceeds the threshold determines whether to enter the bypass state or the encoding state;

[0019] The decision to enter the bypass state or the coding state is based on whether the channel duration and stability exceed the threshold;

[0020] The system determines whether to enter the bypass state or the coding state based on the system information.

[0021] Furthermore, the data processing module includes:

[0022] The media encoding and decoding unit is used to handle the decoding and re-encoding process, add processing to the original data, dynamically add mixing and synthesis operations for audio and video data, and improve the information display of audio and video files;

[0023] The media pass-through unit is used to handle the pass-through process, focusing on the concurrency and file processing of audio and video data, providing high-concurrency audio and video recording requirements, and better segmenting and storing audio and video files.

[0024] Further,

[0025] The media encoding and decoding unit further includes:

[0026] Receive streaming media data sent by multiple terminals, process data packet loss, jitter and delay to ensure data integrity;

[0027] Decapsulate streaming media data into two channels of original data of audio stream and video stream;

[0028] Decode the audio and video channels separately to form their respective original data;

[0029] Add functions according to business needs, including audio mixing, video synthesis, video subtitles, and watermarks;

[0030] Re-encode and encapsulate data and transmit it to the recording unit for storage and query;

[0031] The copied data is transmitted to the network transmission unit for network transmission, display and playback preview.

[0032] Further,

[0033] The media direct pass unit further includes:

[0034] Receive data from multiple terminals and perform anti-jitter and anti-packet loss processing to ensure data integrity and continuity;

[0035] Transmit data to the recording control unit for the purpose of associating the cutting, segmentation and indexing storage information of audio and video files;

[0036] Use the tool library to encapsulate the respective media streams into independent video files;

[0037] The data is directly transmitted to the network transmission unit for network transmission, display, playback and preview.

[0038] Furthermore, the system supports streaming media transmission protocols such as RTP / RTSP / RTMP / WebRTC.

[0039] The streaming media recording and video recording system of the present invention has the following beneficial effects:

[0040] Improve processing efficiency:

[0041] The streaming media control module dynamically analyzes the data characteristics and system resource usage of the audio and video streams, and decides how to process the data stream based on the analysis results, thereby achieving efficient processing of audio and video data.

[0042] The flexible switching between the codec unit and the pass-through unit enables the system to select the appropriate processing method according to actual needs, further improving processing efficiency.

[0043] Reduce server burden:

[0044] The design of the pass-through unit allows the data stream to be directly forwarded to the storage or transmission unit when complex processing of audio and video data is not required, avoiding unnecessary decoding and re-encoding processes, thereby reducing the performance burden of the server.

[0045] Enhanced functionality and flexibility:

[0046] The codec unit supports complex processing of audio and video data such as mixing, synthesis, adding subtitles and watermarks, meeting the diverse needs of different business scenarios.

[0047] The system supports multiple streaming media transmission protocols, such as RTP, RTSP, RTMP, WebRTC, etc., which enhances the compatibility and flexibility of the system.

[0048] Improve data integrity and transmission efficiency:

[0049] The integrity and continuity of audio and video data are ensured through data anti-jitter, anti-packet loss and other processing mechanisms.

[0050] The optimized data transmission process improves the transmission efficiency of audio and video data and reduces delay and packet loss rate.

[0051] Easy to manage and maintain:

[0052] The system supports the association and management of information such as the cutting, segmented index storage, etc. of audio and video files, which facilitates subsequent query and playback operations.

[0053] The modular design makes the system easy to expand and maintain, reducing operation and maintenance costs.

[0054] In summary, the streaming media recording and video recording system of the present invention realizes efficient, flexible and reliable processing of audio and video data through dynamic analysis, flexible decision-making and efficient processing, meets the diverse needs in different business scenarios, and has significant technical advantages and market prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 Schematic diagram of the system architecture of the streaming media recording and video recording system of the present invention;

[0056] Figure 2 This is the framework schematic diagram of the streaming media control module;

[0057] Figure 3 This is the framework schematic diagram of the media encoding and decoding unit;

[0058] Figure 4 This is a schematic diagram of the media pass-through unit framework. DETAILED DESCRIPTION

[0059] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions of the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings herein can be arranged and designed in various different configurations.

[0060] Example:

[0061] This embodiment provides an effective streaming media recording and video recording system, including:

[0062] Streaming media control module, which is used to dynamically analyze the data characteristics and system resource usage of audio and video streams, and decide how to process the data streams based on the analysis results;

[0063] This module decides whether the current data stream should be recorded or recorded using the codec method or the original bit stream based on the data stream size, bit rate, data continuity and stability, data volume and device performance information.

[0064] The data processing module encodes and decodes the data stream or performs direct processing according to the decision result of the streaming control module;

[0065] This module is allocated to different data state processing flows according to the state machine of the media control module. The decoding and re-encoding flow will focus on adding processing for the original data, dynamically adding mixing and synthesis operations for audio and video data, and improving the information display of audio and video files. The through-flow focuses on the concurrency and file processing of audio and video data, and can provide high-concurrency audio and video recording requirements, while better performing segmentation and storage operations for audio and video files.

[0066] Specifically,

[0067] (1) Streaming Media Control Module

[0068] After the audio and video streams are transmitted to the server through the RTP / RTSP / RTMP / WebRTC protocol, they will first enter the media control module, which mainly consists of three units:

[0069] Analysis unit:

[0070] Dynamically analyze the data volume, data bit rate resolution, data stability changes, and system CPU and memory information of multiple channel buffers.

[0071] Decision-making unit:

[0072] According to the information of the analysis unit, a state machine is established. The basic process is as follows:

[0073] The bypass state or encoding state is determined based on whether the channel data volume (current bandwidth usage, number of data packets) exceeds the threshold;

[0074] The bypass state or encoding state is determined based on whether the duration and stability of the channel (the flow is unstable at the beginning and then stabilizes and changes state) exceeds the threshold;

[0075] The system decides whether to enter the bypass state or the coding state based on system information (CPU utilization, memory utilization).

[0076] Execution Unit:

[0077] According to the results of the decision-making unit, the information in various states is processed first, and the data stream is sent to the corresponding unit of the data processing module for processing.

[0078] (2) Data processing module

[0079] After the data is processed by the media control module, it enters the data processing module. This module determines whether the data transmission flow is to go through the media codec unit or the media pass-through unit based on the results of the decision unit of the media processing module. The processes of each unit are as follows:

[0080] Media Codec Unit

[0081] After being processed by the media control module, multiple terminals (clientA, clientB, clientC) will send their respective streaming media data to the codec unit part of the central streaming media processing module;

[0082] The unit first processes data packet loss, jitter, and delay on the media server to ensure data integrity, and decapsulates streaming media data into two channels: audio stream and video stream.

[0083] Then, the audio and video channels are decoded to form their respective raw media;

[0084] The control unit in the processing module adds functions (including audio mixing, video synthesis, video subtitles, and watermarks) to the original streaming media data according to business needs;

[0085] The data is then re-encoded and packaged and transferred to the record unit, which uses the FFmpeg+MP4v2 tool library to package the data into MP4, AVI and other format video files for storage and query;

[0086] At the same time, the data is copied and transferred to the net transport unit for network transmission, display and preview.

[0087] Bypass unit

[0088] Multiple terminal data enter the streaming media direct unit after being decided by the media control unit;

[0089] The media server of the direct-through unit implements some data anti-jitter and anti-packet loss (jitter buffer) logic to ensure data integrity and continuity.

[0090] The data will then be transmitted to the record control unit, which is used to cut and segment the audio and video files and store the information for related business needs;

[0091] The record unit then uses the FFmpeg / avilib tool library to encapsulate the respective media streams into independent MP4 and AVI recording files;

[0092] The data is then directly transferred to the net transport unit for network transmission, display, playback and preview.

[0093] In summary, the core of the effective streaming media recording and video recording system of the present invention lies in the intelligent analysis and flexible processing of audio and video stream data through an innovative streaming media control module and an efficient data processing module; the system can dynamically adapt to data characteristics and system resource conditions, and independently decide whether to use encoding and decoding or direct-through methods to process data streams, which not only overcomes the performance burden and functional limitations of traditional methods, but also significantly improves the efficiency and quality of recording and video recording; at the same time, the system supports multiple streaming media transmission protocols and has powerful information display and file processing capabilities, providing efficient and flexible streaming media recording and video recording solutions for remote office, online education, security monitoring and other fields.

[0094] The present invention and its embodiments are described above. This description is not restrictive. The drawings show only one embodiment of the present invention, and the actual structure is not limited thereto. In short, if a person skilled in the art is inspired by this and, without departing from the purpose of the present invention, designs structures and embodiments similar to this technical solution without inventiveness, they shall fall within the scope of protection of the present invention.

Claims

1. An effective streaming media recording and video recording system, characterized in that: include: Streaming media control module, which is used to dynamically analyze the data characteristics and system resource usage of audio and video streams, and decide how to process the data streams based on the analysis results; The data processing module performs encoding and decoding or direct processing on the data stream according to the decision result of the streaming media control module.

2. An effective streaming media recording and video recording system according to claim 1, characterized in that: The streaming media control module includes: The analysis unit is used to dynamically analyze the data volume, data bit rate resolution, data stability changes, and system CPU and memory information of multiple channel buffers; The decision unit establishes a state machine based on the information from the analysis unit to decide whether the current data stream should be recorded or recorded using the codec method or the original bit stream; The execution unit sends the data stream to the corresponding unit of the data processing module for processing based on the results of the decision unit.

3. An effective streaming media recording and video recording system according to claim 2, characterized in that: The decision unit further comprises: Whether the data volume of the channel exceeds the threshold determines whether to enter the bypass state or the encoding state; The system decides whether to enter the bypass state or the coding state based on whether the channel duration and stability exceed the threshold; The system determines whether to enter the bypass state or the coding state based on the system information.

4. The effective streaming media recording and video recording system according to claim 1, characterized in that: The data processing module includes: The media encoding and decoding unit is used to handle the decoding and re-encoding process, add processing to the original data, dynamically add mixing and synthesis operations for audio and video data, and improve the information display of audio and video files; The media pass-through unit is used to handle the pass-through process, focusing on the concurrency and file processing of audio and video data, providing high-concurrency audio and video recording requirements, and better segmenting and storing audio and video files.

5. An effective streaming media recording and video recording system according to claim 4, characterized in that: The media encoding and decoding unit further includes: Receive streaming media data sent by multiple terminals, process data packet loss, jitter and delay to ensure data integrity; Decapsulate streaming media data into two channels of original data of audio stream and video stream; Decode the audio and video channels separately to form their respective original data; Add functions according to business needs, including audio mixing, video synthesis, video subtitles, and watermarks; Re-encode and encapsulate data and transmit it to the recording unit for storage and query; The copied data is transmitted to the network transmission unit for network transmission, display and playback preview.

6. An effective streaming media recording and video recording system according to claim 4, characterized in that: The media direct pass unit further includes: Receive data from multiple terminals and perform anti-jitter and anti-packet loss processing to ensure data integrity and continuity; Transmit data to the recording control unit for the purpose of associating the cutting, segmentation and indexing storage information of audio and video files; Use the tool library to encapsulate the respective media streams into independent video files; The data is directly transmitted to the network transmission unit for network transmission, display, playback and preview.

7. An effective streaming media recording and video recording system according to any one of claims 1 to 6, characterized in that: The system supports streaming media transmission protocols such as RTP / RTSP / RTMP / WebRTC.

Images