Streaming media data acquisition system and streaming media data acquisition method

Abstract

The invention provides a streaming media data acquisition system and a streaming media data acquisition method. The streaming media data acquisition system comprises at least one service center and a plurality of distributed streaming media cluster nodes. Wherein the distributed streaming media cluster node is used for acquiring first flow data of a streaming media service, and cleaning, calculating and storing the first flow data to obtain stored second flow data. And the service center is used for calling the second flow data correspondingly stored in the plurality of distributed streaming media cluster nodes. According to the invention, the flow data is cleaned, calculated and stored through the distributed streaming media cluster nodes, so that the number of service nodes configured by a service center is reduced, and the resource overhead of the service center is reduced.

Description

Technical Field

[0001] The present application relates to communication technology, and in particular to a streaming media data collection system and a streaming media data collection method. Background Art

[0002] With the rapid development of science and technology, streaming media technology has been widely used in the field of video and audio playback. In order to better provide sufficient traffic for streaming media services, it is often necessary to collect traffic data used by streaming media services.

[0003] In the prior art, when collecting traffic data, the current streaming media data collection system usually obtains traffic data from each streaming media cluster node, and then couples the traffic data obtained from each streaming media cluster node through the business center, and then cleans, calculates and stores the coupled data through multiple service nodes configured in the business center, and finally obtains the collected traffic data.

[0004] However, the existing technology business center is configured with many service nodes, which occupies a large amount of resource overhead. Summary of the invention

[0005] The present application provides a streaming media data collection system and a streaming media data collection method, which are used to solve the problem that a business center in the prior art has a large number of service nodes configured, which occupies a large resource overhead.

[0006] In a first aspect, the present application provides a streaming media data collection system, comprising:

[0007] At least one business center and multiple distributed streaming cluster nodes;

[0008] The distributed streaming media cluster node is used to obtain first flow data of the streaming media service, and clean, calculate and store the first flow data to obtain stored second flow data;

[0009] The business center is used to call the second traffic data correspondingly stored in the multiple distributed streaming media cluster nodes.

[0010] Optionally, the distributed streaming media cluster node includes a log collection module;

[0011] The log collection module is used to read the log files of the streaming media service at preset intervals;

[0012] It is also used to obtain first traffic data of the streaming media service according to the log file, where the first traffic data includes one or more of the following: audio and video playback request data, uplink streaming media traffic data, and generated bandwidth traffic data.

[0013] Optionally, the distributed streaming media cluster node further includes a data processing module;

[0014] The data processing module is used to clean the first flow data of the streaming media service to obtain the flow data after removing abnormal data;

[0015] It is also used to perform flow calculation based on the flow data after the abnormal data is removed and a preset flow algorithm to obtain second flow data, wherein the second flow data includes the flow data after the abnormal data is removed and intermediate data related to the flow.

[0016] Optionally, the distributed streaming media cluster node further includes a storage module;

[0017] The storage module is used to store the second flow data.

[0018] Optionally, the data processing module includes a data collection unit, a message forwarding channel, a message service unit, and a data processing unit;

[0019] The data collection unit is used to receive the first flow data of the streaming media service acquired by the log collection module;

[0020] Also used for uploading the first flow data of the streaming media service to the message service unit through the message forwarding channel;

[0021] The data processing unit is used to capture the first flow data of the streaming media service sent by the message service unit, and clean and calculate the flow data to obtain the second flow data.

[0022] In a second aspect, the present application provides a method for collecting streaming media data, which is applied to a system for collecting streaming media data, comprising:

[0023] Acquire first flow data of a streaming media service, and clean, calculate, and store the first flow data to obtain stored second flow data;

[0024] The second flow data is called.

[0025] Optionally, reading a log file of a streaming media service at a preset time interval;

[0026] According to the log file, first traffic data of the streaming media service is obtained, where the first traffic data includes one or more of the following: audio and video playback request data, uplink streaming media traffic data, and generated bandwidth traffic data.

[0027] Optionally, the cleaning, calculating and storing the first flow data to obtain the stored second flow data includes:

[0028] Cleaning the first flow data of the streaming media service to obtain flow data after removing abnormal data;

[0029] Performing flow calculation according to the flow data after the abnormal data is removed and a preset flow algorithm to obtain and store the second flow data;

[0030] The second flow data after storage is obtained, wherein the second flow data includes the flow data after the abnormal data is removed and the intermediate data related to the flow.

[0031] In a third aspect, the present application provides a streaming media data collection device, which is applied to a streaming media data collection system, including:

[0032] An acquisition module, used for acquiring first flow data of a streaming media service, and cleaning, calculating and storing the first flow data to obtain stored second flow data;

[0033] A calling module is used to call the second flow data.

[0034] In a fourth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by a processor, they are used to implement the streaming media data collection method as described in any one of the second aspects.

[0035] The present application provides a streaming media data collection system and a streaming media data collection method, wherein the streaming media data collection system includes: at least one business center and multiple distributed streaming media cluster nodes. The distributed streaming media cluster nodes are used to obtain the first flow data of the streaming media service, and clean, calculate and store the first flow data to obtain the stored second flow data. The business center is used to call the corresponding second flow data stored in multiple distributed streaming media cluster nodes. The present application cleans, calculates and stores the flow data through the distributed streaming media cluster nodes, thereby reducing the number of service nodes configured by the business center and reducing the resource overhead of the business center. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0037] Figure 1 A schematic diagram of the structure of an existing streaming media data collection system provided in this application;

[0038] Figure 2 A structural diagram of a streaming media data collection system provided in an embodiment of the present application;

[0039] Figure 3 A structural diagram of another streaming media data collection system provided in an embodiment of the present application;

[0040] Figure 4 A structural diagram of another streaming media data collection system provided in an embodiment of the present application;

[0041] Figure 5 A flow chart of a method for collecting streaming media data provided in an embodiment of the present application;

[0042] Figure 6 A schematic diagram of the structure of a streaming media data collection device provided in an embodiment of the present application.

[0043] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0044] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0045] In the description of the embodiments of the present application, terms such as "inside" and "outside" indicating directions or positional relationships are based on the directions or positional relationships shown in the drawings. This is merely for the convenience of description and does not indicate or imply that the device or component must have a specific orientation, be constructed and operated in a specific orientation. Therefore, it should not be understood as a limitation on the present application.

[0046] In addition, it should be noted that in the description of the embodiments of the present application, unless otherwise clearly specified and limited, the terms "connected" and "connection" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For those skilled in the art, the specific meanings of the above terms in the embodiments of the present application can be understood according to the specific circumstances.

[0047] With the rapid development of communication technology, audio and video have become an important means for users to obtain information. Audio and video can be videos from video websites, news and information videos, surveillance videos, live videos, etc. Streaming media technology provides technical support for audio and video playback. Streaming media technology refers to a technology that compresses a series of media data, sends the data in segments over the Internet, and transmits audio and video in real time over the Internet for viewing.

[0048] Streaming media technology transmits audio and video data stream files through the Internet Protocol (IP) network. Streaming media servers are divided into broadcast streaming media servers and on-demand streaming media servers. Streaming media servers are responsible for pushing audio and video data stream files to the broadband transmission network under the control of the operation management system. They can control the transmission bit rate and bandwidth of the continuous data stream. The data stream file of the streaming media does not need to download the entire file before playing. Instead, after a startup delay of several seconds, the content of the beginning part of the data stream file is read into the client memory first, and then the data stream file can be decompressed on the client and played. The remaining part of the data stream file continues to be downloaded in the background. Compared with the traditional download method, this streaming media transmission method of downloading and playing the data stream file not only greatly shortens the startup delay, but also greatly reduces the demand for system cache capacity. Therefore, it has been widely used in the field of audio and video.

[0049] As a distributed cluster, streaming media servers are deployed all over the country according to the actual needs of business broadcasting. The traffic data generated by streaming media services is collected to facilitate the analysis of streaming media service performance, bandwidth traffic recording, business system billing / calculation and other operations based on the traffic data, so as to better provide traffic for streaming media services.

[0050] In the prior art, flow data is usually collected through a streaming media data collection system. Figure 1 A structural diagram of an existing streaming media data collection system provided in this application, such as Figure 1 As shown, the existing streaming media data collection system includes a plurality of distributed streaming media cluster nodes 10 and at least one service center 11 .

[0051] Each distributed streaming cluster node 10 is independently deployed with a streaming collection module 101. The streaming collection module 101 is used to collect the traffic data generated by the distributed streaming cluster node 10 performing streaming services, and report the traffic data to the service center 11 in real time through the http interface based on preset rules.

[0052] The business center 11 is deployed with a data collection center module 111, a big data cleaning and computing center module 112, a data storage center module 113 and a data business application center module 114. The data collection center module 111 is used to provide a reporting interface, which can receive the traffic data reported by the streaming media acquisition module 101 in different distributed streaming media cluster nodes 10, and it can support high-concurrency and high-availability reporting services according to business needs. The big data cleaning and computing center module 112 is used to clean, complete, calculate and store all reported traffic data. The data storage center module 113 is used to store the reported traffic data and the intermediate data calculated based on the traffic data, and serves as the data source for the data business application center module 114. The data business application center module 114 is used to query and display the traffic data and intermediate data stored corresponding to the business needs according to business needs.

[0053] In this application, traffic data includes but is not limited to: audio and video playback request data, upstream streaming media traffic data, generated bandwidth traffic data, etc.

[0054] However, the existing technology business center is configured with many service nodes, which takes up a large amount of resource overhead. For example, the service nodes are highly concurrent, and the business center couples the flow data of multiple distributed streaming cluster nodes for processing (cleaning, calculation and storage), resulting in a large overhead. Moreover, when the business center processes a large amount of flow data, it cannot flexibly and dynamically expand the capacity, resulting in a large pressure on the use of resources in the business center.

[0055] Therefore, in view of the above technical problems in the prior art, the present application proposes a streaming media data collection system and a streaming media data collection method. The streaming media data collection system includes: at least one business center, multiple distributed streaming media cluster nodes, wherein the distributed streaming media cluster nodes clean, calculate and store the flow data, and the business center does not need to configure service nodes to perform the above processing on the flow data, thereby reducing the resource overhead of the business center.

[0056] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0057] Figure 2 A structural diagram of a streaming media data collection system provided in an embodiment of the present application is shown in FIG. Figure 2 As shown, it includes: at least one business center and multiple distributed streaming media cluster nodes.

[0058] The distributed streaming media cluster node is used to obtain first flow data of the streaming media service, and clean, calculate and store the first flow data to obtain stored second flow data.

[0059] Specifically,

[0060] For any distributed streaming media cluster node, the distributed streaming media cluster node includes a log collection module.

[0061] Its functions are:

[0062] The log file of the streaming media service is read at a preset time interval, and the first flow data of the streaming media service is obtained according to the log file. The first flow data includes but is not limited to the following: audio and video playback request data, uplink streaming media flow data, generated bandwidth flow data, etc.

[0063] In this embodiment, the method of obtaining flow data through log files reduces the high concurrency pressure of the business center in reporting the first flow data through http requests.

[0064] For any distributed streaming media cluster node, the distributed streaming media cluster node also includes a data processing module.

[0065] Its functions are:

[0066] The first traffic data of the streaming media service is cleaned to obtain traffic data after removing abnormal data, and traffic calculation is performed based on the traffic data after removing the abnormal data and a preset traffic algorithm to obtain second traffic data, wherein the second traffic data includes the traffic data after removing the abnormal data and intermediate data related to the traffic.

[0067] In the present application, other processing may also be performed on the first flow data, such as data completion processing, etc. The data used for completion may be obtained by interacting with other systems through the calling interface in the data processing module.

[0068] In this embodiment, the first flow data is cleaned and calculated by the data processing module of the distributed streaming media cluster node, thereby reducing the resource pressure required for the business center to perform data processing.

[0069] For any distributed streaming media cluster node, the distributed streaming media cluster node also includes a storage module.

[0070] Its functions are:

[0071] The second flow data is stored. The storage module may be a relational database such as MySQL, or other non-relational databases, etc. It is understood that the present application does not limit the type of database.

[0072] The business center is used to call the second flow data correspondingly stored in multiple distributed streaming media cluster nodes.

[0073] In this embodiment, there is no coupling between the second flow data stored in different distributed streaming media cluster nodes that are last called by the service center, and the second flow data stored in different distributed streaming media cluster nodes will not affect each other.

[0074] In addition, each distributed streaming cluster node processes traffic data, so that each distributed streaming cluster node can flexibly deploy traffic distribution according to the resource demand of the streaming business. Each distributed streaming cluster node can also dynamically expand capacity more flexibly according to the dynamic growth of business resource demand, making resource utilization more reasonable.

[0075] In the above-mentioned embodiment of the present application, the streaming media data collection system includes: at least one business center and multiple distributed streaming media cluster nodes. Among them, the distributed streaming media cluster node is used to obtain the first flow data of the streaming media service, and clean, calculate and store the first flow data to obtain the stored second flow data. The business center is used to call the corresponding second flow data stored in multiple distributed streaming media cluster nodes. This embodiment cleans, calculates and stores the flow data through the distributed streaming media cluster nodes, thereby reducing the service nodes configured by the business center and reducing the resource overhead of the business center.

[0076] Furthermore, based on the above embodiments, another streaming media data collection system is described through the following embodiments. Figure 3 A structural diagram of another streaming media data collection system provided in an embodiment of the present application is shown in FIG. Figure 3 As shown: including: at least one business center and multiple distributed streaming media cluster nodes.

[0077] For any distributed streaming media cluster node, it has been described in the above embodiments that the distributed streaming media cluster node includes a log collection module, a data processing module and a storage module.

[0078] The data processing module may also include a data collection unit, a message forwarding channel, a message service unit, and a data processing unit.

[0079] Its functions are:

[0080] The data collection unit is used to receive the first flow data of the streaming media service acquired by the log collection module.

[0081] Exemplarily, the data collection unit may be a lightweight log collector filebeat.

[0082] It is also used to upload the first flow data of the streaming media service to the message service unit through the message forwarding channel.

[0083] Exemplarily, the message forwarding channel may be the data collection engine logstash.

[0084] The message service unit may be a message system kafka.

[0085] It should be understood that the above examples are only for illustration and are not intended to limit the present application.

[0086] The data processing unit is used to capture the first flow data of the streaming media service sent by the message service unit, and clean and calculate the flow data to obtain second flow data.

[0087] In summary, the structure of the streaming media data collection system of the present application is as follows: Figure 4 As shown, Figure 4 A structural diagram of another streaming media data collection system provided in an embodiment of the present application. The streaming media data collection system includes a distributed streaming media cluster node and a business center, and the distributed streaming media cluster node includes a log collection module, a data processing module, and a storage module. The data processing module includes a data collection unit, a message forwarding channel, a message service unit, and a data processing unit.

[0088] For the specific functions of each module and unit, please refer to the above embodiments, and will not be repeated here to avoid redundancy.

[0089] In the above embodiment of the present application, the data processing module of the distributed streaming media cluster node may also include a data collection unit, a message forwarding channel, a message service unit, and a data processing unit. The first flow data is transmitted and processed through the above units or channels to obtain the second flow data, which reduces the number of service nodes that the business center needs to configure to transmit and process the first flow data, and reduces the resource overhead of the business center.

[0090] This application also provides a method for collecting streaming media data. Figure 5 A flow chart of a method for collecting streaming media data provided in an embodiment of the present application is shown as follows: Figure 5 As shown, the execution subject of the method is a streaming media data collection system, and the method includes the following steps:

[0091] S501: Acquire first flow data of a streaming media service, and clean, calculate, and store the first flow data to obtain stored second flow data.

[0092] The streaming media data collection system includes at least one business center and multiple distributed streaming media cluster nodes.

[0093] One possible implementation is:

[0094] For any distributed streaming media cluster node, the distributed streaming media cluster node reads the log file of the streaming media service at a preset time interval, and obtains the first flow data of the streaming media service according to the log file.

[0095] The first traffic data includes one or more of the following: audio and video playback request data, uplink streaming media traffic data, generated bandwidth traffic data, etc.

[0096] For the distributed streaming media cluster node, the first flow data of the streaming media service is cleaned to obtain the flow data after the abnormal data is removed.

[0097] The flow rate is calculated based on the flow rate data after the abnormal data is removed and a preset flow rate algorithm to obtain and store the second flow rate data.

[0098] The stored second flow data is obtained, wherein the second flow data includes the flow data after the abnormal data is removed and intermediate data related to the flow.

[0099] S502: Call the second flow data.

[0100] The business center calls the second flow data correspondingly stored in the multiple distributed streaming media cluster nodes.

[0101] In the above embodiments of the present application, please refer to the above embodiments for detailed implementation steps and technical effects. To avoid redundancy, they will not be repeated.

[0102] Figure 6 A structural diagram of a streaming media data collection device provided in an embodiment of the present application, such as Figure 6 As shown, the device includes an acquisition module 601 and a calling module 602.

[0103] The acquisition module 601 is used to acquire first flow data of the streaming media service, and clean, calculate and store the first flow data to obtain stored second flow data.

[0104] The calling module 602 is used to call the second flow data.

[0105] One possible implementation is that the acquisition module 601 is specifically configured to:

[0106] Read the log file of the streaming service at a preset interval.

[0107] According to the log file, first traffic data of the streaming media service is obtained, where the first traffic data includes one or more of the following: audio and video playback request data, uplink streaming media traffic data, and generated bandwidth traffic data.

[0108] In a possible implementation manner, the device further includes a processing module, which is used to:

[0109] The first flow data of the streaming media service is cleaned to obtain flow data after the abnormal data is removed.

[0110] The flow rate is calculated based on the flow rate data after the abnormal data is removed and a preset flow rate algorithm to obtain and store the second flow rate data.

[0111] The stored second flow data is obtained, wherein the second flow data includes the flow data after the abnormal data is removed and intermediate data related to the flow.

[0112] The streaming media data collection device provided in the embodiment of the present application can execute the streaming media data collection method in the above method embodiment, and its implementation principle and technical effect are similar, which will not be repeated here.

[0113] It should be noted that the above Figure 6 The division of the virtual modules shown is only a schematic diagram, and the present application does not limit the division of the modules and the naming of the modules.

[0114] The present application also provides a computer-readable storage medium, which may include: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, and other media that can store program codes. Specifically, the computer-readable storage medium stores computer execution instructions, and the computer execution instructions are used for the above-mentioned streaming media data collection method.

[0115] The present application also provides a computer program product, which includes an execution instruction or a computer program, and the execution instruction or the computer program is stored in a readable storage medium. At least one processor of the streaming media data collection system can read the execution instruction from the readable storage medium, and at least one processor executes the execution instruction so that the streaming media data collection system implements the streaming media data collection method provided by the above various embodiments.

[0116] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present application. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the following claims.

[0117] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A streaming media data collection system, characterized in that: include: At least one business center and multiple distributed streaming cluster nodes; The distributed streaming media cluster node is used to obtain first flow data of the streaming media service, and clean, calculate and store the first flow data to obtain stored second flow data; The business center is used to call the second traffic data correspondingly stored in the multiple distributed streaming media cluster nodes.

2. The streaming media data collection system according to claim 1, characterized in that: The distributed streaming media cluster node includes a log collection module; The log collection module is used to read the log files of the streaming media service at preset intervals; It is also used to obtain first traffic data of the streaming media service according to the log file, where the first traffic data includes one or more of the following: audio and video playback request data, uplink streaming media traffic data, and generated bandwidth traffic data.

3. The streaming media data collection system according to claim 2, characterized in that: The distributed streaming media cluster node also includes a data processing module; The data processing module is used to clean the first flow data of the streaming media service to obtain the flow data after removing abnormal data; It is also used to perform flow calculation based on the flow data after the abnormal data is removed and a preset flow algorithm to obtain second flow data, wherein the second flow data includes the flow data after the abnormal data is removed and intermediate data related to the flow.

4. The streaming media data collection system according to claim 3, characterized in that: The distributed streaming media cluster node also includes a storage module; The storage module is used to store the second flow data.

5. The streaming media data collection system according to claim 4, characterized in that: The data processing module includes a data collection unit, a message forwarding channel, a message service unit, and a data processing unit; The data collection unit is used to receive the first flow data of the streaming media service acquired by the log collection module; Also used for uploading the first flow data of the streaming media service to the message service unit through the message forwarding channel; The data processing unit is used to capture the first flow data of the streaming media service sent by the message service unit, and clean and calculate the flow data to obtain the second flow data.

6. A method for collecting streaming media data, characterized in that: The collection system used for streaming media data includes: Acquire first flow data of a streaming media service, and clean, calculate, and store the first flow data to obtain stored second flow data; The second flow data is called.

7. The method according to claim 6, characterized in that The obtaining of first flow data of the streaming media service includes: Read the log file of the streaming service at a preset interval; According to the log file, first traffic data of the streaming media service is obtained, where the first traffic data includes one or more of the following: audio and video playback request data, uplink streaming media traffic data, and generated bandwidth traffic data.

8. The method according to claim 7, characterized in that The cleaning, calculating and storing the first flow data to obtain the stored second flow data includes: Cleaning the first flow data of the streaming media service to obtain flow data after removing abnormal data; Performing flow calculation according to the flow data after the abnormal data is removed and a preset flow algorithm to obtain and store the second flow data; The second flow data after storage is obtained, wherein the second flow data includes the flow data after the abnormal data is removed and the intermediate data related to the flow.

9. A streaming media data collection device, characterized in that: The collection system used for streaming media data includes: An acquisition module, used for acquiring first flow data of a streaming media service, and cleaning, calculating and storing the first flow data to obtain stored second flow data; A calling module is used to call the second flow data.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by a processor, they are used to implement the streaming media data collection method according to any one of claims 6 to 8.

Images