Streaming media hotlinking early warning method based on big data

Abstract

The invention discloses a hotlinking early warning method based on big data streaming media, and the method comprises the following steps: real-time data monitoring: an ETL data collection module which comprises the steps: carrying out the classified collection and induction of eight types of data; data preprocessing and classification: a data computer learning module; an intelligent early warning mechanism: a basic data module; an early warning output module; in conclusion, compared with the prior art, existing hotlinking processing mainly focuses on shielding and rarely has early warning, the patent focuses on business production data, server resources and possible problems in machine learning regression algorithm estimation, early warning is focuses on, and the system has higher practicability and timeliness; compared with the prior art, the main creativity is that various scene data are generated in the watching process of a video APP, server resource occupation conditions and Spark real-time calculation are combined, possible problems are estimated by reasonably using a machine learning regression algorithm, risk information is positioned, and multidirectional combination is timely and efficient.

Description

Technical Field

[0001] The present invention relates to the field of emerging information technology, and in particular to a hotlink warning method based on big data streaming media. Background Art

[0002] During the video playback process of Tianyi Ultra HD APP, users watch the programs by subscribing to membership, and corresponding streaming media log records will be generated in the background. The link address of the video is fixed under certain conditions during the playback process. Some users on the Internet can play it by directly calling the link address of the member user.

[0003] This behavior of stealing links will greatly occupy server resources. In some cases, non-member users can view member users' rights and interests, causing losses to the company. Therefore, an early warning method is needed to detect similar problems in time and take effective measures. This method uses system basic data with the help of spark real-time calculation, combined with machine learning, to detect similar problems in time and issue early warnings. Summary of the invention

[0004] The purpose of the present invention is to provide a hotlink warning method based on big data streaming media, which uses the Spark real-time computing framework, big data processing, calling library function development program, and using the Python Sklearn learning tool to develop model clustering. Through various regression algorithms, combined with historical threshold information, blacklists, etc., through machine learning estimators, potential hotlink risk users, IPs, channels, programs and other dimensions can be discovered and generated warning information in a timely manner.

[0005] To achieve the above object, the present invention provides the following technical solutions:

[0006] A method for early warning of hotlinking based on big data streaming media includes the following steps:

[0007] S1: Real-time data monitoring: ETL data collection module, including eight categories of data classification, collection and summary;

[0008] S2: Data preprocessing and classification: data computing and machine learning module;

[0009] S3: Intelligent early warning mechanism: basic data module;

[0010] S4: warning output module;

[0011] S5: User behavior analysis module.

[0012] Preferably, the eight data classification collection and summary in S1 include user system access link log, user login log, order log, streaming media play log, search log, behavior log, payment table and server resource information.

[0013] Preferably, the user system access link log includes user account, URL address, IP address, business channel, access time and province.

[0014] Preferably, the user login log includes the login account, time, machine type UA, client version, network type, IP address and OS.

[0015] Preferably, the order log includes account number, time, channel, order method, ordered product, order triggering program, order price and order category.

[0016] Preferably, the streaming media playback log includes the playback account, start time, end time, duration, traffic, playback program, playback channel, playback IP address and client type.

[0017] Preferably, the data calculation machine learning module in S2 includes eight categories in the ETL data collection module: 8 access logs, 6 login logs, 6 order logs, 6 streaming logs, 1 search log, 1 behavior log, 5 payment tables, and a server resource curve, a total of 30 input reference indicators, set thresholds and score ratios for each indicator, use library analysis functions to process correlation information between indicators, use Python Sklearn machine learning tools, develop model clustering, call various regression algorithm functions, combine historical threshold information, blacklist, and generate warning information through machine learning estimators.

[0018] Preferably, the warning output of the basic data module in S3 adopts a database trigger. The current output mainly includes two channels: SMS and email. By calling data calculation and hotlink warning module, the SMS and email interfaces are called for the device number, IP, and channel information that reach the threshold, and the warning information is sent to the relevant operation and maintenance personnel.

[0019] Preferably, the warning output module adopts a database trigger, and currently outputs mainly two channels: SMS and email. By calling data calculation, the hotlink warning module calls the SMS and email interfaces for the device number, IP, and channel information that reach the threshold, and sends warning information to relevant operation and maintenance personnel.

[0020] Preferably, the user behavior analysis module collects normal user operation behaviors, constructs a behavior sequence library, and performs matching analysis with user behaviors in real-time information to determine whether there is a threat.

[0021] In summary, due to the adoption of the above technology, the beneficial effects of the present invention are:

[0022] 1. Compared with the existing technology, the main advantage is that the existing hotlink processing is mainly blocking, with few warnings. This patent focuses on business production data and server resources, and machine learning regression algorithms to predict possible problems, with warnings as the main approach, which is more practical and timely.

[0023] 2. Compared with existing technologies, the main creativity lies in: various scene data generated by video APP during the viewing process, combined with server resource usage, Spark real-time calculation, and reasonable use of machine learning regression algorithms to predict possible problems and locate risk information, with all-round combination to be timely and efficient.

[0024] 3. Improve the accuracy and efficiency of data analysis: Through the big data intelligent platform, data modeling and cutting-edge artificial intelligence technology can be used to effectively solve the problems of low efficiency in analyzing large and complex databases, and difficulty in ensuring accuracy and consistency. This method can analyze data quickly and accurately, and can achieve efficient data analysis even when the data volume is large and complex.

[0025] 4. Protect original content: Through the big data streaming media hotlink warning method, it is possible to effectively protect original content from being illegally copied and disseminated, safeguard the rights and interests of creators, and promote the legal use and dissemination of content.

[0026] 5. Improve user experience: The big data streaming hotlink warning method can reduce the impact of hotlinking on user experience. For example, using authentication to watch programs requires a username and password, and changing the username and password regularly can effectively prevent hotlinkers from obtaining content.

[0027] 6. Enhance data security: Through the big data streaming media hotlink warning method, data leakage and hotlinking can be effectively prevented. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] The accompanying drawings constituting a part of the present invention are used to provide a further understanding of the present invention, so that other features, purposes and advantages of the present invention become more obvious. The accompanying drawings of the exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the accompanying drawings:

[0029] Figure 1 It is a flow chart of the present invention. DETAILED DESCRIPTION

[0030] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work belong to the scope of protection of the present invention. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the invention claimed for protection, but merely represents the selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work belong to the scope of protection of the present invention.

[0031] In the description of the present invention, it is necessary to understand that the terms indicating orientation or positional relationship are based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention.

[0032] In the present invention, unless otherwise clearly specified and limited, the terms "installed", "connected", "connected", "fixed" and the like should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, it can be the internal connection of two elements or the interaction relationship between two elements. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to the specific circumstances of the specification.

[0033] The present invention provides Figure 1 The method for early warning of hotlinking based on big data streaming media includes the following steps:

[0034] S1: Real-time data monitoring, monitoring the network traffic data of streaming media services, real-time capture and analysis: ETL data collection module, including eight categories of data classification collection and summary;

[0035] (1) User system access link log: including user account, URL address, IP address, business channel, access time, province, etc.;

[0036] Take the account visits TOP1000, URL visits TOP100, access channels TOP50, access IP addresses TOP100, a total of 4 items.

[0037] (2) User login log: including login account, time, machine model UA, client version, network type, IP address, OS, etc.;

[0038] Take the login frequency greater than 5 times per hour, login channel TOP100, login model UA TOP50, login network type TOP10, login version TOP100, login OS TOP100, a total of 6 items.

[0039] (3) Order log: account number, time, channel, order method, ordered product, program that triggered the order, order price, order category (month, quarter, year), etc.;

[0040] Take the TOP50 triggered subscription programs, TOP50 call bill subscriptions, TOP50 self-subscriptions, TOP50 card subscriptions, TOP50 coupon subscriptions, and TOP50 subscription categories, a total of 6 items.

[0041] (4) Streaming media playback log: playback account, start time, end time, duration, traffic, playback program, playback channel, playback IP address, and client type;

[0042] Take the TOP50 triggered playback programs, TOP50 playback duration, TOP50 playback traffic, TOP50 playback programs, TOP50 live playback types, and TOP50 on-demand playback types, a total of 6 items.

[0043] (5) Search log: account, time, keyword; select the top 100 search keywords.

[0044] (6) Behavior log: account, time, behavior type, channel; select the top 100 behavior types.

[0045] (7) Payment table: account number, time, payment type, UA; TOP50 call bills, TOP50 points, TOP50 WeChat, TOP50 Alipay, TOP50 Best Pay, a total of 5 items.

[0046] (8) Server resource information: CPU, memory, bandwidth, I / O; take photos of CPU, memory, bandwidth, and I / O every minute to generate a server resource geometry curve.

[0047] S2: Data preprocessing and classification: cleaning and reducing traffic data, extracting features, and classifying them into normal events, abnormal events, and suspicious events: Data computing machine learning module:

[0048] There are eight categories in the ETL data collection module - 8 access logs, 6 login logs, 6 order logs, 6 streaming logs, 1 search log, 1 behavior log, 5 payment tables, and a total of 30 input reference indicators in the server resource curve; set thresholds and score ratios for each indicator, use library analysis functions to process the correlation information between indicators, use PythonSklearn machine learning tools, develop model clustering, call various regression algorithm functions, combine historical threshold information, blacklists, etc., and generate early warning information through machine learning estimators.

[0049] S3: Intelligent early warning mechanism, which provides security warnings for abnormal events, and identifies suspicious events through sandboxes to verify whether they are malicious hotlinks: basic data module;

[0050] The early warning output uses a database trigger. Currently, the main output channels are SMS and email. By calling data calculation, the hotlink warning module calls the SMS and email interfaces for the device number, IP, channel and other information that reaches the threshold, and sends the early warning information to the relevant operation and maintenance personnel.

[0051] S4: Warning output module, which uses the identified suspicious event features as new data sets to continuously optimize the classification model and improve the accuracy of warnings;

[0052] The early warning output uses database triggers. Currently, the output mainly consists of two channels: SMS and email. By calling data calculation, the hotlink warning module calls the SMS and email interfaces for the device number, IP, channel and other information that reaches the threshold, and sends the early warning information to the relevant operation and maintenance personnel.

[0053] S5: User behavior analysis module: collects normal user operation behaviors, builds a behavior sequence library, matches and analyzes the user behaviors in real-time information, and determines whether there is a threat.

[0054] Compared with the prior art, the present invention has the advantages and effects.

[0055] Compared with the existing technology, the main advantage is that the existing hotlink processing is mainly blocking, with few early warnings. This patent focuses on business production data and server resources, and machine learning regression algorithms to predict possible problems, with early warning as the main approach, which is more practical and timely.

[0056] Compared with existing technologies, the main creativity lies in: various scene data generated by video APP during the viewing process, combined with server resource usage, Spark real-time calculation, and reasonable use of machine learning regression algorithms to predict possible problems and locate risk information, and all-round combination is timely and efficient.

[0057] Example:

[0058] Take a warning for a certain IP address as an example: the preset threshold is 60. If the IP is in the TOP100 in the access log, it will get 10 points. If the login log exceeds 5 times per hour, it will get 10 points. If the IP is in the TOP50 in the playback log, it will get 10 points. If the IP is in the TOP50 in the behavior log, it will get 5 points. If the CPU and memory usage exceed 10%, they will get 20 points each. The total score of 75 is far higher than the threshold of 60. This will trigger the warning condition, send the corresponding risky IP to the operation and maintenance personnel, and take timely measures.

[0059] This method has the following two prediction directions:

[0060] 1. Monitoring and early warning of system operation anomalies. For example, based on whether the change of a certain data curve in the business data exceeds the set threshold, it can be determined whether there is a problem in the system.

[0061] 2. Predict the effectiveness of business development activities. Through historical data analysis, threshold setting, and the use of machine learning regression algorithms, valuable suggestions can be made for the promotion of new business activities to promote business development.

[0062] This method uses big data technology to monitor and warn of streaming hotlinking in real time, effectively improving the security of streaming services. At the same time, by continuously optimizing the warning model, it can better adapt to new hotlinking methods and provide strong protection for streaming services. It has the following benefits:

[0063] 1. Improve security: Through big data analysis, it is possible to monitor the use of streaming media data in real time, promptly detect and warn of hotlinking, effectively protect the security of streaming media resources, and reduce economic losses.

[0064] 2. Enhance management efficiency: The big data streaming media hotlink warning method can automatically and intelligently monitor and analyze data, reduce the burden of manual monitoring, and improve management efficiency.

[0065] 3. Optimize user experience: By warning and preventing hotlinking, the streaming media viewing experience of legitimate users can be guaranteed, avoiding service interruption or quality degradation caused by hotlinking.

[0066] 4. Promote technological innovation: The research, development and application of big data streaming media hotlink warning methods have promoted the application and innovation of big data, artificial intelligence and other technologies in the field of streaming media, and provided technical support for the healthy development of the streaming media industry.

Claims

1. A method for early warning of hotlinking based on big data streaming media, characterized by: The following steps are involved: S1: Real-time data monitoring: ETL data collection module, including eight categories of data classification, collection and summary; S2: Data preprocessing and classification: data computing and machine learning module; S3: Intelligent early warning mechanism: basic data module; S4: warning output module; S5: User behavior analysis module.

2. According to claim 1, a method for early warning of hotlinking based on big data streaming media is characterized in that: The eight data categories collected and summarized in S1 include user system access link log, user login log, order log, streaming media playback log, search log, behavior log, payment table and server resource information.

3. According to claim 2, a method for early warning of hotlinking based on big data streaming media is characterized in that: The user system access link log includes user account, URL address, IP address, business channel, access time and province.

4. According to claim 2, a method for early warning of hotlinking based on big data streaming media is characterized in that: The user login log includes login account, time, machine model UA, client version, network type, IP address and OS.

5. According to claim 2, a method for early warning of hotlinking based on big data streaming media is characterized in that: The order log includes the account number, time, channel, order method, ordered product, order triggering program, order price and order category.

6. According to claim 2, a method for early warning of hotlinking based on big data streaming media is characterized in that: The streaming media playback log includes the playback account, start time, end time, duration, traffic, playback program, playback channel, playback IP address and client type.

7. According to claim 1, a method for early warning of hotlinking based on big data streaming media is characterized in that: The data calculation machine learning module in S2 includes eight categories in the ETL data collection module: 8 access logs, 6 login logs, 6 order logs, 6 streaming logs, 1 search log, 1 behavior log, 5 payment tables, and a server resource curve, a total of 30 input reference indicators, set thresholds and score ratios for each indicator, use library analysis functions to process the correlation information between individual indicators, use Python Sklearn machine learning tools, develop model clustering, call various regression algorithm functions, combine historical threshold information, blacklist, and generate warning information through machine learning estimators.

8. The method for early warning of hotlinking based on big data streaming media according to claim 1 is characterized in that: The basic data module in S3 uses a database trigger for early warning output. Currently, the output mainly consists of two channels: SMS and email. By calling data calculation and hotlink early warning modules, the SMS and email interfaces are called for device numbers, IP addresses, and channel information that reaches the threshold, and early warning information is sent to relevant operation and maintenance personnel.

9. The method for early warning of hotlinking based on big data streaming media according to claim 1 is characterized in that: The warning output module uses a database trigger and currently outputs two main channels: SMS and email. By calling data calculation, the hotlink warning module calls the SMS and email interfaces for the device number, IP, and channel information that reach the threshold, and sends warning information to relevant operation and maintenance personnel.

10. The method for early warning of hotlinking based on big data streaming media according to claim 1 is characterized in that: The user behavior analysis module collects normal user operation behaviors, builds a behavior sequence library, and performs matching analysis with user behaviors in real-time information to determine whether there is a threat.

Images