A multi-element streaming media data acquisition method

Abstract

The application provides a kind of multi-element stream media data acquisition method, and the method includes the following steps: step 1, obtaining the stream media source address of the stream media data to be collected; step 2, verifying the compliance of the stream media source address; step 3, collecting the stream media data by distributing the stream media source address that has passed the verification; step 4, extracting the stream media feature information from the collected stream media data; and step 5, fusing the stream media data according to the stream media feature information to obtain structured stream media data, thereby completing the multi-element stream media data acquisition.

Description

TECHNICAL FIELD

[0001] The application relates to a data collection method, in particular to a multi-element stream media data collection method. BACKGROUND

[0002] With the continuous development and popularization of Internet technology, due to the low threshold of stream media platform publishing, the stream media platform has gradually become one of the main information publishing platforms. The characteristics such as 'explosiveness, blindness and groupness' of stream media data bring great challenges to supervision and management. In order to quickly and accurately obtain these information to effectively prevent and respond to related risks, it is necessary to use crawler means to collect and analyze the data on the stream media platform. By regularly collecting and monitoring the data on the stream media platform, the possible risk events can be tracked and warned in real time, and measures can be taken in time to respond and handle. According to the characteristics of the data on the stream media platform, such as high concurrency, massiveness and diversification, artificial intelligence and other technical means are needed for data processing and analysis. For example, natural language processing technology is used to classify and sentiment analyze the text content, and abnormal information and harmful information are identified. Image recognition technology is used to identify and monitor multimedia data such as videos and pictures, and possible violations and dangerous situations are found. Through continuous optimization and upgrading of technical means, the risks and challenges on the stream media platform can be better responded to, and a good network ecological environment can be maintained.

[0003] The traditional stream media collection method mainly acquires the content on the target website by simulating user behavior. For example, a web browser is used to simulate user access to the target website, and stream media data is downloaded to the local for analysis. However, this method has some problems. First, since many websites have adopted some interference technology, it is difficult to collect the above-mentioned stream media data if a simple simulation of user behavior is used to acquire data. Second, the collection efficiency cannot meet the requirements of the explosive growth of stream media data, and the maintenance cost is difficult to control. SUMMARY

[0004] The technical problem to be solved by the application is to provide a multi-element stream media data collection method to solve the problems of the prior art.

[0005] In order to solve the above technical problems, the application discloses a multi-element stream media data collection method, which comprises the following steps:

[0006] Step 1, acquiring the stream media source address of the stream media data to be collected, that is, collecting the stream media data information to be collected, and extracting the view source link of the stream media data, specifically including:

[0007] The source link address embedded in the source code of the target website where the to-be-collected streaming media data is located is searched by using a regular expression, the HTML source code, JavaScript code, CSS code and response message in the page of the target website are analyzed to obtain the source address information of the streaming media data, and the source address information is fused to generate a streaming media source address.

[0008] Step 2, compliance verification is performed on the streaming media source address, including:

[0009] Step 2-1, it is judged whether the streaming media source address is a specific data platform, if yes, step 2-2 is performed, otherwise, step 2-4 is performed.

[0010] The specific data platform is a source address generated by analyzing and fusing a response message.

[0011] Step 2-2, it is judged whether identity verification is required, if yes, step 2-3 is performed, otherwise, step 2-5 is performed.

[0012] Step 2-3, a dynamic rendering method is used to obtain a visitor identity, and step 2-5 is performed.

[0013] The dynamic rendering includes: constructing an open source headless browser library based on a Docker container technology, and starting more than one Docker container to run different browser instances to obtain complete dynamic data.

[0014] Step 2-4, a random request header is used, and step 2-6 is performed.

[0015] The random request header is a HTTP header information library, a new HTTP header information is randomly assembled when a request is made, and the HTTP header information library includes: randomly preset User-Agent, Referer, Cookie, Accept-Language and Connection information.

[0016] Step 2-5, a random device fingerprint is used, and step 2-6 is performed.

[0017] The random device fingerprint is a device fingerprint library, a new device information is randomly assembled for anti-crawling verification of the target website, and the device information is used to collect streaming media data.

[0018] The device fingerprint library at least includes: operating system type and version, browser type and version, screen resolution, time zone, language, IP address, CPU model and GPU model.

[0019] Step 2-6, determine whether a random IP is required, if so, execute step 2-7, otherwise execute step 2-8;

[0020] Step 2-7: Use dynamic random IP and execute steps 2-8;

[0021] The use of dynamic random IP includes: presetting an IP pool and regularly testing the IPs in the IP pool to maintain the validity and availability of the IPs in the IP pool.

[0022] Step 2-8: Determine whether liveness verification is required. If yes, proceed to step 2-9; otherwise, proceed to step 2-10.

[0023] Step 2-9: Use liveness verification and execute steps 2-10;

[0024] The liveness verification method uses a universal verification code recognition library to perform human-computer interaction to complete the verification request. Specific methods include: a puzzle-type slider verification method, a non-puzzle trial slider verification method, and a character verification method.

[0025] Among them, the puzzle slider verification method is as follows:

[0026] First, the image to be verified is converted into a grayscale image and binarized. The position of the slider gap in the image to be verified is determined through contour detection. The sliding trajectory is simulated by dynamic rendering to complete the puzzle-style slider verification.

[0027] The non-jigsaw-type slider verification method includes: directly completing the slider verification by simulating the sliding trajectory through dynamic rendering;

[0028] The character verification method includes: using a convolutional neural network to extract local features of the image to be verified, using a recurrent neural network to transform the feature sequence, using a fully connected layer to combine the above transformed features, and outputting the probability distribution corresponding to each character category in the image to be verified. The character category with the highest probability is used as the recognition result. The specific method is as follows:

[0029] Input the image to be verified χ into the first convolution layer, and after the convolution operation, obtain the feature map h (1) , and then perform nonlinear transformation on the feature map through the ReLU activation function to obtain a new activation map a (1) , the new activation map a (1) Input to the first pooling layer, the pooling operation adopts maximum pooling; after the pooling operation, the new feature map h is obtained (2) After a preset number of convolution, activation and pooling operations, the final feature map h is obtained. f ;

[0030] The final feature map h fThe input is input into the LSTM network, and sequence information is gradually extracted through multiple LSTM units to obtain a one-dimensional vector h of length T t The vector h output by the LSTM network is input into a fully connected layer containing more than one neuron t The output s of the fully connected layer is represented as:

[0031] s = wh t +b

[0032] Where W is a weight matrix and b is a bias term

[0033] The softmax operation is performed on the output s of the fully connected layer to obtain the probability distribution p of each character category i The method is as follows:

[0034]

[0035] Where i and j represent the i-th and j-th character categories, taking values in the range of 0, 1, …, k-1, k is the number of character categories, exp() is used to convert the output vector into a probability distribution vector, i.e. each element is exponentiated and normalized by the sum of all elements as the divisor, in this formula, each element s i in the output s of the fully connected layer is subjected to the exp() operation

[0036] The character category with the maximum probability is selected as the recognition result y, represented as follows:

[0037] y = afgmax i p i

[0038] Step 2-10, adopt dynamic control request frequency, and perform step 2-11;

[0039] The dynamic control request frequency specifically includes:

[0040] A minimum time interval Tmin and a maximum time interval Tmax are set, and the time interval Δt is calculated according to the number of pages N of the target website and the page type T of the target website that have been collected

[0041] If the time interval Δt exceeds the maximum time interval Tmax, it is adjusted to Tmax; if the time interval Δt is less than the minimum time interval Tmin, it is adjusted to Tmin, i.e. the following method cf is used for adjustment:

[0042] cf = Min(Max(Tmin, f(N, T) * k), Tmax)

[0043] Wherein, Min() and Max() functions represent taking minimum value and maximum value respectively, k is a coefficient for adjusting frequency; function f(N, T) represents the request frequency calculated according to the page number N and target page type T that have been collected.

[0044] Step 2-11, completing compliance verification, entering step 3, starting to collect streaming media data.

[0045] Step 3, for the streaming media source address that passes the verification, performing distributed collection to obtain streaming media data.

[0046] Step 4, for the collected streaming media data, performing streaming media feature information extraction.

[0047] Step 5, according to the streaming media feature information, performing streaming media data fusion to obtain structured streaming media data, and completing the multi-element streaming media data collection.

[0048] Beneficial effects:

[0049] The application collects streaming media data in a distributed and parallel manner, greatly improves the collection efficiency, avoids the situation of collecting invalid data and being unable to collect streaming media data through the method in the application, improves the accuracy and integrity of the collection result, and improves the universality and scalability of the streaming media collection system. BRIEF DESCRIPTION OF DRAWINGS

[0050] The above and / or other aspects of the application will become more apparent by describing in detail the preferred embodiments thereof with reference to the attached drawings.

[0051] Figure 1 It is a method architecture diagram of the application.

[0052] Figure 2 It is a general flow chart of the compliance verification of the streaming media source address in the method of the application. DETAILED DESCRIPTION

[0053] The application provides a multi-element streaming media collection method to solve the technical interference problem and the collection efficiency problem of the traditional streaming media view collection method, realizes automatic acquisition of multi-streaming media platform view source addresses, avoids security protection detection by using verification request technology, and realizes distributed multi-node parallel download, so as to realize diversified view collection, including the following steps:

[0054] Step 1, acquiring a streaming media source address of streaming media data to be collected.

[0055] Step 2, performing compliance verification on the streaming media source address.

[0056] Step 3, performing distributed collection of streaming media data.

[0057] Step 4, stream media feature information extraction;

[0058] Step 5, stream media data fusion;

[0059] The multi-element stream media collection method in the application mainly includes view source address automatic acquisition, stream media platform verification, and distributed multi-node collection.

[0060] The view source address automatic acquisition mainly includes: establishing a video analysis library for different stream media platforms, on the one hand, finding the source link with.mp4 / .m3u8 / .mov / .m4v as the suffix embedded in the HTML source code through the regular expression method, and on the other hand, analyzing the HTML file, JavaScript file, CSS file, and response message of the target site through the reverse engineering technology to obtain the information related to the stream media source address, integrating the related information to obtain the complete stream media source address.

[0061] The stream media platform compliance verification mainly includes: establishing a verification request library for different stream media platforms, on the one hand, preventing being identified as a crawler by the target website through the User-Agent, Referer, Cookie, Accept-Language, Connection, and other information of the pseudo-random request header, when facing the collection target needing a specific Cookie, adopting the dynamic rendering technology to generate a tourist Cookie pool in batches, for the verification code in the simulated login, adopting the image recognition technology to accurately identify the target gap and coordinates, completing the verification through the trajectory simulation technology to obtain the Cookie, and on the other hand, using the high-anonymity proxy to randomly switch the proxy IP and the random request time interval through the proxy server to simulate the operation behavior of human beings to reduce the probability of being detected.

[0062] The steps of the compliance verification of the stream media source address mainly include:

[0063] Step 2-1, judging whether it is a specific data platform according to the stream media source address, if yes, executing step 2-2, otherwise executing step 2-4;

[0064] The specific data platform is the source address generated by analyzing and fusing the response message.

[0065] Step 2-2, judging whether the identity needs to be verified, if yes, executing step 2-3, otherwise executing step 2-5;

[0066] Step 2-3, adopting the method of dynamic rendering to obtain the tourist identity, and executing step 2-5;

[0067] The dynamic rendering includes: constructing an open-source headless browser library based on a Docker container technology, starting more than one Docker container to run different browser instances, and obtaining complete dynamic data.

[0068] Step 2-4, random request header is used, and step 2-6 is performed.

[0069] The random request header is used, that is, a new HTTP header information is randomly assembled in the request header information library for the request, and the HTTP header information library includes: randomly preset User-Agent, Referer, Cookie, Accept-Language and Connection information.

[0070] Step 2-5, random device fingerprint is used, and step 2-6 is performed.

[0071] The random device fingerprint is used, that is, a new device information is randomly assembled in the device fingerprint library for the target website anti-crawling verification, and the device information is used for collecting streaming media data.

[0072] The device fingerprint library includes at least: operating system type and version, browser type and version, screen resolution, time zone, language, IP address, CPU model and GPU model.

[0073] Step 2-6, it is judged whether random IP is needed, if yes, step 2-7 is performed, otherwise step 2-8 is performed.

[0074] Step 2-7, dynamic random IP is used, and step 2-8 is performed.

[0075] The dynamic random IP includes: presetting an IP pool and periodically detecting the IP in the IP pool to maintain the validity and availability of the IP in the IP pool.

[0076] Step 2-8, it is judged whether live body verification is needed, if yes, step 2-9 is performed, otherwise step 2-10 is performed.

[0077] Step 2-9, live body verification is used, and step 2-10 is performed.

[0078] The live body verification is used, that is, a general verification code recognition library is used for human-computer interaction operation to complete the verification request, and the specific method includes: a puzzle type slider verification method, a non-puzzle type slider verification method and a character verification method.

[0079] The puzzle type slider verification method is as follows:

[0080] First, the image to be verified is converted into a grayscale image and binarized. The position of the slider gap in the image to be verified is determined through contour detection. The sliding trajectory is simulated by dynamic rendering to complete the puzzle-style slider verification.

[0081] The non-jigsaw-type slider verification method includes: directly completing the slider verification by simulating the sliding trajectory through dynamic rendering;

[0082] The character verification method includes: using a convolutional neural network to extract local features of the image to be verified, using a recurrent neural network to transform the feature sequence, using a fully connected layer to combine the above transformed features, and outputting the probability distribution corresponding to each character category in the image to be verified. The character category with the highest probability is used as the recognition result. The specific method is as follows:

[0083] Input the image to be verified χ into the first convolution layer, and after the convolution operation, obtain the feature map h (1) , and then perform nonlinear transformation on the feature map through the ReLU activation function to obtain a new activation map a (1) , the new activation map a (1) Input to the first pooling layer, the pooling operation adopts maximum pooling; after the pooling operation, the new feature map h is obtained (2) After a preset number of convolution, activation and pooling operations, the final feature map h is obtained. f ;

[0084] The final feature map h f Input into the LSTM network, through the multi-layer LSTM unit, gradually extract the sequence information to obtain a one-dimensional vector h of length T t , the vector h output by the LSTM network t Input to the fully connected layer, which contains more than one neuron; the output s of the fully connected layer is expressed as:

[0085] s=wh t +b

[0086] Among them, W is the weight matrix and b is the bias term;

[0087] Perform a softmax operation on the output s of the fully connected layer to obtain the probability distribution p of each character category i , the method is as follows:

[0088]

[0089] where i and j represent the i-th and j-th character class, taking values in the range 0, 1, …, k-1, k is the number of character classes, exp() is used to convert the output vector into a probability distribution vector, i.e., exponentiating each element and normalizing by the sum of all elements, in this formula, each element s in the output of the full connection layer s i performing an exp() operation;

[0090] selecting the character class with the largest probability as the recognition result y, denoted as follows:

[0091] y = argmax i pi

[0092] Step 2-10, adopt dynamic control request frequency, and perform step 2-11;

[0093] The dynamic control request frequency specifically includes:

[0094] Set a minimum time interval Tmin and a maximum time interval Tmax, calculate the time interval Δt according to the number of pages N of the target website and the page type T of the target website that have been collected;

[0095] If the time interval Δt exceeds the maximum time interval Tmax, adjust it to Tmax; if the time interval Δt is less than the minimum time interval Tmin, adjust it to Tmin, that is, adjust it by the following method cf:

[0096] cf = Min(Max(Tmin, f(N, T) * k), Tmax)

[0097] Where Min() and Max() functions represent taking the minimum and maximum values respectively, k is a coefficient for adjusting the frequency; the function f(N, T) represents the request frequency calculated according to the number of pages N and the target page type T that have been collected.

[0098] Step 2-11, complete compliance verification, enter step 3, and start collecting streaming media data.

[0099] The distributed multi-node collection mainly includes: establishing a complete distributed scheduling, monitoring mechanism, task deduplication mechanism, through the CacheCloud node monitoring component, node failure and network flash can be found in time, and the task is automatically transferred to other healthy nodes for running, flexible strategy configuration function, different load balancing can be set according to actual demand, such as intelligent scheduling based on node load, network bandwidth, geographical location factors, so as to improve the collection efficiency and stability to the greatest extent. The task deduplication adopts the signature-based method to reduce the dimension of URL, converts the URL into a fixed length string through digital signature, and maps the digital signature to an array through multiple hash functions, so as to greatly improve the deduplication efficiency and accuracy.

[0100] Embodiments:

[0101] The distributed stream media crawler system of the application extracts source link and performs behavior camouflage on massive stream media data, wherein the system mainly involves: continuously acquiring original material data according to the crawler system and preliminarily filtering the data, extracting view source link, verifying compliance, extracting media feature information, and storing data. The related storage components are composed of CacheCloud, Kafka, Elasticsearch, HBase and MySQL, the original data stream is uniformly gathered through the CacheCloud cache service, the original collected data is preliminarily deduplicated in the process, the view resource source link is automatically acquired, the media feature information is extracted through the camouflage view crawler, the stream media feature information is input into the big data platform search engine Elasticsearch and the structured HBase table through Kafka in the downstream, the stream media file data is sent to the view storage server through FTP, and the MySQL is used for recording specific tag information and configuration data in the processing process.

[0102] As shown in Figure 1 , the application provides a multi-element stream media collection method, which comprises:

[0103] Step 1, continuously acquire original collection data through the crawler system, and preliminarily filter the data: gather the original collection data through the CacheCloud cache service, the service layer needs to integrate related load balancing components to ensure its high availability, extract the view source link, on the one hand, use regular expressions to find the source link embedded in the HTML source code with.mp4 / .m3u8 / .mov / .m4v / .flv as the suffix, realize the extraction of the source link of the commonly used media platform, on the other hand, analyze the HTML file, JavaScript file, CSS file, and response message of the target site through reverse engineering technology to obtain the necessary information of the streaming media source address, integrate the information to generate the source link, realize the extraction of the source link of the specific media platform, and through the source link extraction of the two methods, realize the coverage of commonly used streaming media platforms at home and abroad, some examples are as follows:

[0104]

[0105]

[0106] Step 2, compliance verification, as shown in Figure 2 , mainly includes: random request header, random device fingerprint, dynamic control frequency, live body verification, dynamic rendering, and dynamic random IP. Among them, the random request header can avoid frequent requests with the same request header, and the random information includes: browser identifier (User-Agent), language preference setting (Accept-Language), encoding preference setting (Accept-Encoding), connection option (Connection), and reference page (Referer). Random device fingerprint can avoid frequent requests with the same device and browser, and the random information includes: operating system type and version, browser type and version, screen resolution, time zone and language, IP address, CPU model and GPU model, etc. The device fingerprint generated by combining multiple information makes the fingerprint more real. Dynamic control frequency can avoid giving the collection target too much burden, first send requests with a small frequency, record the response time, and adjust the access frequency according to the response time. Live body verification can avoid the target requiring human-computer interaction operation leading to collection failure, common verification includes: puzzle test slider verification, non-puzzle test slider verification, and character verification. Dynamic rendering can avoid the inability to obtain dynamic data on the page, and through simulating real user behavior makes the data collection more comprehensive and stable. Dynamic random IP can avoid excessive use of one IP to obtain data leading to ban, through dynamically and randomly switching the request IP makes the crawler more anonymous, thereby improving the stability and reliability of the crawler. Combine these technical means to build a more perfect compliance verification system, and ensure that the crawler can efficiently and stably collect data.

[0107] Step 2-1, User-Agent generation in random request header, specific implementation as follows:

[0108]

[0109]

[0110] Step 2-2, dynamic control frequency, set a minimum time interval Tmin and maximum time interval Tmax, then according to the number of pages N and target page type T, calculate a time interval Δt. Where, Δt represents the increment of time interval, that is, according to the number of pages crawled and target page type and other factors to calculate the appropriate time interval, for example, some websites may set access frequency limit, require in a certain time can only send a certain number of requests. Then, according to this limit to calculate the appropriate Δt, to ensure that it does not exceed the access frequency limit. If Δt exceeds Tmax, it needs to be adjusted to Tmax; If Δt is less than Tmin, it needs to be adjusted to Tmin. In this way, it can ensure that the crawler program will not crawl pages too fast, at the same time, also will not be too slow to lead to low efficiency.

[0111] cf = Min(Max(Tmin, f(N, T) * k), Tmax)

[0112] Where, Min() and Max() function respectively represent the minimum value and maximum value, Tmin and Tmax are the minimum time interval and maximum time interval allowed, k is a coefficient, used to adjust the frequency.

[0113] Function f(N, T) represents the request frequency calculated according to the number of pages N and target page type T, specific implementation can be different according to different situations.

[0114] When the number of pages to be crawled is small, a fixed time interval can be used as the frequency;

[0115] When the number of pages to be crawled is large, the frequency can be calculated according to the proportion of the number of pages N and the total number of pages M, the formula is as follows:

[0116] f(N, T) = (M - N) / M

[0117] In this way, it can ensure that the collection speed is faster at the beginning, and when a large number of pages have been crawled, the collection speed will gradually decrease.

[0118] Step 2-3, random device fingerprint, specific implementation as follows:

[0119]

[0120]

[0121] Step 2-4, live verification, general verification code recognition library, the specific implementation is as follows:

[0122] Puzzle test slider verification, through image processing tool OpenCV to identify the gap coordinates, dynamic rendering technology to simulate the sliding track of real user behavior to complete the verification:

[0123]

[0124]

[0125] Among them, the non-puzzle test slider verification can directly use dynamic rendering technology to simulate the sliding track of real user behavior to complete the verification.

[0126] Character verification, take image x as an example, input into the first convolutional layer, where the convolution kernel size is 5x5, the step is 1, and the depth is 32. After convolution operation, the feature map h (1) , the size is 24x24x32, and the feature map is nonlinearly transformed by ReLU activation function to get the new activation map a (1) , input the activation map into the first pooling layer, the pooling operation uses maximum pooling, the pooling kernel size is 2x2, and the step is 2. After the pooling operation, the new feature map h (2) , the size is 12x12x32, after multiple convolution, activation and pooling operations, the final feature map h f , the size is 1x6x128.

[0127] The feature map h f is input into the LSTM network, through multiple LSTM units, the sequence information is gradually extracted, and a one-dimensional vector h t of length T is obtained, which is input into the fully connected layer. The output of the fully connected layer can be represented as:

[0128] s=wh t +b

[0129] Where W is the weight matrix and b is the bias term

[0130] The output of the fully connected layer is subjected to softmax operation to get the probability distribution of each character category:

[0131]

[0132] Where i represents the character category, and the value range is 0, 1,..., k-1

[0133] The character category with the maximum probability is selected as the recognition result:

[0134] y = argmax i p i

[0135] In practical applications, it can also be adjusted and optimized according to the needs

[0136] Step 2-5, dynamic rendering, based on Docker container technology to build open source headless browser library Browserless, start multiple Docker containers to run different browser instances, use Docker Compose to manage and orchestrate these containers, distribute requests to different browser instances through load balancing, and use Prometheus to monitor the performance of each browser instance and the status of the entire cluster.

[0137] Step 2-6, dynamic random IP, based on CacheCloud caching service as IP pool storage medium, by using multiple different proxy IP service providers to improve the stability and availability of the proxy pool, regularly detect the IP in the proxy pool to determine its availability. When the IP is invalid, it needs to be removed from the proxy pool and new IP is obtained from the proxy IP service provider to supplement the proxy pool, maintaining the effectiveness and availability of the IP in the proxy pool, and achieving automatic detection and maintenance of the proxy pool.

[0138] Step 3, distributed collection of streaming media data, including source link task aggregation, node scheduling, node monitoring, task deduplication, etc. Modules include:

[0139] Step 3-1, source link task aggregation, CacheCloud caching service, aggregation of collection tasks.

[0140] Step 3-2, node scheduling, using Celery framework Worker process and Beat process to complete node task distribution and scheduling, where each crawler node uses compliance verification technology such as random request header, random device fingerprint, and random IP to disguise collection requests before collection, and dynamically controls request frequency during collection through frequency calculation, while adapting to multiple compliance verification technologies in the compliance verification module according to response messages.

[0141] Step 3-3, node monitoring, start CacheCloud monitoring module, can see the health status and monitoring index information of all nodes, preset alarm rules, once an exception occurs in a node, automatically send email or SMS notification to collection personnel.

[0142] Step 3-4, task deduplication, convert URL to a digital signature, use hash functions, encryption functions, etc. to convert URL to a fixed length digital string, define a hash family H = {h1, h2,..., hn} composed of multiple hash functions, where each hash function maps the digital signature to an integer value within a fixed range. For each URL, apply each hash function in the hash family to its digital signature in turn to get n integer values, and store these values in an array. Traverse the array of all URLs and find the minimum value calculated by the hash function for all URLs. For any two URLs, if their digital signatures produce the same minimum value in at least one hash function calculation, they are considered similar, thus deduplicating the task URL.

[0143] Step 4, streaming media feature element extraction: identify elements such as title, media type, publication time, author, etc. from the original Html page, including:

[0144] Step 4-1, title recognition, get the original Html page's web page Title as org_title, recursively traverse the preprocessed Tag text node to get the <title>The text set list_title of the label, the sim value of each text content of org_title and list_title is calculated and the maximum sim is taken as the title, and the specific implementation is as follows:< / title>

[0145] soup = BeautifulSoup(html, 'html.parser')

[0146] org_title = soup.title.string.strip()

[0147] def get_title_text(tag):

[0148] title_texts = []

[0149] for child in tag.children:

[0150] if child.name == 'title':

[0151] title_texts.append(child.string)

[0152] else:

[0153] title_texts.extend(get_title_text(child))

[0154] return title_texts

[0155] list_title = get_title_text(soup)

[0156] max_sim = 0

[0157] best_title = ""

[0158] for title in list_title:

[0159] sim = difflib.SequenceMatcher(None, org_title, title).ratio()

[0160] if sim > max_sim:

[0161] max_sim = sim

[0162] best_title = title

[0163] Step 4-2, media type identification, using regular matching method to extract media type from response message.

[0164] Step 4-3, publication time identification, using TimeNLP model to extract from the text between the title and the body, if failed, then expand the radius step by step from the title node to find the time, the radius threshold is 100 characters.

[0165] Step 4-4, author extraction, using regular matching method to extract the text of the author node.

[0166] Step 5, streaming media data fusion: structured media data is uniformly stored in big data platform for persistence, big data storage nodes are mainly composed of Kafka, ES and HBase, data stream is delivered to ES and HBase through Kafka component, including:

[0167] Step 5-1, push to Kafka cluster, push the extracted media data to Kafka in JSON format, use File Transfer Protocol (FTP) to gather the streaming media data of each node to the file storage server.

[0168] Step 5-2, push to big data storage node, Kafka consumer accesses ES full-text search cluster and Hbase columnar storage cluster, mainly used for subsequent fusion data retrieval and storage, thus obtaining the fusion of the whole media data.

[0169] In a specific implementation, the present application provides a computer storage medium and a corresponding data processing unit, wherein the computer storage medium is capable of storing a computer program that, when executed by the data processing unit, executes the invention of a multi-streaming media data acquisition method provided by the present invention and some or all of the steps in each embodiment. The storage medium may be a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0170] Those skilled in the art can clearly understand that the technical solutions in the embodiments of the present invention can be implemented by means of computer programs and their corresponding general hardware platforms. Based on this understanding, the technical solutions in the embodiments of the present invention, in essence or in other words, the part that contributes to the prior art, can be embodied in the form of a computer program, i.e., a software product. The computer program software product can be stored in a storage medium and includes a number of instructions for enabling a device including a data processing unit (which can be a personal computer, server, single-chip microcomputer, MUU or network device, etc.) to execute the methods described in various embodiments of the present invention or certain parts of the embodiments.

[0171] The present invention provides a method and approach for collecting multi-streaming media data. There are numerous methods and approaches for implementing this technical solution. The above is merely a preferred embodiment of the present invention. It should be noted that those skilled in the art may make improvements and modifications without departing from the principles of the present invention, and such improvements and modifications are also within the scope of protection of the present invention. Any components not specified in this embodiment may be implemented using existing technologies.

Claims

1. A method for collecting multi-element streaming media data, characterized in that, The method comprises the following steps: Step 1, obtaining a streaming media source address of streaming media data to be collected; Step 2, verifying the compliance of the streaming media source address; Step 3, collecting the streaming media data from the streaming media source address that passes the verification in a distributed manner; Step 4, extracting streaming media feature information from the collected streaming media data; Step 5, fusing the streaming media data according to the streaming media feature information to obtain structured streaming media data, thereby completing the multi-element streaming media data collection; In step 2, the compliance verification of the streaming media source address comprises: Step 2-1, judging whether the streaming media source address is a specific data platform, if yes, executing step 2-2, otherwise executing step 2-4; Step 2-2, judging whether identity verification is needed, if yes, executing step 2-3, otherwise executing step 2-5; Step 2-3, obtaining the visitor identity by dynamic rendering, and executing step 2-5; Step 2-4, using a random request header, and executing step 2-6; Step 2-5, using a random device fingerprint, and executing step 2-6; Step 2-6, judging whether a random IP is needed, if yes, executing step 2-7, otherwise executing step 2-8; Step 2-7, using a dynamic random IP, and executing step 2-8; Step 2-8, judging whether live body verification is needed, if yes, executing step 2-9, otherwise executing step 2-10; Step 2-9, using live body verification, and executing step 2-10; Step 2-10, using dynamic control of request frequency, and executing step 2-11; Step 2-11, completing the compliance verification, and executing step 3 to start collecting the streaming media data; In step 2-9, the live body verification is performed by using a general verification code recognition library to complete the verification request through human-computer interaction, and the specific method comprises a puzzle slider verification method, a non-puzzle slider verification method and a character verification method; In the puzzle slider verification method, the specific steps are as follows: First, convert the image to be verified into a grayscale image and perform binaryzation processing, determine the slider gap position in the image to be verified through contour detection, and complete the puzzle slider verification through dynamic rendering simulation of the sliding track; The non-puzzle slider verification method comprises directly simulating the sliding track through dynamic rendering to complete the slider verification; The character verification method comprises extracting local features of the image to be verified by using a convolutional neural network, converting the feature sequence by using a recurrent neural network, combining the converted features together by using a fully connected layer, and outputting the probability distribution of each character category in the image to be verified, and the character category with the maximum probability is taken as the recognition result, and the specific method is as follows: The to-be-verified image χ is input into a first convolutional layer, and after convolution operation, a feature map h is obtained (1) , and the feature map is nonlinearly transformed by a ReLU activation function to obtain a new activation map a (1) , and the new activation map a (1) is input into a first pooling layer, and a maximum value pooling is adopted in the pooling operation; after the pooling operation, a new feature map h (2) is obtained, and after the preset number of convolution, activation and pooling operations, a final feature map h f is obtained; The final feature map h f Input into the LSTM network, through the multi-layer LSTM unit, gradually extract the sequence information to obtain a one-dimensional vector h of length T t , the vector h output by the LSTM network t Input to the fully connected layer, which contains more than one neuron; the output s of the fully connected layer is expressed as: s = wh t + b Wherein, W is a weight matrix, and b is a bias term; The softmax operation is performed on the output s of the full connection layer to obtain the probability distribution p of each character category i The method is as follows: where i and j represent the i-th and j-th character class, taking values in the range 0, 1,..., k-1, k being the number of character classes, exp() is used to convert the output vector into a probability distribution vector, i.e., exponentiating each element and normalizing by the sum of all elements as a divisor, in this formula each element s of the output of the fully connected layer s i an exp() operation is performed; The character category with the maximum probability is selected as the recognition result y, and is represented as follows: y = argmax i p i ; In step 2-10, the dynamic control of request frequency comprises: Set a minimum time interval Tmin and a maximum time interval Tmax, calculate the time interval Δt according to the number N of pages of the target website and the page type T of the target website; If the time interval At exceeds the maximum time interval Tmax, it is adjusted to Tmax; if the time interval At is less than the minimum time interval Tmin, it is adjusted to Tmin, that is, the following method cf is used for adjustment: cf = Min(Max(Tmin, f(N, T) * k), Tmax) Wherein, the Min() and Max() functions represent taking the minimum value and the maximum value respectively, k is a coefficient for adjusting the frequency; the function f(N, T) represents the request frequency calculated according to the number N of pages already collected and the target page type T.

2. The method of claim 1, wherein the plurality of streams are collected simultaneously. The flow media source address of the flow media data to be collected in step 1, that is, the flow media data information to be collected is converged, and the view source link extraction of the flow media data is carried out, which specifically includes: The source link address embedded in the source code of the target website where the flow media data to be collected is located is searched in the form of a regular expression, the HTML source code, JavaScript code, css code and response message in the page of the target website are analyzed, the source address information of the flow media data is obtained, and the above information is fused to generate the flow media source address.

3. The method of claim 2, wherein the step of collecting the multi-element streaming media data is performed by a plurality of data collectors. The dynamic rendering in steps 2-3 includes: based on the Docker container technology, an open source headless browser library is constructed, and more than one Docker container is started to run different browser instances, so as to obtain complete dynamic data.

4. The method of claim 3, wherein the step of collecting the multi-element streaming media data is performed by a plurality of servers. The random request header in step 2-4, that is, a new HTTP header information is randomly assembled for request in the HTTP header information library; the HTTP header information library includes: randomly preset User-Agent, Referer, Cookie, Accept-Language and Connection information.

5. The method of claim 4, wherein the step of collecting the multi-element streaming media data is performed by a plurality of servers. The random device fingerprint in step 2-5, that is, a new device information is randomly assembled for collecting flow media data in the device fingerprint library for anti-crawling verification of the target website. The device fingerprint library at least includes: operating system type and version, browser type and version, screen resolution, time zone, language, IP address, CPU model and GPU model.

6. The method of claim 5, wherein the step of collecting the multi-element streaming media data is performed by a plurality of data collectors. The dynamic random IP in step 2-7 includes: presetting an IP pool and periodically detecting the IP in the IP pool to maintain the effectiveness and availability of the IP in the IP pool.

7. The method of claim 6, wherein the step of collecting the multi-element streaming media data is performed by a plurality of servers. The specific data platform in step 2-1, that is, the source address generated by analyzing and fusing the response message is needed.

Images