A method for quickly obtaining user download hotspot data

Abstract

The application provides a quick acquisition method for user downloading hotspot data, comprising the following steps: S1: when the user is online normally, a packet capturing server mirrors and copies user data stream and binds the physical network card of the user by using netmap; S2: the packet capturing server intercepts all message information in the user data stream and uses hyperscan to quickly scan all the message information, and reorganizes the message information related to file, audio and video downloading to form reorganized urls; S3: the packet capturing server calls a record table and a cache table in a web management machine and compares with the reorganized urls, if the record table is not recorded, the reorganized urls are subjected to flow statistics and written into the cache table, then the reorganized urls meeting the requirements are written into the record table and the ATS server is informed of the source information, if the record table is recorded, the flow statistics is not performed and the packet capturing server is directly informed to send a redirection status code to the user. The application solves the problems of long transmission time, low timeliness and hit rate and repeated cache in the prior art.

Description

Technical Field

[0001] This invention relates to the field of intelligent network technology, and in particular to a method for rapidly acquiring user-downloaded hotspot data. Background Technology

[0002] We provide high-performance caching server equipment, which can solve the problem of limited outbound bandwidth of operators and greatly improve the response speed of users accessing websites. The basic idea is to cache high-traffic data such as images, videos, and files with high user access frequency per unit time on the server. When users access the resource, they are directed to retrieve it directly through the cache server.

[0003] The statistics on download hotspots (high-traffic data) within a unit of time are crucial prerequisites for caching servers. Current technologies mostly utilize a splitter to temporarily mirror and copy user data streams over a period of time and perform traffic statistics. For download hotspots that meet expectations, corresponding URLs are extracted, and servers within the local area network (LAN) use these URLs to cache the relevant data. When a user accesses the URL again, they directly access the information within the LAN. However, because each mirrored user data stream is relatively large, transmission and data conversion are time-consuming. Furthermore, to save cache space, after extracting the URL and caching the corresponding data, all user data streams are discarded. This results in low timeliness and low hit rate for download hotspot statistics. When mirroring new user data streams again, the same data may be cached repeatedly, wasting server cache space. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention provides a method for users to quickly obtain hotspot data for download, which solves the problems of long transmission time, low timeliness and hit rate, and duplicate caching in existing technologies.

[0005] According to an embodiment of the present invention, a method for quickly obtaining user download hotspot data includes:

[0006] S1: When a user is online normally, the packet capture server mirrors and copies the user's data stream over a period of time and uses netmap to bind the user's physical network card.

[0007] S2: The packet capture server intercepts all message information in the user data stream and uses hyperscan to quickly scan all message information, reconstructing the message information involving file, audio and video downloads to form a reconstructed URL;

[0008] S3: The packet capture server calls the record table and cache table in the web management machine and compares them with the reconstructed URL. If the record table does not record it, the reconstructed URL is counted for traffic and written to the cache table. Then, the reconstructed URL that meets the requirements is written to the record table and the ATS server is notified to retrieve the origin information. If the record table has already recorded it, no traffic statistics are performed. The packet capture server is directly notified to send a redirection status code to the user, so that the user can access the ATS server and obtain the data according to the redirection status code when downloading data.

[0009] Preferably, the packet capture server includes a splitter and a traffic analyzer. The splitter is used to mirror and copy user data streams, and the traffic analyzer is used to bind and analyze user data streams to physical network cards.

[0010] Preferably, if the reconstructed URL conforming to the specification is a dynamic URL, it needs to be converted into a static URL using the pcre library.

[0011] Preferably, the data fields of the cache table include the key value ats_key, url, total number of url requests get_total_count, and last request time of the url last_get_time, and the data fields of the record table include the key value ats_key, url, and the IP address corresponding to the url ats_ip.

[0012] Preferably, in step S3, when the reconstructed URL is written to the cache table, a hash algorithm is used to convert the reconstructed URL into a unique key value (ats_key) and write it to the cache table.

[0013] The writing criteria for the cache table are as follows: the reconstructed URL must conform to the specifications, the file size corresponding to the reconstructed URL must meet the set range, and the reconstructed URL must be a static URL.

[0014] The criteria for writing to the record table are: the URL must be a URL in the cache table, the get_total_count corresponding to the URL in the cache table must reach or exceed the set number of requests, and the corresponding last_get_time must not exceed three days.

[0015] Preferably, when writing the reconstructed URL from the cache table into the record table, the physical network card is converted into an IP address and written into the ATS_IP data field. At the same time, the packet capture server sends a back-to-origin notification and the corresponding reconstructed URL to the ATS server. The ATS server generates a redirection status code corresponding to the back-to-origin information based on the reconstructed URL back-to-origin information, and then sends back-to-origin feedback information and redirection status code to the packet capture server.

[0016] Compared with the prior art, the present invention has the following beneficial effects:

[0017] Netmap's high-efficiency I / O framework can transmit user data streams directly without excessive copying to improve reception speed. It also binds to the user's physical network card, quickly locating the user corresponding to the data stream. Furthermore, leveraging Hyperscan's features, it can quickly convert regular expressions in streaming message information into easily identifiable and storable pattern databases during packet scanning. It also reassembles message information involving file, audio, and video downloads into easily identifiable and storable reconstructed URLs. This reduces the steps in data transmission and conversion processes, shortens transmission and data conversion time, and improves efficiency.

[0018] Additionally, if the reconstructed URL is not in the record table, it means that the URL is not yet a download hotspot. After traffic statistics are performed on it, it will be written to the cache table. If the cache table already records the URL, it will be updated; otherwise, it will be added. In this way, after mirroring, the user data stream will be completely preserved. The traffic analysis results of multiple time periods can be combined to determine whether the URL is a download hotspot, extending the timeliness and increasing the hit rate. Moreover, if the reconstructed URL is in the record table, it means that the corresponding file has already been cached on the server, so it will not be cached again. In this case, traffic statistics can be omitted, reducing operation steps and allowing users to directly obtain the file from the server, saving server cache resources. Attached Figure Description

[0019] Figure 1 This is an overall device architecture diagram of an embodiment of the present invention.

[0020] Figure 2 This is a flowchart of user data stream processing according to an embodiment of the present invention. Detailed Implementation

[0021] The technical solutions of the present invention will be further described below with reference to the accompanying drawings and embodiments.

[0022] like Figure 1 As shown in the diagram, this is the overall device architecture diagram used for capturing and analyzing user data. The packet capture server includes a splitter and a traffic analyzer. The splitter is used to mirror and copy the user data stream, and the traffic analyzer is used to bind the user data stream to physical network cards and perform analysis. The web management machine stores cache tables and record tables.

[0023] like Figure 1 and Figure 2 As shown, this embodiment of the present invention proposes a method for quickly obtaining user download hotspot data, including:

[0024] S1: When a user is online normally, the packet capture server mirrors and copies the user's data stream over a period of time and uses netmap to bind the user's physical network card.

[0025] Multiple users are accessing the internet normally, and their data streams are as follows: Figure 1 Line ① in the diagram represents users directly accessing and downloading data via the core router (public network) provided by the operator. The packet capture server uses a splitter to mirror and copy user data streams for the same duration (also manually set, typically 30 minutes to 1 hour to account for bandwidth limitations and server overload) during the same time period each day (usually during peak usage hours, from 7 PM to 11 PM). The data stream is as follows: Figure 1 Line number ② in the text.

[0026] The splitter then sends the user data stream to the traffic analyzer. The traffic analyzer uses the Netmap high-efficiency I / O framework to receive the user data stream. Normally, when receiving data, the terminal first stores the received data in a reserved temporary memory space, and then copies the data in the temporary memory space to the permanent memory space. After the data is received, the temporary memory space is released to avoid data loss. However, since the network space distance between the splitter and the traffic analyzer is short, there is almost no data loss. Therefore, using the Netmap framework can reduce the number of data copying or shared bus operations when receiving data, eliminating unnecessary copying of communication data, effectively improving communication efficiency, saving storage space and processing time. At the same time, it binds the user data stream to the corresponding user's physical network card, quickly locating the user corresponding to the data stream.

[0027] S2: The packet capture server intercepts all message information in the user data stream and uses hyperscan to quickly scan all message information, reconstructing the message information involving file, audio and video downloads to form a reconstructed URL;

[0028] The traffic analysis device in the packet capture server intercepts all TCP and HTTP packets in the user data stream. Hyperscan is used to quickly scan the packets. Taking advantage of Hyperscan's characteristics, since the user data stream is continuous, the packets are scanned in streaming mode. The packets involving file, audio, and video downloads are quickly reassembled. Difficult-to-recognize regular expressions in the URLs are converted into easily identifiable and stored pattern database patterns, forming compliant reassembled URLs. This reduces the steps in data transmission and conversion processes, shortens transmission and data conversion time, and improves efficiency.

[0029] Due to limited server cache space and bandwidth, files larger than 1GB are typically not cached. An HTTP request is sent by reconstructing the URL to obtain the size of the download file corresponding to the reconstructed URL. If the download file size exceeds 1GB, it is discarded without caching. Next, it is determined whether the reconstructed URL is dynamic. This example uses the URL "http: / / www.baidu.com / displayCategory.aspx?CategoryID=5" as an example. When this URL is accessed, all data for the entered parameter "CategoryID=5" will be displayed in a block named displayCategory.aspx on the website. If the value of "CategoryID" changes, the data in this block will also change. Therefore, using the PCRE library, the data for "CategoryID=5" is extracted and replaced in the displayCategory.aspx block, forming a static page. Thus, the dynamic URL becomes a static URL: "http: / / www.baidu.com / CategoryID5.html". On this static URL website, the data for "CategoryID=5" will be displayed consistently and will not change.

[0030] S3: The packet capture server calls the record table and cache table in the web management machine and compares them with the reconstructed URL. If the record table does not record it, the reconstructed URL is counted for traffic and written to the cache table. Then, the reconstructed URL that meets the requirements is written to the record table and the ATS server is notified to retrieve the origin information. If the record table has already recorded it, no traffic statistics are performed. The packet capture server is directly notified to send a redirection status code to the user, so that the user can access the ATS server and obtain the data according to the redirection status code when downloading data.

[0031] The data fields of the cache table include the key value ats_key, url, total number of requests to the url get_total_count, and the last request time of the url last_get_time.

[0032] The data fields in the record table include: key value ats_key, url, and the IP address corresponding to the url ats_ip.

[0033] After converting all recombined URLs into static URLs, the squared middle method in the hash algorithm is used to convert the recombined URLs into unique, easy-to-record, identify and statistically analyze numeric key values. Different recombined URLs will have different key values.

[0034] Next, the cache and record tables in the web management machine are accessed. The key-value pair of the reconstructed URL is precisely compared with the `ats_key` in the record table. If the record table does not contain a corresponding reconstructed URL, traffic statistics are performed on the reconstructed URL. The total number of requests to the same reconstructed URL within the specified time period and the time of the last request are calculated. Then, the key-value pair, reconstructed URL, total number of requests, and time of the last request are written into the corresponding data fields in the cache table. Simultaneously, if data with the same key-value pair exists in the cache table, the total number of requests is added together, and the time of the last request is updated to the latest time. In this way, after mirroring, the user data stream is completely preserved. Traffic analysis results from multiple time periods can be combined to determine if the URL is a download hotspot, extending timeliness and increasing the hit rate.

[0035] When the total number of requests to a reconstructed URL (the value of the `get_total_count` field) reaches or exceeds the set request limit (manually set according to requirements, default is 500 times), it is determined to be a user download hotspot. At this time, it is checked whether the last request time for the reconstructed URL exceeds 3 days. If it exceeds 3 days, the physical network card corresponding to the reconstructed URL is converted to an IP address, and the key value of the reconstructed URL, `ats_key`, `url`, and IP address are written to the corresponding data fields in the record table. At the same time, the packet capture server also sends a back-to-origin notification and the reconstructed URL to the ATS server. Figure 1 Line ③ in the URL (instructions) allows the ATS server to cache the corresponding download file based on the reconstructed URL and generate the corresponding 302 redirect status code. Once caching is complete, it sends a feedback message indicating completion of the origin retrieval process, along with the 302 redirect status code, to the packet capture server. Figure 1 (Line ③ in the image) The packet capture server associates the 302 redirect status code with the corresponding reconstructed URL in the record table.

[0036] If, after calculating the key-value pair, the reconstructed URL is compared with the record table and found to already contain the same reconstructed URL, it indicates that the ATS server has already cached the corresponding download file. In this case, traffic statistics can be omitted, and the packet capture server will send the 302 redirect status code associated with the reconstructed URL to the user (e.g., ...). Figure 1 (Line 5 in the diagram) When a user initiates an HTTP GET request, the request will be redirected to the ATS server to retrieve the corresponding file (the user data stream passes through lines 1 and 4) and will no longer be retrieved from the public network.

[0037] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A method for fast acquisition of user downloaded hotspot data, characterized by: The method comprises the following steps: S1: when the user is surfing the Internet normally, the packet capturing server mirrors the user data stream of the user in a period of time and binds the physical network card of the user using netmap; S2: the packet capturing server intercepts all the message information in the user data stream and uses hyperscan to quickly scan all the message information, and reorganizes the message information related to file, audio and video download to form a reorganized url; S3: the packet capturing server calls the record table and cache table in the web management machine and compares them with the reorganized url, if the record table is not recorded, the reorganized url is subjected to traffic statistics and written into the cache table, then the reorganized url meeting the requirements is written into the record table and the ATS server is notified of the source information, if the record table is recorded, no traffic statistics is performed, and the packet capturing server is directly notified to send a redirection status code to the user, so that the user accesses the ATS server according to the redirection status code when downloading data and obtains the data.

2. The method for quickly obtaining user download hotspot data as described in claim 1, characterized in that: The packet capturing server comprises a splitter and a traffic analyzer, the splitter is used for mirroring the user data stream, and the traffic analyzer is used for binding the physical network card and analyzing the user data stream.

3. The method for quickly obtaining user download hotspot data as described in claim 1, characterized in that: If the reorganized url meeting the specification is a dynamic url, it needs to be converted into a static url using a pcre link library.

4. The method for quickly obtaining user download hotspot data as described in claim 1, characterized in that: The data fields of the cache table include a key value ats_key, a url, a total request number of the url get_total_count and a last request time of the url last_get_time, and the data fields of the record table include a key value ats_key, a url and an IP address ats_ip corresponding to the url.

5. The method of claim 1, wherein the method further comprises: receiving a request for the data from the user; and transmitting the data to the user. 5 In S3, when the reorganized url is written into the cache table, a hash algorithm is used to convert the reorganized url into a unique key value ats_key and write it into the cache table; The writing standard of the cache table is that the reorganized url must meet the specification, the file size corresponding to the reorganized url must meet the set range, and the reorganized url must be a static url; The writing standard of the record table is that it must be a url in the cache table, the get_total_count corresponding to the url in the cache table reaches or exceeds the set request number, and the time of the last_get_time does not exceed three days.

6. The method of claim 1, wherein: When the reorganized url in the cache table is written into the record table, the physical network card is converted into an IP address and written into the ats_ip data field, and the packet capturing server sends a source notification and a corresponding reorganized url to the ATS server, the ATS server returns the source information according to the reorganized url and generates a redirection status code corresponding to the source information, and then sends the source feedback information and the redirection status code to the packet capturing server.

Images