A method for intelligent collection of website asynchronous sequence data based on dynamic self-adaptation

Through dynamic adaptive website asynchronous sequence data intelligent collection method, user behavior learning and hidden channel transmission technology are used to solve the problems of low efficiency and high cost of existing anti-crawler technology, and efficient and secure collection of target website data is achieved. transmission.

CN114297462BActive Publication Date: 2024-11-12THE 28TH RES INST OF CHINA ELECTRONICS TECH GROUP CORP +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202111515500.5
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-12-13
Publication Date
2024-11-12
Estimated Expiration
2041-12-13

AI Technical Summary

Technical Problem

Existing anti-crawler technologies are inefficient and costly, making it difficult to effectively collect and transmit asynchronous sequence data from target websites, especially when facing complex anti-crawler strategies.

Method used

Adopt dynamic adaptive intelligent collection method for website asynchronous sequence data, through user Internet access behavior learning, non-hacker invasive restricted content access, automatic account registration and keep-alive, automatic compilation of Ajax-oriented browser, and adaptive matching of web elements With extraction and hidden channel data backhaul technology, the secure backhaul of distributed network crawlers is realized.

Benefits of technology

Real-time data collection and hidden transmission of the target website are achieved, and the anti-crawling strategy is broken through, ensuring the continuity and security of data, and reducing the collection cost.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114297462B_ABST
    Figure CN114297462B_ABST
Patent Text Reader

Abstract

The present invention provides a method for intelligent collection of asynchronous sequence data of websites based on dynamic self-adaptation. Based on the existing crawler cluster, account pool and IP pool, the present invention targets common anti-crawling technologies, such as ajax dynamic loading technology, dynamic cookie access restriction, system security protection, abnormal access behavior monitoring and abnormal account monitoring, and realizes real-time data collection of target mainstream websites through a dynamic self-adaptive intelligent collection system, realizes continuous acquisition and covert transmission of target website data through distributed network crawler security feedback, and paves the way for further data analysis.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention relates to the field of data collection, and in particular to a method for intelligently collecting website asynchronous sequence data based on dynamic self-adaptation. Background Art

[0002] With the open and explosive growth of the Internet, the value of data has become increasingly important, especially for e-commerce, media, social networking and other businesses. It is not an exaggeration to compare data to gold. As a result, web crawler technology was born. Hackers use the free interfaces opened by websites to obtain valuable data in batches for data mining and analysis of industry conditions. However, a large number of illegal crawlers will cause huge pressure on website servers and even affect the access of normal users; and the theft of valuable data will also have a negative impact on the commercial interests of the website.

[0003] Therefore, anti-crawler technology came into being. Anti-crawler technology generally includes two steps: "crawler identification" and "crawler countermeasure". The latter is mainly used to punish and counter the crawlers identified by the former, mainly including access restrictions, verification code verification, data poisoning, etc. The former is currently commonly used based on rule judgment. For example, taking a certain user or IP as a unit, counting its access records within a certain period of time, and then using some artificially set thresholds, this can be called an expert rule method. Its advantages are clear and reliable rules, and can set rules in real time according to the discovered crawler features, so as to achieve confrontation with crawlers. When the rules become more and more complex and the number is large, the efficiency of the rule engine will become lower and lower, and the cost will be relatively high. Summary of the invention

[0004] Purpose of the invention: The present invention is based on the existing crawler cluster, account pool and IP pool, and targets common anti-crawling technologies, such as ajax dynamic loading technology, dynamic cookie access restriction, system security protection, abnormal access behavior monitoring and abnormal account monitoring. It realizes real-time data collection of target mainstream websites through a dynamic and adaptive intelligent collection system, and realizes continuous acquisition and covert transmission of target website data through distributed network crawler security feedback, laying a good foundation for further data analysis.

[0005] The present invention specifically provides a method for intelligent collection of website asynchronous sequence data based on dynamic self-adaptation, realizes dynamic decoupling of collection services, dynamic collection resources, anti-crawling technology and scheduling framework, supports dynamic self-adaptation of scheduling framework according to collection tasks, and realizes intelligent collection based on asynchronous sequence, including the following steps:

[0006] Step 1: Implement crawler scheduling based on user Internet access behavior learning;

[0007] Step 2: Implement non-hacker-intrusive access to restricted content;

[0008] Step 3: Automatically register and keep accounts active based on user behavior learning;

[0009] Step 4: The Ajax-oriented browser completes automatic compilation;

[0010] Step 5: adaptive matching and extraction of web page elements;

[0011] Step 6: Implement data feedback based on covert channels.

[0012] Step 1 includes: using the existing technology to obtain the operational feature information of user network access, and establishing a crawler scheduling model that simulates user operations through the page replacement algorithm (Least Recently Used, LRU), using more than two sets of browser disguised fingerprints (user-agent) to reduce the access frequency of a single crawler to the behavior frequency of real users, and at the same time, through the working mode of the multi-crawler system, that is, through the crawler task scheduling system, the tasks are distributed to each collection node according to the collection frequency, and different agents and account disguise strategies are used to obtain the content under different sections of the same website, and finally spliced ​​according to the data storage dictionary standard, the original unstructured intermediate results of the collection response content are converted into final structured data.

[0013] Step 2 includes: performing adaptive entity extraction on the request response content by setting website address parameters and cookies in HTTP or HTTPS requests, and using a keep-alive account to achieve non-hacker-intrusive access to restricted content and collection of web page elements.

[0014] Step 3 includes: mimicking the browser fingerprint, IP address, and login session of real users to register, log in, post, and repost from time to time, thereby avoiding the target website's monitoring of abnormal accounts and ensuring that the number of active accounts in the account pool is not less than the threshold (usually 50).

[0015] Step 4 includes: implementing script parsing and dynamic execution functions through a JavaScript parser, reconstructing the target page document object model (DOM), and fully rendering the target source file.

[0016] Step 5 includes: after the crawler obtains the page resources, extract the target page DOM tree, locate the path expression of each page element with the help of existing similarity matching and xpath syntax, and at the same time, through traversal algorithms such as pre-order traversal, in-order traversal, and post-order traversal, realize automatic recognition and extraction of important elements such as page title, time, and text.

[0017] The method includes: in order to obtain the access relationship between web pages, the web pages in the website are cyclically parsed, and the links between the web pages are stored in a two-dimensional dictionary; a relationship matrix is ​​generated, in which two dictionaries are defined through the enumeration function in Python to effectively distinguish each link, and the two dictionaries are index nodes and node indexes respectively;

[0018] After generating the dictionary, define a zero matrix M with a length of n*n, where n represents the number of crawled web pages;

[0019] The two layers of loops are nested to form a relationship matrix. The outer loop uses the key of the two-dimensional dictionary as the index, and the inner loop obtains the value corresponding to the key to generate the initial probability matrix. The more hyperlinks a web page points to, the higher the probability of randomly jumping to the web page, and the higher the weight of the web page. Assume that the transfer probability from a node to all nodes connected by directed edges is equal, and the probability matrix M is calculated as follows:

[0020] M=[m ij ]n×n

[0021] The element m in the i-th row and j-th column ij The value selection rule is as follows: If node j has k directed edges out of degree, and node i is one of its out-degree nodes, then m ij =1 / K; otherwise m ij =0,i,j=1,2,……,n;

[0022] Finally, the equidistant probability matrix and the first element full capture probability matrix are generated through iterative probability matrix. The equidistant probability matrix is ​​generated iteratively to assign the value of 1 / n to each element in the zero matrix M. The first element full capture probability matrix assigns the probability of 1 to the first element of the zero matrix M. The specific weight calculation of each page is as follows:

[0023]

[0024] Among them, Mp i Is all the web page p i The set of web pages with out-degree, L(p j ) is the web page p j The number of out-degrees, N is the total number of web pages, α is the threshold, PR(p i ) is the web page p i The weight of PR(p j ) is the web page p j The weight of

[0025] The new probability matrix is ​​updated with the newly calculated weights. The process of generating the probability matrix and calculating the page weights is iterative until the suspension criterion is met.

[0026] Step 6 includes:

[0027] Step 6-1, network encrypted transmission;

[0028] Step 6-2: Encrypt sensitive and non-sensitive warehouse data (data sets built by domain based on data warehouse)

[0029] Step 6-1 includes:

[0030] Step 6-1-1, using the existing Secure Sockets Layer (SSL) encryption technology to encrypt data transmitted between systems;

[0031] Step 6-1-2, use the existing SSL VPN (a new type of virtual private network technology that uses the Secure Sockets Layer (SSL) protocol to achieve remote access) technology to achieve remote access to the server, ensure that the data in the remote access transmission is securely protected, and use encryption to ensure that the data is not illegally intercepted.

[0032] Step 6-2 includes:

[0033] Step 6-2-1, collect interface information, including sensitive information such as personal name, email address, contact information, address, and non-sensitive information such as news title, release time, author, content, etc.

[0034] Step 6-2-2: sensitive information is collected and stored in the sensitive information user database. After successful storage, the local file is deleted.

[0035] Step 6-2-2 includes:

[0036] Step 6-2-2-1, modify the original data transmission channel (FTP) so that sensitive information is not stored in the database;

[0037] Step 6-2-2-2, collect non-sensitive information, and ensure the daily production of non-sensitive information through distributed collection system and collection program status monitoring;

[0038] Step 6-2-2-3, integrate sensitive data such as personal name, email address, contact information, address, etc., encrypt, compress, and subpackage to generate files, push the files to the sensitive information database (a knowledge base with sensitive information characteristics accumulated through long-term practice), and delete the local files after the push is successful;

[0039] Step 6-2-2-4: sensitive information is collected and stored in the database, and labels are processed according to the real-name system. At the same time, sensitive fields are encrypted into irreversible MD5 ciphertext. After labeling and encryption are completed, sensitive information data in the database is cleared;

[0040] Step 6-2-2-5, import the desensitized label and MD5 ciphertext into the non-sensitive information user;

[0041] Step 6-2-2-6, transformation of the application service process for plaintext sensitive data, specifically includes the following steps:

[0042] Step 6-2-2-6-1: If the customer's requirements involve plaintext sensitive information data, the person in charge of the requirements interface needs to initiate a plaintext sensitive information application to the sensitive information user administrator of the big data platform;

[0043] Step 6-2-2-6-2: After the review is passed, the sensitive information administrator of the big data platform submits the request to the sensitive information database;

[0044] Step 6-2-2-6-3, the request is passed. If the data volume is less than the threshold (usually 1M), the required information is obtained through the remote call technology (Webservice) across programming languages ​​and operating system platforms; if the data volume is greater than the threshold (usually 1M), the decryptable ciphertext data file is obtained through the File Transportation Protocol (FTP) which can output a large amount of information, and the key is obtained at the same time;

[0045] Step 6-2-2-6-4, generate an encrypted file with sensitive information and other required information, such as the number of fans, friends, followers, likes, shares, comments, etc., and push it to the demander, and provide the key. If the push is successful, the sensitive information user of the big data platform deletes the corresponding data and files.

[0046] The intelligent collection and preprocessing technology of dynamically adaptive mainstream websites mainly includes adaptive matching collection for mainstream network architecture and distributed web crawler security feedback.

[0047] The adaptive matching collection content for mainstream website architecture mainly includes the following three parts: research on crawler scheduling models based on user Internet access behavior learning, research on non-hacker intrusive restricted content access technology, account automatic registration and keep-alive technology based on user behavior learning, browser automatic compilation technology for ajax and research on adaptive matching and extraction technology of web page elements, to achieve intelligent scheduling of crawler resources, thereby achieving automatic avoidance of anti-crawling technology of target websites and ensuring correct and rapid crawling of target resources.

[0048] Distributed web crawler secure feedback is aimed at solving the problems of data difficulty in acquisition and information receiver exposure caused by the target website's anti-crawling strategies and traceability measures. It breaks through the data acquisition technology of highly anti-crawling websites and the data feedback technology based on covert channels to achieve continuous acquisition and covert transmission of target website data.

[0049] Beneficial effects: The present invention designs an adaptive distributed web crawler system, breaks through the dynamic adjustment of the website's anti-crawler strategy, applies data encryption, steganography and carrying technology, prevents data parsing and tracing, and ensures the continuity and security of collected data. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] The present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments, and the above and / or other advantages of the present invention will become more clear.

[0051] Figure 1 It is a diagram of the architecture of the method of the present invention.

[0052] Figure 2 It is a collection flow chart for mainstream website architecture.

[0053] Figure 3 Data encryption flow chart for data transmission back to the warehouse.

[0054] Figure 4 Data return requirement processing flow chart.

[0055] Figure 5 It is a schematic diagram of the probability matrix. DETAILED DESCRIPTION

[0056] Through account registration and preservation technology based on user behavior learning, the automatic update and dynamic preservation of the collection resource account pool can be realized. Through the research of crawler scheduling model based on user Internet access behavior learning, the intelligent distributed scheduling of crawler clusters and IP resources can be realized, the concealment of crawlers can be improved, and the crawlers can be highly anti-crawling for abnormal access monitoring of websites. Through the research of non-hacker intrusive restricted content access technology and automatic allocation of account pools, it can be ensured that the crawler can successfully collect resources from the target website. Through the research of intelligent automatic compilation technology and adaptive matching and extraction technology of web page elements, high-precision identification of target website resources can be achieved.

[0057] Step 1, crawler scheduling based on user Internet access behavior learning: By using the current architecture to learn the operational characteristics of user network access, a crawler scheduling model that simulates user operations is established through the page replacement algorithm (Least Recently Used, LRU), and two or more sets of browser camouflage fingerprints (user-agent) are used to reduce the access frequency of a single crawler to the behavior frequency of real users. At the same time, through the multi-crawler system working mode, that is, through the crawler task scheduling system, tasks are distributed to each collection node according to the collection frequency, and different agents and account camouflage strategies are used to obtain content under different sections of the same website. Finally, the original unstructured intermediate results of the collection response content are converted into final structured data according to the data storage dictionary standard for splicing (see the flowchart). Figure 1). For example: in the Zhihu collection process, it is necessary to disguise different browser fingerprints and different proxy IPs for collection. On the contrary, if a fixed IP accesses the website quickly and in large quantities within a period of time, it will trigger the website anti-crawling strategy and cause the collection program to terminate.

[0058] Step 2: Non-hacker-intrusive access to restricted content: In order to effectively collect data in the long term, it is necessary to study effective anti-crawler functions to deal with social media websites such as Weibo that have strict anti-crawler blocking policies. By setting website address parameters and cookies in HTTP / HTTPS requests, and performing adaptive entity extraction on request response content, non-hacker-intrusive access to restricted content can be achieved.

[0059] Step 3, automatic account registration and keeping active based on user behavior learning, by imitating the browser fingerprint, IP address, login session of real users to register, log in, post, and repost activities from time to time, avoid the target website's monitoring of abnormal accounts, and ensure that there are enough active accounts in the account pool. For example: in Facebook collection, enough active accounts are needed to support the collection program to simulate user behavior for collection. At the same time, in account registration, it is also necessary to simulate the real user's IP address and registration time to register the collection account. In account maintenance, the account is used to post, comment and other real user operations to ensure the activity of the account and avoid being monitored by the platform's anti-crawling strategy.

[0060] Step 4, automatic compilation for Ajax-oriented browsers: For the mainstream website architecture implemented by Ajax technology, the first HTTP request through the URL uniform resource locator (URL) cannot load all the data. For most Ajax pages, the first HTTP request can only load the framework of the web page. Therefore, in order to fully obtain the information of the dynamic page, the JavaScript code is dynamically executed in the crawler program, and a fast JavaScript parser developed independently is used to implement fast script parsing and dynamic execution functions, realize the rapid reconstruction of the target page DOM tree, and realize the complete drawing of the target source file, which is convenient for further extraction of web page elements (see the flowchart). Figure 2 ).

[0061] Step 5, adaptive matching and extraction of web page elements: After the crawler obtains the page resources, it quickly extracts the target page DOM tree, locates the path expression of each page element with the help of existing similarity matching and xpath syntax, and uses traversal algorithms such as pre-order traversal, in-order traversal, and post-order traversal. Among them, the adaptive page sorting algorithm is a static algorithm that can calculate the weight of the web page offline. The advantage is that it can effectively reduce the amount of calculation during online queries and improve response efficiency. This system has designed an improved web page sorting algorithm, the iterative process of which is controlled by a predefined threshold or the consistency between different web page anchor texts. First, in order to obtain the access relationship between web pages, the web pages are parsed in a loop on the website, and the links between web pages are stored in a two-dimensional dictionary. Then, each link is effectively distinguished through this relationship matrix, and two dictionaries are defined, namely index nodes and node indexes. After the dictionary is generated, a zero matrix M is defined with a length of n*n (n: the number of crawled web pages). Two layers of loops are nested to form a relationship matrix. The outer loop uses the key of the two-dimensional dictionary as the index, and the inner loop obtains the value corresponding to the key to generate an initial probability matrix. As Figure 5 As shown, nodes A, B, C, and D represent web pages, nodes represent links between web pages, and the weights on the edges represent the probability of random jumps between web pages. The more hyperlinks a web page points to, the higher the probability of randomly jumping to the web page, and the higher the weight of the web page. Set the transition probability from a node to all nodes connected by directed edges to be equal. Finally, according to different situations, the system can iteratively generate an equidistant probability matrix and a first-element full capture probability matrix. The former assigns a value of 1 / n to each element in the zero matrix, and the latter assigns a probability of "1" to the first element of the zero matrix. The specific weight calculation for each page is as follows:

[0062]

[0063] Mpi is the set of all web pages linked to web page Pi, L(pj) is the number of web pages linked to web page pj, N is the total number of web pages, and α is generally taken as 0.85 in experience. Then, the new probability matrix is ​​updated with the new calculated weights. The process of generating the probability matrix and calculating the page weights is iterative until the suspension criteria are met. In the method of the present invention, the page sorting algorithm stops until the error between the previous and next two iterations is less than 0.001 or the consistency between high-weight web pages is less than a specified value. Automatic recognition and extraction of important elements such as page title, time, and text are achieved.

[0064] Step 6: Data transmission based on covert channels. Through the research of covert channel data transmission technology, ensure the high confidentiality of data transmission and prevent the target website from tracing the source (see the flowchart) Figure 3 , Figure 4 ).

[0065] Step 6 includes:

[0066] Step 6-1, network encrypted transmission;

[0067] Step 6-2: Encrypt sensitive and non-sensitive warehouse data (data sets built by domain based on data warehouse)

[0068] Step 6-1 includes:

[0069] Step 6-1-1, using the existing Secure Sockets Layer (SSL) encryption technology to encrypt data transmitted between systems;

[0070] Step 6-1-2, use the existing SSL VPN technology to achieve remote access to the server, ensure that the data in remote access transmission is securely protected, and use encryption to ensure that the data is not illegally intercepted.

[0071] Step 6-2 includes:

[0072] Step 6-2-1, collect interface information, including sensitive information such as personal name, email address, contact information, address, and non-sensitive information such as news title, release time, author, content, etc.

[0073] Step 6-2-2: sensitive information is collected and stored in the sensitive information user database. After successful storage, the local file is deleted.

[0074] Step 6-2-2 includes:

[0075] Step 6-2-2-1, modify the original data transmission channel (FTP) so that sensitive information is not stored in the database;

[0076] Step 6-2-2-2, collect non-sensitive information, and ensure the daily production of non-sensitive information through distributed collection system and collection program status monitoring;

[0077] Step 6-2-2-3, integrate sensitive data such as personal name, email address, contact information, address, etc., encrypt, compress, and subpackage to generate files, push the files to the sensitive information database (a knowledge base with sensitive information characteristics accumulated through long-term practice), and delete the local files after the push is successful;

[0078] Step 6-2-2-4: sensitive information is collected and stored in the database, and labels are processed according to the real-name system. At the same time, sensitive fields are encrypted into irreversible MD5 ciphertext. After labeling and encryption are completed, sensitive information data in the database is cleared;

[0079] Step 6-2-2-5, import the desensitized label and MD5 ciphertext into the non-sensitive information user;

[0080] Step 6-2-2-6, transformation of the application service process for plaintext sensitive data, specifically includes the following steps:

[0081] Step 6-2-2-6-1: If the customer's requirements involve plaintext sensitive information data, the person in charge of the requirements interface needs to initiate a plaintext sensitive information application to the sensitive information user administrator of the big data platform;

[0082] Step 6-2-2-6-2: After the review is passed, the sensitive information administrator of the big data platform submits the request to the sensitive information database;

[0083] Step 6-2-2-6-3, the request is passed. If the data size is less than 1M, the required information is obtained through the remote call technology (Webservice) across programming languages ​​and operating system platforms; if the data size is greater than 1M, the decryptable ciphertext data file is obtained through the File Transportation Protocol (FTP) which can output a large amount of information, and the key is obtained at the same time;

[0084] Step 6-2-2-6-4, sensitive information and other information required by the demand are used together to generate an encrypted file and pushed to the demander, and a key is provided. If the push is successful, the sensitive information user of the big data platform deletes the corresponding data and files. The present invention provides a method for intelligent collection of asynchronous sequence data of websites based on dynamic self-adaptation. There are many methods and ways to implement the technical solution. The above is only a preferred implementation of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention. All components not specified in this embodiment can be implemented using existing technologies.

Claims

1. A method for intelligent collection of website asynchronous sequence data based on dynamic self-adaptation, characterized in that: The steps include: Step 1: Based on the operational feature information of user network access, a crawler scheduling model simulating user operations is established through a page replacement algorithm. The access frequency of a single crawler is reduced to the behavior frequency of real users by using more than two sets of browser disguised fingerprints. At the same time, the crawler task scheduling system distributes tasks to each collection node according to the collection frequency. Different proxies and account disguise strategies are used to obtain content under different sections of the same website. Finally, the content is spliced ​​according to the data storage dictionary standard, and the original unstructured intermediate results of the collection response content are converted into final structured data, realizing crawler scheduling based on user Internet access behavior learning; Step 2, by setting website address parameters and cookies in HTTP or HTTPS requests, adaptive entity extraction is performed on the request response content, and a keep-alive account is used to achieve non-hacker-intrusive access to restricted content and collection of web page elements, thereby achieving non-hacker-intrusive access to restricted content; Step 3: By imitating the browser fingerprint, IP address, and login session of real users to register, log in, post, and repost from time to time, the target website can avoid monitoring abnormal accounts, ensure that the number of active accounts in the account pool is not lower than the threshold, and automatically register and keep accounts active based on user behavior learning; Step 4, implement script parsing and dynamic execution functions through JavaScript parser, achieve target page document object model reconstruction, achieve complete rendering of target source file, and complete automatic compilation for Ajax-oriented browser; Step 5, adaptive matching and extraction of web page elements: After the crawler obtains the page resources, it extracts the target page DOM tree, locates the path expression of each page element with the help of similarity matching and xpath syntax, and realizes automatic recognition and extraction of page title, time, and text through pre-order traversal, in-order traversal, and post-order traversal algorithms. In order to obtain the access relationship between web pages, it cyclically parses the web pages in the website and stores the links between web pages in a two-dimensional dictionary; Generate a relationship matrix. In this relationship matrix, in order to effectively distinguish each link, two dictionaries are defined through the enumeration function in Python. The two dictionaries are index nodes and node indexes respectively. After generating the dictionary, define a zero matrix M with a length of n*n, where n represents the number of crawled web pages; The two layers of loops are nested to form a relationship matrix. The outer loop uses the key of the two-dimensional dictionary as the index. The inner loop obtains the value corresponding to the key and generates an initial probability matrix. The transition probability from one node to all nodes connected by directed edges is set to be equal. The probability matrix M is calculated as follows: M=[m ij ]n×n The element m in the i-th row and j-th column ij The value selection rule is as follows: If node j has k directed edges out of degree, and node i is one of its out-degree nodes, then m ij =1 / K; otherwise m ij =0,i,j=1,2,……,n; Finally, the equidistant probability matrix and the first element full capture probability matrix are generated through iterative probability matrix. The equidistant probability matrix is ​​generated iteratively to assign the value of 1 / n to each element in the zero matrix M. The first element full capture probability matrix assigns the probability of 1 to the first element of the zero matrix M. The specific weight calculation of each page is as follows: Among them, Mp i Is all the web page p i The set of web pages with out-degree, L(p j ) is the web page p j The number of out-degrees, N is the total number of web pages, α is the threshold, PR(p i ) is the web page p i The weight of PR(p j ) is the web page p j The weight of The new probability matrix is ​​updated with the newly calculated weights. The process of generating the probability matrix and calculating the page weight is iterative until the suspension criterion is met. Step 6: Realize data transmission based on covert channel; Step 6 includes: Step 6-1, network encrypted transmission; Step 6-2: Encryption transformation of sensitive and non-sensitive warehouse data; Step 6-1 includes: Step 6-1-1, using secure socket protocol encryption technology to encrypt data transmitted between systems; Step 6-1-2, use SSL VPN technology to achieve remote access to the server, ensure that the data in remote access transmission is securely protected, and use encryption to ensure that the data is not illegally intercepted; Step 6-2 includes: Step 6-2-1, interface collection, including sensitive information and non-sensitive information collection; Step 6-2-2: sensitive information is collected and stored in the sensitive information user database. After successful storage, the local file is deleted.

2. The method according to claim 1, characterized in that Step 6-2-2 includes: Step 6-2-2-1, modify the original data transmission channel so that sensitive information is not stored; Step 6-2-2-2, collect non-sensitive information, and ensure the daily production of non-sensitive information through distributed collection system and collection program status monitoring; Step 6-2-2-3, integrate sensitive data, encrypt, compress, and package to generate files, push the files to the sensitive information database, and delete the local files after the push is successful; Step 6-2-2-4: sensitive information is collected and stored in the database, and labels are processed according to the real-name system. At the same time, sensitive fields are encrypted into irreversible MD5 ciphertext. After labeling and encryption are completed, sensitive information data in the database is cleared; Step 6-2-2-5, import the desensitized label and MD5 ciphertext into the non-sensitive information user; Step 6-2-2-6, transformation of the application service process for plaintext sensitive data, specifically includes the following steps: Step 6-2-2-6-1: If the customer's requirements involve plaintext sensitive information data, the person in charge of the requirements interface needs to initiate a plaintext sensitive information application to the sensitive information user administrator of the big data platform; Step 6-2-2-6-2: After the review is passed, the sensitive information administrator of the big data platform submits the request to the sensitive information database; Step 6-2-2-6-3, the request is passed. If the data volume is less than the threshold, the required information is obtained through the remote call technology across programming languages ​​and operating system platforms; if the data volume is greater than the threshold, the decryptable ciphertext data file is obtained through the file transfer protocol, and the key is obtained at the same time; Step 6-2-2-6-4, generate an encrypted file with sensitive information and other required information and push it to the demander, and provide the key. If the push is successful, the sensitive information user of the big data platform deletes the corresponding data and files.