Safe and efficient Cookie pool management system supporting multiple platforms

Abstract

The invention relates to the technical field of network security, and discloses a safe and efficient Cookie pool management system supporting multiple platforms, which comprises a multi-source Cookie acquisition module, a Cookie storage and classification module and a Cookie scheduling and distribution module. By adopting the method and the device, an anti-crawler mechanism of a target website can be effectively avoided, the stability and the continuity of a crawler program in a data acquisition process are improved, more comprehensive website information is obtained by managing and applying various Cookies with different permissions, and the flexibility and the expandability of the Cookie pool are enhanced, so that the Cookie pool can adapt to changes of different types of websites; and the requirements of large-scale crawler application scenes are met.

Description

Technical Field

[0001] The present application relates to the field of network security technology, and specifically is a cookie pool management system that supports multiple platforms safely and efficiently. Background Art

[0002] In traditional crawler programs, for websites that require login or are user-based, a fixed user account and password are typically embedded in the crawler program, simulating login to obtain a cookie, and then using that cookie in subsequent requests. This approach is simple and straightforward, but each crawler instance often only uses a single cookie. Some crawler systems have simple cookie storage mechanisms that store acquired cookies in local files or memory for continuous use during the crawler's operation. When a cookie expires or becomes invalid, the crawler may log in again to obtain a new cookie, but this lacks comprehensive management and flexible use of multiple cookies.

[0003] Because crawlers use a single cookie to make numerous requests, they are easily detected by the target website's anti-crawler mechanisms. Websites can analyze cookie access frequency, request patterns, and other characteristics to identify crawlers as legitimate users, leading to restrictive measures such as IP address restrictions, verification code requirements, and account bans, severely impacting the crawler's normal operation and data collection efficiency.

[0004] Secondly, a single cookie often corresponds to a specific user's permissions and browsing scope, and can only retrieve information within that user's permissions. Traditional methods cannot effectively capture content that requires different user roles or permissions, resulting in incomplete data collection.

[0005] Furthermore, traditional cookie management methods lack sufficient flexibility when dealing with a large number of different websites and complex user permission systems. They are unable to quickly adapt to website changes (such as updated login verification methods and changes in cookie structures), and are difficult to scale to large-scale crawler application scenarios, limiting the development and application of crawler systems.

[0006] Therefore, a more suitable cookie pool management technology is urgently needed. Summary of the Invention

[0007] The purpose of this application is to provide a secure and efficient cookie pool management system that supports multiple platforms to solve the technical problems raised in the above background technology.

[0008] To achieve the above objectives, this application discloses the following technical solutions: a secure and efficient cookie pool management system that supports multiple platforms, including:

[0009] The multi-source cookie collection module is configured to automatically collect cookies from multiple platforms and users. The automated collection involves recording login information for different domain name websites and ordinary users, member users, and administrator users through configuration files or databases. The login information includes account numbers, passwords, and login URLs.

[0010] The cookie storage and classification module is configured to: classify cookies according to website domain name, user type and permission level, adopt security management measures, encrypt the classified data and store it in a distributed storage unit, and perform secure management of cookies;

[0011] The cookie scheduling and allocation module is configured to: select cookies from the cookie storage and classification module and allocate them to crawler tasks based on dynamic scheduling strategy and load balancing processing, the dynamic scheduling strategy is used to select target cookies, and the load balancing processing is used to adjust the allocation of crawler tasks.

[0012] Preferably, the multi-source cookie collection module includes a simulated form login interface and an API login interface; the simulated form login interface is used to collect cookies from ordinary websites that do not provide public API or OAuth authentication, and the API login interface is used to collect cookies from websites that support API or OAuth authentication.

[0013] Preferably, the distributed storage unit includes a distributed file system or a distributed database.

[0014] Preferably, the scheduling strategy includes random selection, weighted selection, and user behavior simulation-based selection; wherein, the weighted selection specifically includes: selecting cookies after assigning weights based on the cookie's historical success rate and freshness; and the user behavior simulation-based selection specifically includes: selecting cookies based on the collected data type and the target user's behavior pattern;

[0015] The load balancing process is to automatically adjust the distribution strategy by monitoring the frequency of cookie usage and request load.

[0016] Preferably, the execution process of the dynamic scheduling strategy specifically includes:

[0017] After receiving a crawler task request, select any of the above scheduling strategies; when the random selection strategy is selected, obtain a list of all available cookies from the cookie pool, generate a random number, and select the corresponding cookie; when the weighted selection strategy is selected, obtain a list of all available cookies from the cookie pool, calculate a score based on the historical success rate and freshness, and select the cookie with the highest score; when the behavioral simulation selection strategy is selected, analyze the collected data type and target user behavior pattern, and query for matching cookies;

[0018] Assign the selected cookie to the crawler task.

[0019] Preferably, the system is integrated with a web crawler, specifically including:

[0020] Interface design module: provides interfaces for obtaining cookies, returning cookies, and querying cookie status. The crawler program interacts with the cookie pool through function calls;

[0021] Automatic update and maintenance module: monitors the validity of cookies in real time. When a cookie expires, becomes invalid, or is marked as abnormal, it triggers the update mechanism and regularly cleans up invalid or expired cookies.

[0022] Preferably, the security management measures of the Cookie storage and classification module specifically include:

[0023] Encryption processing: Use symmetric encryption algorithm or asymmetric encryption algorithm to encrypt cookies, and store the decryption key through the security key management unit;

[0024] Compliance check: During the cookie collection and use process, verify the target website's terms of use and compliance with laws and regulations, avoid collecting protected user information, and perform legal authorization checks before obtaining cookies;

[0025] Access control and auditing: Limit access to the cookie pool to authorized crawlers and administrators, and generate audit records for login collection, cookie allocation, and update operations.

[0026] Beneficial effects: The application supports a multi-platform, safe and efficient Cookie pool management system, which can effectively circumvent the anti-crawler mechanism of the target website and improve the stability and continuity of the crawler program during the data collection process. By managing and using a variety of cookies with different permissions, more comprehensive website information can be obtained, and the flexibility and scalability of the Cookie pool can be enhanced, so that it can adapt to changes in different types of websites and meet the needs of large-scale crawler program application scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without paying any creative work.

[0028] Figure 1 A schematic diagram of the flow architecture of a secure and efficient cookie pool management system supporting multiple platforms provided in an embodiment of the present application;

[0029] Figure 2 A schematic diagram of the execution flow of the dynamic scheduling strategy provided in an embodiment of the present application. DETAILED DESCRIPTION

[0030] The following is a clear and complete description of the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments of this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0031] In this document, the term "comprising" is intended to encompass non-exclusive inclusion, such that a process, method, article, or apparatus that includes a list of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising..." does not preclude the presence of additional identical elements in the process, method, article, or apparatus that includes the element.

[0032] This embodiment provides a Figure 1 The shown cookie pool management system supports multiple platforms, which is safe and efficient, including:

[0033] The multi-source cookie collection module is configured to automatically collect cookies from multiple platforms and users. The automated collection involves recording login information for different domain name websites and ordinary users, member users, and administrator users through configuration files or databases. The login information includes account numbers, passwords, and login URLs.

[0034] The cookie storage and classification module is configured to: classify cookies according to website domain name, user type and permission level, and adopt security management measures to encrypt the classified data and store it in a distributed storage unit and perform secure management of cookies. Preferably, the distributed storage unit includes a distributed file system or a distributed database;

[0035] The cookie scheduling and allocation module is configured to: select cookies from the cookie storage and classification module and allocate them to crawler tasks based on dynamic scheduling strategy and load balancing processing, the dynamic scheduling strategy is used to select target cookies, and the load balancing processing is used to adjust the allocation of crawler tasks.

[0036] In one embodiment, the multi-source cookie collection module includes a simulated form login interface and an API login interface; the simulated form login interface is used to collect cookies from ordinary websites that do not provide public API or OAuth authentication, and the API login interface is used to collect cookies from websites that support API or OAuth authentication.

[0037] The simulated form login interface simulates the process of a user manually filling out a login form (such as an account and password input box) on a web page and submitting it, and sends an HTTP request (usually a POST request) to the login interface of the target website to obtain the cookies after login. The technical implementation includes the following steps: parsing the HTML form structure of the target website's login page, extracting form fields (such as username, password, csrf_token, etc.); constructing request parameters containing user login information (account, password) and form fields, and sending a request through an HTTP client (such as Python's requests library); parsing the response results, extracting and storing the cookies returned by the server (such as the fields in the Set-Cookie response header). Applicable scenarios are: most ordinary websites (websites that do not provide public APIs or OAuth authentication); scenarios where simple anti-crawl mechanisms need to be bypassed (such as verification code recognition requires combination with other tools).

[0038] The API login interface completes the login by calling the programming interface (API) or authentication protocol (such as OAuth, OpenIDConnect) provided by the target website, and directly obtains authorization credentials (such as Token) or Cookie. The technical implementation includes the following steps: For websites that provide public APIs: by calling their login API, passing in the account number, password or API key, and obtaining the Cookie or Token in the response; For OAuth authentication websites: follow the OAuth process (such as AuthorizationCodeGrant), obtain the authorization code through front-end redirection or server-side interaction, exchange it for an access token (AccessToken), and carry the token in subsequent requests to obtain the user Cookie. Applicable scenarios are: websites that support API or third-party authentication (such as OAuth) (such as GitHub, WeChat and other platforms); scenarios that require efficient and standardized login processes can directly use existing API documents to achieve automated docking.

[0039] The simulated form login interface covers traditional websites, and the API login interface is adapted to modern platforms. The combination of the two makes the system compatible with various types of target sites such as ordinary websites and OAuth-certified websites, realizing "multi-platform cookie collection". By pre-storing login information (account, password, login URL) of different websites (domain names) and user types (ordinary users, members, administrators) in configuration files or databases, the system can automatically select the login method according to the site type without manual intervention, realizing "automatic collection of multi-user cookies". For the login verification mechanisms of different websites (such as form verification and OAuth token verification), using the corresponding login interface can reduce the risk of anti-crawl triggered by abnormal login behavior (such as simulated form login can be combined with browser fingerprint simulation, and API login can use the official SDK compliance call).

[0040] In one embodiment, the scheduling strategy includes random selection, weighted selection, and user behavior simulation-based selection; wherein, the weighted selection specifically includes: selecting cookies after assigning weights based on the cookie's historical success rate and freshness; and the user behavior simulation-based selection specifically includes: selecting cookies based on the collected data type and the target user's behavior pattern;

[0041] The load balancing process is to automatically adjust the distribution strategy by monitoring the frequency of cookie usage and request load.

[0042] like Figure 2 As shown, the execution process of the dynamic scheduling strategy specifically includes:

[0043] After receiving a crawler task request, select any of the above scheduling strategies; when the random selection strategy is selected, obtain a list of all available cookies from the cookie pool, generate a random number, and select the corresponding cookie; when the weighted selection strategy is selected, obtain a list of all available cookies from the cookie pool, calculate a score based on the historical success rate and freshness, and select the cookie with the highest score; when the behavioral simulation selection strategy is selected, analyze the collected data type and target user behavior pattern, and query for matching cookies;

[0044] Assign the selected cookie to the crawler task.

[0045] As a feasible implementation of this embodiment, the random selection strategy is to generate a dynamic random factor by combining timestamp entropy and task type entropy to avoid the predictability of pure randomness. Specifically, the execution process of the random selection strategy includes:

[0046] 1-Dynamic entropy calculation, the formula is:

[0047]

[0048] Where timestamp(t) is the 32-bit hash value of the current timestamp (unit: milliseconds); task_type(c) is the crawler task type code (such as text collection = 001, image collection = 010, structured data = 100); It is a bitwise XOR operation to enhance randomness.

[0049] 2-Cookie index mapping

[0050]

[0051] Where N is the length of the available cookie list; mod is the modulo operation, which ensures that the index falls within the valid range.

[0052] The random selection strategy of this embodiment is based on the dual entropy sources of time and task type, which makes the random selection context-relevant, and the dynamic entropy value randomness prevents fixed random patterns from being recognized by the website, thereby enhancing the anti-crawler capability.

[0053] As a feasible implementation of this embodiment, a weighted selection strategy is to abstract the cookie's historical success rate, freshness, and task matching degree into a three-dimensional space vector and calculate the score through a gravitational field model to avoid the limitations of linear weighting. Specifically, the execution process of the weighted selection strategy includes:

[0054] (1) Calculate the historical success rate S, S∈[0,1]

[0055]

[0056] Calculate freshness F, F∈[0,1]

[0057]

[0058] The task matching degree M is obtained based on the matching score (manually preset or generated by machine learning) between the user permissions of the cookie and the sensitivity of the task data, where M∈[0,1].

[0059] 2. Calculate the gravitational field score using the following formula:

[0060]

[0061] Among them, ∈ is a minimum constant used to prevent the denominator from being 0;

[0062] In the calculation formula of the gravitational field score E, the numerator is the product of three-dimensional parameters (equivalent to volume), and the denominator is the Euclidean distance of the parameter deviation from the maximum value. The score reflects the attraction intensity of the gravitational center in the parameter space.

[0063] 3. Decision rule: Select the cookie with the largest gravitational field score E as the target cookie. If there are multiple maximum values, further compare the arithmetic sum of S+F+M, and take the cookie with the largest arithmetic sum as the target cookie.

[0064] Based on the weight selection strategy, a nonlinear product and inverse distance model is adopted to simulate the interaction of the physical gravitational field, avoiding the linear limitations of weighted average. The nonlinear score of the gravitational field model makes the cookie usage frequency distribution closer to real users, enhancing anti-crawler capabilities.

[0065] As a feasible implementation of this embodiment, the user behavior simulation selection strategy is to abstract the user behavior pattern into a state transition sequence of a hidden Markov model and match the historical behavior trajectory of the cookie through a dynamic programming algorithm. Specifically, the execution process of the user behavior simulation selection strategy includes:

[0066] 1. Behavioral feature extraction

[0067] Define the basic behavior state set Q = {q1,q2,…,q n}(e.g. “Homepage → List Page → Details Page” is q1 → q2 → q3);

[0068] Each Cookie corresponds to a historical behavior track O = {o1, o2, ..., o T}, where o t ∈Q.

[0069] 2. Target behavior pattern modeling

[0070] Construct the HMM model of the target behavior λ=(π,A,B), where π is the initial state probability vector; A is the state transition matrix, A ij =P(q j |q i ); B is the observation probability matrix, B j (k)=P(o t =q k |q j ).

[0071] 3. Matching calculation

[0072]

[0073] Among them, α t (q i ) is the time t calculated by the forward algorithm in state q i The probability of ; taking the logarithm to avoid numerical underflow, the final score reflects the degree of fit between the cookie behavior trajectory and the target pattern.

[0074] The user behavior simulation selection strategy is based on the probabilistic modeling of temporal state transitions, which can capture the dynamic dependencies of behavior patterns and handle the complex matching of dynamic behavior sequences. Hidden Markov matching can simulate the long-term dependencies of user behaviors, reduce the predictability of behavior patterns, and enhance anti-crawler capabilities.

[0075] It is feasible to integrate the system with a web crawler, specifically including:

[0076] Interface design module: provides interfaces for obtaining cookies, returning cookies, and querying cookie status. The crawler program interacts with the cookie pool through function calls;

[0077] Automatic update and maintenance module: monitors the validity of cookies in real time. When a cookie expires, becomes invalid, or is marked as abnormal, it triggers the update mechanism (re-acquires it according to the original collection method or selects a replacement from the backup cookie pool), and regularly cleans up invalid or expired cookies.

[0078] In one embodiment, the security management measures of the Cookie storage and classification module specifically include:

[0079] Encryption processing: Use a symmetric encryption algorithm or an asymmetric encryption algorithm to encrypt the cookie, and store the decryption key through a security key management unit, where the security key management unit can be any one of the existing technologies;

[0080] Compliance check: During the cookie collection and use process, verify the target website's terms of use and compliance with laws and regulations, avoid collecting protected user information, and perform legal authorization checks before obtaining cookies;

[0081] Access control and auditing: Limit access to the cookie pool to authorized crawlers and administrators, and generate audit records for login collection, cookie allocation, and update operations.

[0082] In the Cookie pool management system supporting multiple platforms, which is secure and efficient in this embodiment:

[0083] 1. The Cookie Storage and Classification Module is the core data storage unit of the Cookie pool and is responsible for completing the following functions:

[0084] 1. Use a distributed storage system (such as a distributed file system or database) to store cookies;

[0085] 2. Categorize and store cookies based on website domain name, user type, and permission level to form a structured cookie pool;

[0086] 3. Encrypt cookies (such as AES / RSA algorithm) to ensure data security.

[0087] 2. The Cookie Scheduling and Allocation Module is the dynamic control unit of the Cookie pool. Based on the Cookie resources in the Cookie Storage and Classification Module, it implements the following functions:

[0088] 1. Select appropriate cookies from the cookie storage and classification module through dynamic scheduling strategies (random selection, weighted selection, and user behavior simulation selection);

[0089] 2. Monitor the frequency of cookie usage through load balancing to avoid excessive calls to a single cookie and distribute resources evenly.

[0090] 3. The Cookie scheduling and allocation module relies on the Cookie storage and classification module's Cookies, and is the policy execution layer for achieving secure and efficient management of the Cookie pool.

[0091] The cookie pool is a system composed of a multi-source cookie collection module, a cookie scheduling and allocation module, and a cookie storage and classification module. The cookie storage and classification module provides storage, classification, and security protection for cookies. The cookie scheduling and allocation module provides dynamic selection and load balancing of cookies. The multi-source cookie collection module provides automated acquisition of cookies (such as simulated login and API docking). The three work together to realize the "collection → storage → scheduling" life cycle management of cookies.

[0092] In summary, the secure and efficient cookie pool management system supporting multiple platforms in this embodiment has the following technical features:

[0093] 1. Effectively circumvent anti-crawler mechanisms

[0094] By using multiple different cookies for requests and employing dynamic scheduling and load balancing strategies, the web crawler's request pattern is closer to real user behavior, significantly reducing the risk of being identified by the target website's anti-crawler mechanism. This improves the stability of the web crawler and reduces data collection interruptions caused by restrictions or bans.

[0095] 2. Expand the scope of data collection

[0096] Because the cookie pool encompasses cookies from various user types and permission levels, web crawlers can access a wider range of website content. Whether it's public information, member-only content, or other restricted information, they can collect it more comprehensively, improving the integrity and richness of the data and providing stronger data support for subsequent data analysis and applications.

[0097] 3. Enhance system flexibility and scalability

[0098] The multi-source cookie collection module easily adds support for new types of websites or user types by simply configuring the corresponding login information and collection rules. Its distributed storage architecture and flexible scheduling strategies address the massive cookie management demands of large-scale web crawler applications. The system can be easily scaled to accommodate growing web crawler traffic and quickly adapt to changes in target websites, such as changes in login methods and adjustments to cookie structures.

[0099] 4. Improve security and privacy protection

[0100] Encrypted storage and strict security measures ensure the security of cookies and protect user privacy. Compliance checks and audit mechanisms ensure that the use of the entire cookie pool is within legal and compliant boundaries, reducing legal risks arising from privacy issues or illegal operations, and enhancing the trust of users and website operators in the web crawler system.

[0101] In the embodiments provided herein, it should be understood that the embodiments described herein can be implemented in hardware, software, firmware, middleware, code, or any appropriate combination thereof. For hardware implementation, the processor can be implemented in one or more of the following units: an application-specific integrated circuit (ASIC), a digital signal processor (DSP), a digital signal processing device (DSPD), a programmable logic device (PLD), a field programmable gate array (FPGA), a processor, a controller, a microcontroller, a microprocessor, other electronic units designed to implement the functions described herein, or a combination thereof. For software implementation, part or all of the processes of the embodiments can be completed by instructing the relevant hardware through a computer program. When implemented, the above program can be stored in a computer-readable storage medium or transmitted as one or more instructions or codes on a computer-readable storage medium. Computer-readable storage media include computer storage media and communication media, wherein the communication media include any medium that facilitates the transmission of a computer program from one place to another. The storage medium can be any available medium that a computer can access. The computer-readable storage medium can include, but is not limited to, RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage media or other magnetic storage devices, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer.

[0102] Finally, it should be noted that the above is only a preferred embodiment of the present application and is not intended to limit the present application. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art can still modify the technical solutions described in the aforementioned embodiments or make equivalent replacements for some of the technical features therein. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application should be included in the scope of protection of the present application.

Claims

1. A secure and efficient cookie pool management system that supports multiple platforms, characterized by: include: The multi-source cookie collection module is configured to automatically collect cookies from multiple platforms and users. The automated collection involves recording login information for different domain name websites and ordinary users, member users, and administrator users through configuration files or databases. The login information includes account numbers, passwords, and login URLs. The cookie storage and classification module is configured to: classify cookies according to website domain name, user type and permission level, adopt security management measures, encrypt the classified data and store it in a distributed storage unit, and perform secure management of cookies; The cookie scheduling and allocation module is configured to: select cookies from the cookie storage and classification module and allocate them to crawler tasks based on dynamic scheduling strategy and load balancing processing, the dynamic scheduling strategy is used to select target cookies, and the load balancing processing is used to adjust the allocation of crawler tasks.

2. The Cookie pool management system supporting multiple platforms, which is safe and efficient according to claim 1, is characterized in that: The multi-source cookie collection module includes a simulated form login interface and an API login interface; the simulated form login interface is used to collect cookies from ordinary websites that do not provide public API or OAuth authentication, and the API login interface is used to collect cookies from websites that support API or OAuth authentication.

3. The Cookie pool management system supporting multiple platforms, which is safe and efficient according to claim 1, is characterized in that: The distributed storage unit includes a distributed file system or a distributed database.

4. The Cookie pool management system supporting multiple platforms, which is safe and efficient according to claim 1, is characterized in that: The scheduling strategies include random selection, weighted selection, and user behavior simulation selection. The weighted selection specifically involves assigning weights based on the cookie's historical success rate and freshness, and the user behavior simulation selection specifically involves selecting cookies based on the collected data type and target user behavior pattern. The load balancing process is to automatically adjust the distribution strategy by monitoring the frequency of cookie usage and request load.

5. The Cookie pool management system supporting multiple platforms, which is safe and efficient according to claim 4, is characterized in that: The execution process of the dynamic scheduling strategy specifically includes: After receiving a crawler task request, select any of the above scheduling strategies; when the random selection strategy is selected, obtain a list of all available cookies from the cookie pool, generate a random number, and select the corresponding cookie; when the weighted selection strategy is selected, obtain a list of all available cookies from the cookie pool, calculate a score based on the historical success rate and freshness, and select the cookie with the highest score; when the behavioral simulation selection strategy is selected, analyze the collected data type and target user behavior pattern, and query for matching cookies; Assign the selected cookie to the crawler task.

6. The Cookie pool management system supporting multiple platforms, which is safe and efficient according to claim 1, is characterized in that: The system is integrated with web crawlers, specifically including: Interface design module: provides interfaces for obtaining cookies, returning cookies, and querying cookie status. The crawler program interacts with the cookie pool through function calls; Automatic update and maintenance module: monitors the validity of cookies in real time. When a cookie expires, becomes invalid, or is marked as abnormal, it triggers the update mechanism and regularly cleans up invalid or expired cookies.

7. The cookie pool management system supporting multiple platforms, which is safe and efficient according to claim 1 or 3, is characterized in that: The security management measures of the Cookie storage and classification module specifically include: Encryption processing: Use symmetric encryption algorithm or asymmetric encryption algorithm to encrypt cookies, and store the decryption key through the security key management unit; Compliance check: During the cookie collection and use process, verify the target website's terms of use and compliance with laws and regulations, avoid collecting protected user information, and perform legal authorization checks before obtaining cookies; Access control and auditing: Limit access to the cookie pool to authorized crawlers and administrators, and generate audit records for login collection, cookie allocation, and update operations.

Images