Automatic data acquisition method and system and related equipment
Through the context isolation and code injection functions of the Playwright framework, the data acquisition difficulties of the existing technology in dynamic pages and complex encryption environments are solved, and the accuracy and stability of efficient multi-task concurrency, data isolation and data acquisition are achieved.
Patent Information
- Application Number
- CN202510078761.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-01-17
- Publication Date
- 2025-05-06
AI Technical Summary
When existing automated data acquisition technologies face the dynamic pages, complex encryption algorithms and anti-acquisition mechanisms of target websites, it is difficult to achieve the accuracy and stability of efficient multi-task concurrency, data isolation and data acquisition.
Using the context isolation and code injection function provided by the Playwright framework, we create an independent context environment in a single browser instance, and use code injection technology to track the page changes of the target website in real time, and execute encryption algorithms to obtain the encryption parameters required for API requests.
It realizes efficient multi-task concurrency and data isolation, improves the success rate and accuracy of data acquisition, significantly improves the efficiency and stability of data acquisition, and has strong scalability to adapt to the collection needs of multiple platforms and multiple accounts.
Smart Images
Figure CN119945768A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of automated processing, and more specifically, to an automated data acquisition method, system and related equipment. Background Art
[0002] With the rapid development of the Internet, data has become a key factor in decision-making, innovation and competition in various industries. In the field of e-commerce, real-time grasp of competitive product price dynamics and inventory information is required to accurately price and optimize the supply chain; the financial industry needs to closely track market trends and stock price fluctuations to provide a basis for investment strategies; public opinion monitoring is to capture the direction of social public opinion in real time to help companies respond to public relations crises in a timely manner. Automated data collection technology has emerged as the times require, opening up an efficient way to meet the demand for massive data.
[0003] However, existing technologies face many difficulties. On the one hand, in order to protect data security, target websites are constantly upgrading their anti-collection strategies. Page loading uses multi-layer nested frameworks, dynamically loaded content, and asynchronous data requests. Traditional tools such as Puppeteer and Selenium often have difficulty capturing complete information, resulting in data omissions. At the same time, anti-collection mechanisms are frequently introduced, and verification codes have evolved from simple graphics to complex sliding and clicking forms. IP restrictions and user behavior analysis have also become more accurate. Automated tools are easily detected, and task interruptions occur frequently.
[0004] On the other hand, the demand for multi-tasking concurrency is high, but traditional tools are not up to the task. Puppeteer and Selenium open independent browser instances for multiple tasks, which consumes huge amounts of memory and CPU resources and is inefficient. Commercial tools such as Shadow Blade, although they have certain concurrency capabilities, have shortcomings in resource optimization and task isolation, and are difficult to adapt to large-scale tasks. Especially in the scenario of switching between multiple accounts, account information is easily confused and data accuracy is worrying. In addition, in the face of network fluctuations and page anomalies, existing tools lack effective responses, which often leads to waste of resources and failure of tasks.
[0005] Based on this, the present application provides a more comprehensive automated data collection solution, which effectively makes up for the deficiencies of the existing technology and provides a comprehensive and breakthrough solution to the difficulties of automated data collection in various industries. Summary of the invention
[0006] In view of this, the present application provides an automated data collection method, system and related equipment, which are superior to the existing technology in terms of concurrency performance, dynamic page support, security and scalability, and are particularly suitable for data collection needs in high-concurrency, high-dynamic and complex scenarios.
[0007] An automated data collection method, comprising:
[0008] Receive the account and password information of the target website to be collected entered by the user through the configuration file, and generate a task queue according to the collection requirements;
[0009] Use the context isolation function provided by the Playwright framework to create an independent context environment in a single browser instance, start multiple browser instances at the same time and isolate data;
[0010] Using the code injection function provided by the Playwright framework, the encryption algorithm consistent with the target website to be collected is executed inside the browser page to obtain the encryption parameters;
[0011] An API request carrying the encryption parameters is generated and initiated according to the task queue to complete data collection.
[0012] Optionally, use the code injection function provided by the Playwright framework to execute an encryption algorithm consistent with the target website to be collected inside the browser page to obtain encryption parameters, including:
[0013] Detect the current encryption execution environment;
[0014] Start a browser and navigate to the target website to be collected, and determine the encryption algorithm of the target website to be collected;
[0015] If the current encryption execution environment is to execute encryption after the page has been loaded, then the target code consistent with the encryption algorithm is introduced into the context of the browser page through the first code injection method, and the encryption parameters are returned after the execution is completed;
[0016] If the current encryption execution environment is to perform encryption before the request is sent, the specified API request is intercepted by the second code injection method, and the encryption operation and request reconstruction are performed by the encryption algorithm in the interception processing function to obtain encryption parameters.
[0017] Optionally, use the context isolation feature provided by the Playwright framework to create an independent context environment in a single browser instance, start multiple browser instances at the same time and isolate data, including:
[0018] In a single browser instance, a context environment containing independent account information is created by calling the Playwright framework, wherein a persistence operation is adopted for Cookies;
[0019] Multiple browser instances are started in parallel, and during the startup process, data isolation in multiple attribute dimensions is performed for the context environment created by each browser instance.
[0020] Optionally, before using the code injection function provided by the Playwright framework to execute an encryption algorithm consistent with the target website to be collected inside the browser page and obtaining the encryption parameters, it also includes:
[0021] The verification code processing function provided by the Playwright framework is used to perform OCR automatic recognition and dynamic behavior simulation on the target website to be collected.
[0022] Optionally, also include:
[0023] The collected data is cleaned and deduplicated, and then stored in local and cloud databases.
[0024] Optionally, the Playwright framework is connected to each of the browsers via a CDP protocol.
[0025] An automated data collection system, comprising a development configuration module, a data collection module and a Playwright framework;
[0026] The development configuration module is used to receive the account and password information of the target website to be collected input by the user through the configuration file, and generate a task queue according to the collection requirements;
[0027] The Playwright framework is used to create an independent context environment in a single browser instance, start multiple browser instances at the same time and isolate data, and execute an encryption algorithm consistent with the target website to be collected inside the browser page to obtain encryption parameters;
[0028] The data collection module is used to generate and initiate an API request carrying the encryption parameters according to the task queue to complete data collection.
[0029] An automated data collection device comprising a memory and a processor;
[0030] The memory is used to store programs;
[0031] The processor is used to execute the program to implement each step of the automatic data collection method as described in any one of the above items.
[0032] A readable storage medium having a computer program stored thereon, characterized in that when the computer program is executed by a processor, each step of the automated data collection method as described in any one of the above items is implemented.
[0033] A computer program product comprises a computer program, wherein when the computer program is run by a processor, each step of the automatic data collection method as described in any one of the above items is executed.
[0034] It can be seen from the above technical solutions that the embodiments of the present application provide an automated data collection method, system and related equipment, and their beneficial effects include:
[0035] First, this application can achieve efficient multi-tasking concurrency and data isolation. With Playwright's independent context management, multiple browser instances can run stably in parallel on different system platforms, with tasks independent of each other, eliminating resource competition;
[0036] Second, this application has the ability to track dynamic pages and encrypt data. It uses code injection technology to track changes in target website pages in real time, accurately executes encryption algorithms to obtain encryption parameters required for API requests, and greatly improves the success rate and accuracy of data collection.
[0037] Third, this application significantly improves the efficiency and stability of data collection, and has strong scalability, can adapt to the collection needs of multiple platforms and multiple accounts, and has significant technical advantages and market application potential;
[0038] In summary, the present invention is superior to the prior art in terms of concurrency performance, dynamic page support, security and scalability, and is particularly suitable for data collection needs in high-concurrency, highly dynamic and complex scenarios. BRIEF DESCRIPTION OF THE DRAWINGS
[0039] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.
[0040] Figure 1 A flowchart of an automated data collection method disclosed in an embodiment of the present application;
[0041] Figure 2 A schematic diagram of an automated data acquisition system disclosed in an embodiment of the present application;
[0042] Figure 3 This is a hardware structure block diagram of an automated data acquisition device disclosed in an embodiment of the present application. DETAILED DESCRIPTION
[0043] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0044] The present application can be used in many general or special computing device environments or configurations, such as personal computers, server computers, handheld or portable devices, tablet devices, multi-processor devices, distributed computing environments including any of the above devices or equipment, etc.
[0045] Next, the present application scheme is introduced. The present application proposes the following technical scheme, please see below for details.
[0046] Figure 1 This is a flow chart of an automated data collection method disclosed in an embodiment of the present application.
[0047] like Figure 1 As shown, the method may include:
[0048] Step S1: receiving the account and password information of the target website to be collected input by the user through the configuration file, and generating a task queue according to the collection requirements.
[0049] Specifically, the system first receives the account and password information of the target website to be collected, which is entered by the user through the configuration file. These configuration files are usually prepared by the user in advance and contain the basic authentication data required to perform the collection task. The configuration file can be in a variety of formats, such as JSON, YAML, or INI, so that the user can choose according to actual needs. After the system reads these configuration files, it saves the account and password information as the basic data for task initialization.
[0050] Next, the system generates a task queue based on the user-defined collection target requirements. These requirements may include the page address to be collected, the type of data to be obtained, the collection frequency, etc. Based on this information, the system builds collection tasks one by one and adds them to the task queue. The task queue is an ordered collection of tasks used to manage and schedule subsequent collection work. The system will assign tasks one by one to subsequent processing modules for execution according to the order of the task queue.
[0051] Step S2: Using the context isolation function provided by the Playwright framework, an independent context environment is created in a single browser instance, and multiple browser instances are started at the same time and data is isolated.
[0052] Specifically, the system uses the context isolation function provided by the Playwright framework to create independent context environments in a single browser instance. These context environments are isolated from each other, and each environment contains independent account information, such as cookies and sessions. By persisting cookies, the system can quickly switch logins between multiple accounts, making it easier to collect data using API requests later.
[0053] At the same time, the system starts multiple browser instances and isolates their data. The Playwright framework connects to each browser through the CDP protocol to ensure that the data between tasks will not conflict or interfere with each other. The parallel operation of multiple instances can significantly improve the collection efficiency and realize the simultaneous processing of multiple tasks. Through refined management and scheduling, the system can make full use of computing resources to ensure the smooth progress of collection tasks.
[0054] Step S3: Using the code injection function provided by the Playwright framework, an encryption algorithm consistent with the target website to be collected is executed inside the browser page to obtain encryption parameters.
[0055] Specifically, the system uses the code injection function provided by the Playwright framework to execute an encryption algorithm consistent with the target website inside the browser page to obtain the encrypted parameters of the API request. In dynamic page collection, the target website usually performs complex encryption on the data parameters of key API requests to protect its data content from direct external access.
[0056] In order to bypass these anti-collection mechanisms, the system implements code injection technology through Playwright's evaluate() and route() methods. These methods allow the system to dynamically insert custom scripts into browser pages and execute encryption algorithms consistent with the target website inside the page. By using the browser's JavaScript environment to directly execute the front-end code of the target page, the system can accurately obtain legitimate encryption results.
[0057] After obtaining the encrypted parameters, the system uses a standard HTTP client (such as Python's requests or aiohttp) to initiate an API request and carry the necessary encrypted parameters to complete data acquisition. This method not only improves collection efficiency, but also reduces unnecessary page rendering and loading, further reducing system resource consumption. In a complex anti-crawling environment, this technology can ensure that the system accurately and efficiently obtains target data.
[0058] Step S4: Generate and initiate an API request carrying the encryption parameters according to the task queue to complete data collection.
[0059] Specifically, the system generates and initiates API requests with encrypted parameters one by one according to the previously generated task queue to complete data collection. The system will take tasks from the task queue and construct API requests based on the page address, data type and other information defined in the task. In the request, the system will carry the previously obtained encrypted parameters to ensure that the request can be correctly identified and processed by the target website.
[0060] The system improves the collection efficiency by concurrent requests, and manages concurrent requests using asynchronous programming techniques or thread pools / process pools. Multiple requests can be made simultaneously without blocking each other. After the request is sent, the system receives the response from the target website and parses the response content to obtain the required data. The parsed data is stored in a specified location, such as a database, file system, or remote storage service. At the same time, the system updates the task status in the task queue and feeds back the execution results and progress of the task to the user. Through this series of operations, the system can efficiently complete the data collection task.
[0061] After completing data collection, the collected data can be cleaned and deduplicated and stored in local and cloud databases.
[0062] Specifically, after completing data collection, the next important step is to clean and deduplicate the collected data. This step is crucial because it is directly related to the accuracy and efficiency of subsequent data analysis and application. Data cleaning mainly includes removing invalid data, such as incomplete, erroneous or obviously inconsistent records with business logic. At the same time, for inconsistent data, the system will perform necessary format conversion to ensure that all data can be presented in a unified format. In addition, in order to avoid data redundancy, the system will also perform deduplication operations to eliminate duplicate data records. After this series of processing, the accuracy and consistency of the data have been significantly improved.
[0063] Subsequently, on the one hand, the processed data will be stored in the local database so that users can access and analyze it locally at any time. On the other hand, for the security and accessibility of the data, the data will also be uploaded to the cloud database for backup and remote access. Cloud storage not only provides higher data security, but also allows data to be accessed at any time and any place, greatly improving the flexibility and availability of data. Through such a storage strategy, the system ensures the integrity, security and accessibility of the data.
[0064] It can be seen from the above technical solutions that the embodiments of the present application provide an automated data collection method, system and related equipment, and their beneficial effects include:
[0065] First, this application can achieve efficient multi-tasking concurrency and data isolation. With Playwright's independent context management, multiple browser instances can run stably in parallel on different system platforms, with tasks independent of each other, eliminating resource competition;
[0066] Second, this application has the ability to track dynamic pages and encrypt data. It uses code injection technology to track changes in target website pages in real time, accurately executes encryption algorithms to obtain encryption parameters required for API requests, and greatly improves the success rate and accuracy of data collection.
[0067] Third, this application significantly improves the efficiency and stability of data collection, and has strong scalability, can adapt to the collection needs of multiple platforms and multiple accounts, and has significant technical advantages and market application potential;
[0068] In summary, the present invention is superior to the prior art in terms of concurrency performance, dynamic page support, security and scalability, and is particularly suitable for data collection needs in high-concurrency, highly dynamic and complex scenarios.
[0069] In some embodiments of the present application, the process of obtaining encryption parameters by executing an encryption algorithm consistent with the target website to be collected inside the browser page using the code injection function provided by the Playwright framework in step S3 is further introduced, which may specifically include:
[0070] Step S31, detecting the current encryption execution environment.
[0071] Specifically, first detect and determine the encryption execution environment of the target website. The encryption execution environment may vary from website to website. Some websites may perform encryption operations after the page is loaded, while others may encrypt data before the API request is sent. By parsing the JavaScript code or network request of the target website, the system can identify the specific execution time and location of the encryption operation, providing key information for subsequent code injection and encryption parameter acquisition.
[0072] Step S32: start the browser and navigate to the target website to be collected, and determine the encryption algorithm of the target website to be collected.
[0073] Specifically, start a browser instance controlled by the Playwright framework and navigate to the target website to be collected. When the browser loads and renders the page, the system will use the tools and methods provided by Playwright to parse and analyze the JavaScript code in the page to determine the encryption algorithm used by the target website.
[0074] Step S33: If the current encryption execution environment is to execute encryption after the page has been loaded, the target code consistent with the encryption algorithm is introduced into the context of the browser page through the first code injection method, and the encryption parameters are returned after execution.
[0075] Specifically, if the system detects that the current encryption execution environment is to perform encryption after the page has been loaded, it will use the first code injection method. Specifically, the system will use Playwright's evaluate() method or other similar execution environment injection techniques to pass a target code that is consistent with the target website's encryption algorithm into the context of the browser page. This code will be executed after the page is loaded and return the encrypted parameters. In this way, the system can simulate the user's normal operation and obtain the encrypted data parameters.
[0076] Step S34: If the current encryption execution environment is to perform encryption before the request is sent, the specified API request is intercepted by the second code injection method, and the encryption operation and request reconstruction are performed by the encryption algorithm in the interception processing function to obtain encryption parameters.
[0077] Specifically, if the system detects that the current encryption execution environment is to perform encryption before the request is sent, it will use the second code injection method. Specifically, the system will use Playwright's route() method or other similar network request interception technologies to intercept the specified API request issued by the target website. In the intercepted request, the system will call an interception processing function, which will use the encryption algorithm of the target website to encrypt the request data and reconstruct the request. In this way, the system can encrypt the data before the request is sent, thereby ensuring that the request issued can meet the encryption requirements of the target website and successfully obtain the encrypted data parameters.
[0078] In some embodiments of the present application, the process of step S2, using the context isolation function provided by the Playwright framework to create an independent context environment in a single browser instance, and simultaneously starting multiple browser instances and isolating data is further introduced, which may specifically include:
[0079] Step S21: In a single browser instance, a context environment containing independent account information is created by calling the Playwright framework, wherein a persistence operation is adopted for Cookie.
[0080] Specifically, step S21 focuses on creating independent context environments within a single browser instance. These environments are generated through the browser.new_context() method, ensuring the isolation of cookies, caches, etc. In this environment, the user login process can be simulated, and the login status (especially Cookies) can be selectively persisted.
[0081] Specifically, once the user successfully logs in, the context.storage_state(path=...) method will be called to save the current storage state (including the login state) to a local file. The advantage of this step is that it allows subsequent tests or data collection to quickly restore the user's login state by loading the saved storage state without having to log in again.
[0082] Step S22: Start multiple browser instances in parallel, and during the startup process, perform data isolation in multiple attribute dimensions for the context environment created by each browser instance.
[0083] Specifically, step S22 focuses on launching multiple browser instances in parallel and creating independent context environments in each of these instances to achieve deeper data isolation. The parallel operation capability of the Playwright framework is fully utilized here, which allows multiple browser instances to run simultaneously. In order to ensure that each instance has a unique data environment, a series of different configuration parameters are specified when they are launched. These parameters cover the user data directory (responsible for persistent storage of user data such as login status, browser cache, etc.) and the proxy IP (used to simulate user behavior in different network environments). Through these carefully designed configurations, we ensure that each browser instance can maintain the independence of its data during operation, thereby achieving the goal of data isolation. This strategy not only improves the efficiency of testing or data collection, but also enhances the accuracy and reliability of the results.
[0084] In addition, before using the code injection function provided by the Playwright framework to execute the encryption algorithm consistent with the target website to be collected inside the browser page and obtain the encryption parameters, it also includes:
[0085] The verification code processing function provided by the Playwright framework is used to perform OCR automatic recognition and dynamic behavior simulation on the target website to be collected.
[0086] Specifically, when using the Playwright framework for automated testing or data collection, you will often encounter verification code problems. To deal with these problems, you can use the verification code processing functions provided by Playwright, including OCR automatic recognition and dynamic behavior simulation.
[0087] OCR automatic recognition: Use OCR technology to recognize the text or numbers in the verification code image and automatically fill them into the corresponding input box. Playwright can be used in conjunction with a third-party OCR library (such as ddddocr) to achieve automatic recognition of verification codes.
[0088] Dynamic behavior simulation: For verification codes that require dynamic operations such as sliding verification codes and click verification codes, Playwright can be used to simulate the user's sliding, clicking and other behaviors to bypass the verification code. For example, for sliding verification codes, Playwright's drag_and_drop() method can be used to simulate sliding operations; for click verification codes, the mouse.click() method can be used to simulate clicking operations.
[0089] In summary, the context isolation function, parallel operation function and verification code processing function provided by the Playwright framework can effectively realize the creation and data isolation of multiple independent context environments, as well as the automatic recognition and dynamic behavior simulation of verification codes. These functions have important application value in scenarios such as automated testing and data collection.
[0090] An automated data collection system provided in an embodiment of the present application is described below. The automated data collection system described below and the automated data collection method described above can refer to each other.
[0091] See also Figure 2 , Figure 2 A schematic diagram of an automated data acquisition system disclosed in an embodiment of the present application.
[0092] like Figure 2 As shown, the automated data acquisition system comprises a development configuration module, a data acquisition module and a Playwright framework;
[0093] The development configuration module is used to receive the account and password information of the target website to be collected input by the user through the configuration file, and generate a task queue according to the collection requirements;
[0094] The Playwright framework is used to create an independent context environment in a single browser instance, start multiple browser instances at the same time and isolate data, and execute an encryption algorithm consistent with the target website to be collected inside the browser page to obtain encryption parameters;
[0095] The data collection module is used to generate and initiate an API request carrying the encryption parameters according to the task queue to complete data collection.
[0096] It can be seen from the above technical solutions that the embodiments of the present application provide an automated data collection method, system and related equipment, and their beneficial effects include:
[0097] First, this application can achieve efficient multi-tasking concurrency and data isolation. With Playwright's independent context management, multiple browser instances can run stably in parallel on different system platforms, with tasks independent of each other, eliminating resource competition;
[0098] Second, this application has the ability to track dynamic pages and encrypt data. It uses code injection technology to track changes in target website pages in real time, accurately executes encryption algorithms to obtain encryption parameters required for API requests, and greatly improves the success rate and accuracy of data collection.
[0099] Third, this application significantly improves the efficiency and stability of data collection, and has strong scalability, can adapt to the collection needs of multiple platforms and multiple accounts, and has significant technical advantages and market application potential;
[0100] In summary, the present invention is superior to the prior art in terms of concurrency performance, dynamic page support, security and scalability, and is particularly suitable for data collection needs in high-concurrency, highly dynamic and complex scenarios.
[0101] Optionally, the Playwright framework uses the code injection function provided by the Playwright framework to execute an encryption algorithm consistent with the target website to be collected inside the browser page to obtain the encryption parameters, which may include:
[0102] Detect the current encryption execution environment;
[0103] Start a browser and navigate to the target website to be collected, and determine the encryption algorithm of the target website to be collected;
[0104] If the current encryption execution environment is to execute encryption after the page has been loaded, then the target code consistent with the encryption algorithm is introduced into the context of the browser page through the first code injection method, and the encryption parameters are returned after the execution is completed;
[0105] If the current encryption execution environment is to perform encryption before the request is sent, the specified API request is intercepted by the second code injection method, and the encryption operation and request reconstruction are performed by the encryption algorithm in the interception processing function to obtain encryption parameters.
[0106] Optionally, the Playwright framework executes the process of creating an independent context environment in a single browser instance by utilizing the context isolation function provided by the Playwright framework, and simultaneously starting multiple browser instances and performing data isolation, which may include:
[0107] In a single browser instance, a context environment containing independent account information is created by calling the Playwright framework, wherein a persistence operation is adopted for Cookies;
[0108] Multiple browser instances are started in parallel, and during the startup process, data isolation in multiple attribute dimensions is performed for the context environment created by each browser instance.
[0109] Optionally, the automated data collection system may further include a verification processing module;
[0110] The verification processing module is used to utilize the code injection function provided by the Playwright framework to execute an encryption algorithm consistent with the target website to be collected inside the browser page, and before obtaining the encryption parameters, utilize the verification code processing function provided by the Playwright framework to perform OCR automatic recognition and dynamic behavior simulation on the target website to be collected.
[0111] Optionally, the data collection module can also be used to store the collected data in local and cloud databases after cleaning and deduplication.
[0112] Optionally, the Playwright framework is connected to each of the browsers via a CDP protocol.
[0113] The automated data collection device provided in the embodiment of the present application can be applied to automated data collection equipment. Figure 3 The hardware structure diagram of the automated data acquisition equipment is shown in FIG. Figure 3 ,The hardware structure of the automated data acquisition device may include: at least one processor 1, at least one communication interface 2, at least one memory 3 and at least one communication bus 4;
[0114] In the embodiment of the present application, the number of the processor 1, the communication interface 2, the memory 3, and the communication bus 4 is at least one, and the processor 1, the communication interface 2, and the memory 3 communicate with each other through the communication bus 4;
[0115] The processor 1 may be a central processing unit CPU, or an application-specific integrated circuit ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement the embodiments of the present invention, etc.;
[0116] The memory 3 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), etc., such as at least one disk memory;
[0117] The memory stores a program, and the processor can call the program stored in the memory, wherein the program is used to:
[0118] Receive the account and password information of the target website to be collected entered by the user through the configuration file, and generate a task queue according to the collection requirements;
[0119] Use the context isolation function provided by the Playwright framework to create an independent context environment in a single browser instance, start multiple browser instances at the same time and isolate data;
[0120] Using the code injection function provided by the Playwright framework, the encryption algorithm consistent with the target website to be collected is executed inside the browser page to obtain the encryption parameters;
[0121] An API request carrying the encryption parameters is generated and initiated according to the task queue to complete data collection.
[0122] Optionally, the detailed functions and extended functions of the program may refer to the above description.
[0123] The embodiment of the present application further provides a readable storage medium, which may store a program suitable for execution by a processor, wherein the program is used to:
[0124] Receive the account and password information of the target website to be collected entered by the user through the configuration file, and generate a task queue according to the collection requirements;
[0125] Use the context isolation function provided by the Playwright framework to create an independent context environment in a single browser instance, start multiple browser instances at the same time and isolate data;
[0126] Using the code injection function provided by the Playwright framework, the encryption algorithm consistent with the target website to be collected is executed inside the browser page to obtain the encryption parameters;
[0127] An API request carrying the encryption parameters is generated and initiated according to the task queue to complete data collection.
[0128] Optionally, the detailed functions and extended functions of the program may refer to the above description.
[0129] The present application also provides a computer program product, including a computer program. When the computer program is executed by a processor, the execution method is as follows:
[0130] Receive the account and password information of the target website to be collected entered by the user through the configuration file, and generate a task queue according to the collection requirements;
[0131] Use the context isolation function provided by the Playwright framework to create an independent context environment in a single browser instance, start multiple browser instances at the same time and isolate data;
[0132] Using the code injection function provided by the Playwright framework, the encryption algorithm consistent with the target website to be collected is executed inside the browser page to obtain the encryption parameters;
[0133] An API request carrying the encryption parameters is generated and initiated according to the task queue to complete data collection.
[0134] Optionally, the detailed functions and extended functions of the program may refer to the above description.
[0135] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.
[0136] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.
[0137] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. An automated data collection method, characterized in that: include: Receive the account and password information of the target website to be collected entered by the user through the configuration file, and generate a task queue according to the collection requirements; Use the context isolation function provided by the Playwright framework to create an independent context environment in a single browser instance, start multiple browser instances at the same time and isolate data; Using the code injection function provided by the Playwright framework, the encryption algorithm consistent with the target website to be collected is executed inside the browser page to obtain the encryption parameters; An API request carrying the encryption parameters is generated and initiated according to the task queue to complete data collection.
2. The method according to claim 1, characterized in that Using the code injection function provided by the Playwright framework, the encryption algorithm consistent with the target website to be collected is executed inside the browser page to obtain encryption parameters, including: Detect the current encryption execution environment; Start a browser and navigate to the target website to be collected, and determine the encryption algorithm of the target website to be collected; If the current encryption execution environment is to execute encryption after the page has been loaded, then the target code consistent with the encryption algorithm is introduced into the context of the browser page through the first code injection method, and the encryption parameters are returned after the execution is completed; If the current encryption execution environment is to perform encryption before the request is sent, the specified API request is intercepted by the second code injection method, and the encryption operation and request reconstruction are performed by the encryption algorithm in the interception processing function to obtain encryption parameters.
3. The method according to claim 1, characterized in that Use the context isolation function provided by the Playwright framework to create an independent context environment in a single browser instance, start multiple browser instances at the same time and isolate data, including: In a single browser instance, a context environment containing independent account information is created by calling the Playwright framework, wherein a persistence operation is adopted for Cookies; Multiple browser instances are started in parallel, and during the startup process, data isolation in multiple attribute dimensions is performed for the context environment created by each browser instance.
4. The method according to claim 1, characterized in that Before using the code injection function provided by the Playwright framework to execute the encryption algorithm consistent with the target website to be collected inside the browser page and obtain the encryption parameters, it also includes: The verification code processing function provided by the Playwright framework is used to perform OCR automatic recognition and dynamic behavior simulation on the target website to be collected.
5. The method according to claim 1, characterized in that Also includes: The collected data is cleaned and deduplicated, and then stored in local and cloud databases.
6. The method according to claim 1, characterized in that The Playwright framework is connected to each of the browsers via the CDP protocol.
7. An automated data acquisition system, characterized in that: Including development configuration module, data collection module and Playwright framework; The development configuration module is used to receive the account and password information of the target website to be collected input by the user through the configuration file, and generate a task queue according to the collection requirements; The Playwright framework is used to create an independent context environment in a single browser instance, start multiple browser instances at the same time and isolate data, and execute an encryption algorithm consistent with the target website to be collected inside the browser page to obtain encryption parameters; The data collection module is used to generate and initiate an API request carrying the encryption parameters according to the task queue to complete data collection.
8. An automated data collection device, characterized in that: including memory and processor; The memory is used to store programs; The processor is used to execute the program to implement each step of the automatic data collection method as described in any one of claims 1-6.
9. A readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, each step of the automated data collection method according to any one of claims 1 to 6 is implemented.
10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, each step of the automatic data collection method according to any one of claims 1 to 6 is executed.