Universal data acquisition method and system
By configuring the packet capture environment, reverse parsing parameters, and writing customized programs, the problem of bypassing anti-crawling methods during data collection was solved, and efficient data collection was achieved.
Patent Information
- Application Number
- CN202511014737.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-07-23
- Publication Date
- 2025-11-14
AI Technical Summary
Existing technologies require a significant amount of time and effort to reverse engineer and anti-scraping methods during data collection, and the cost of collection may exceed the value of the data, resulting in low collection efficiency.
This paper provides a general data acquisition method and system, including packet capture environment configuration, parameter reverse parsing, customized program writing, and data acquisition and processing. It bypasses anti-scraping detection by compiling open-source Chromium, modifying mobile kernel source code, using Xposed or Frida scripts, rewriting the ssl_context environment, adding RPC remote interaction interface, and designing a collection entry table to bypass common anti-scraping methods.
It enables the rapid development of customized data collection programs, bypassing common anti-scraping methods, reducing analysis and bypass time, and improving data collection efficiency.
Smart Images
Figure CN120956597A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of data acquisition technology, specifically to general data acquisition methods and systems. Background Technology
[0002] In today's rapidly developing information and intelligent world, big data is constantly influencing people's lives and greatly improving their convenience. However, collecting various types of data often requires a significant investment of time and energy in reverse engineering and countermeasures. Even after successfully reverse engineering the parameters, there are still various risk control measures to contend with. The ultimate challenge in this struggle is cost; when the cost of data collection exceeds the value of the data itself, no one will bother collecting it.
[0003] The technical problem that needs to be solved is how to develop a customized data collection program to collect data according to requirements and bypass common anti-scraping methods. Summary of the Invention
[0004] The technical objective of this invention is to address the above-mentioned shortcomings by providing a general data acquisition method and system to solve the technical problem of how to develop a customized data acquisition program to collect data according to requirements and bypass common anti-scraping methods.
[0005] In a first aspect, the present invention provides a general data acquisition method, comprising the following steps:
[0006] Packet capture environment configuration: Determine the platform and platform type of the interface to be captured based on the collection requirements, and configure the packet capture environment and packet capture scheme. The platform type includes web, mobile application and mini-program.
[0007] Parameter reverse parsing: For the captured data packets, the special interface parameters in the data packets that conform to the predetermined rules are analyzed to determine the generation method of the special parameters and obtain the parameter parsing results;
[0008] Customized program writing: Based on the parameter parsing results, a customized data acquisition program is written, and the interface to be collected obtains data from the target evaluation by executing the data acquisition program;
[0009] Data Acquisition and Processing: Execute the data acquisition program, collect data specified in the predefined acquisition entry table, perform anti-crawling processing during the acquisition process, and upload the acquired data to the data server.
[0010] As a preferred option, the packet capture environment configuration includes the following steps:
[0011] For web-based platforms, compile the open-source Chromium to implement random fingerprints and bypass certain anti-debugging measures at the underlying level.
[0012] For the WeChat Mini Program platform, wechatwin.dll is patched according to the WeChat version to enable DevTools to open in the Mini Program, which facilitates packet capture and debugging.
[0013] For mobile platforms, modify the mobile kernel source code to customize the ROM, set up the packet capture environment and hook scripts, install the packet capture software certificate in the system directory, and bypass some common SSL pinning detection, VPN detection and proxy detection through Xposed or Frida scripts.
[0014] As a preferred option, the packet capture environment is configured with special parameters including account parameters, encryption parameters, fingerprint parameters, and dynamic cookies. After reverse analysis of these special parameters, the data is collected through parameter restoration or RPC remote calls.
[0015] As a preferred option, configuring the packet capture environment and writing customized programs includes the following steps:
[0016] For general classes, the ssl_context environment is rewritten to bypass the ja3 fingerprint detection of the underlying Python request library, and proxy IP switching and UA switching are encapsulated.
[0017] For the RPC solution, add an RPC remote interaction interface and add account pool reading and switching when collecting account parameters.
[0018] As a preferred embodiment, data acquisition and processing includes the following operations:
[0019] This involves a data collection entry table, which is used to determine which data to collect and the scope of the collection.
[0020] Traverse the entry table and execute the data acquisition program to collect the required data;
[0021] Add anti-scraping logic to specific locations based on anti-scraping measures.
[0022] Secondly, the present invention provides a general data acquisition system, including a packet capture environment configuration module, a parameter reverse parsing module, a customized program writing module, and a data acquisition and processing module;
[0023] The packet capture environment configuration module is used to perform the following: determine the platform and platform type of the interface to be collected according to the collection requirements, and configure the packet capture environment and packet capture scheme. The platform type includes web, mobile application and mini-program.
[0024] The parameter reverse parsing module is used to perform the following: For the captured data packets, it analyzes the special interface parameters in the data packets that conform to the predetermined rules to determine the generation method of the special parameters and obtain the parameter parsing results;
[0025] The customized program writing module is used to perform the following: write a customized data acquisition program based on the parameter parsing results, and the interface to be collected obtains data from the target evaluation by executing the data acquisition program;
[0026] The data acquisition and processing module is used to perform the following: execute the data acquisition program, collect data specified in the predefined acquisition entry table, perform anti-crawling processing during the acquisition process, and upload the acquired data to the data server.
[0027] As a preferred option, the packet capture environment configuration module is used to perform the following operations:
[0028] For web-based platforms, compile the open-source Chromium to implement random fingerprints and bypass certain anti-debugging measures at the underlying level.
[0029] For the WeChat Mini Program platform, wechatwin.dll is patched according to the WeChat version to enable DevTools to open in the Mini Program, which facilitates packet capture and debugging.
[0030] For mobile platforms, modify the mobile kernel source code to customize the ROM, set up the packet capture environment and hook scripts, install the packet capture software certificate in the system directory, and bypass some common SSL pinning detection, VPN detection and proxy detection through Xposed or Frida scripts.
[0031] Preferably, the special parameters include account parameters, encryption parameters, fingerprint parameters, and dynamic cookies. After reverse analysis of the special parameters, they are collected through parameter restoration or RPC remote calls.
[0032] As a preferred option, the customized programming module is used to perform the following operations:
[0033] For general classes, the ssl_context environment is rewritten to bypass the ja3 fingerprint detection of the underlying Python request library, and proxy IP switching and UA switching are encapsulated.
[0034] For the RPC solution, add an RPC remote interaction interface and add account pool reading and switching when collecting account parameters.
[0035] Preferably, the data acquisition and processing module is used to perform the following operations:
[0036] This involves a data collection entry table, which is used to determine which data to collect and the scope of the collection.
[0037] Traverse the entry table and execute the data acquisition program to collect the required data;
[0038] Add anti-scraping logic to specific locations based on anti-scraping measures.
[0039] The general data acquisition method and system of the present invention have the following advantages: the data acquisition platform is designed for common web, app and mini-program clients, and a customized acquisition program can be quickly developed according to requirements. It can also bypass some common anti-scraping methods, avoiding the need to spend time analyzing and bypassing common anti-scraping points each time. Attached Figure Description
[0040] To more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0041] The invention will be further described below with reference to the accompanying drawings.
[0042] Figure 1 This is a flowchart of a general data acquisition method in Example 1. Detailed Implementation
[0043] The present invention will be further described below with reference to the accompanying drawings and specific embodiments, so that those skilled in the art can better understand and implement the present invention. However, the embodiments are not intended to limit the present invention. In the absence of conflict, the embodiments of the present invention and the technical features in the embodiments can be combined with each other.
[0044] This invention provides a general data acquisition method and system to solve the technical problem of how to develop a customized acquisition program to collect data according to requirements and bypass common anti-crawling methods.
[0045] Example 1:
[0046] This invention provides a general data acquisition method, comprising four steps: packet capture environment configuration, parameter reverse parsing, customized program writing, and data acquisition and processing.
[0047] Step S100 Packet capture environment configuration: Determine the platform and platform type of the interface to be captured according to the collection requirements, and configure the packet capture environment and packet capture scheme. The platform type includes web page, mobile application and mini program.
[0048] As a specific implementation of the packet capture environment configuration, this step includes the following operations:
[0049] (1) For web-based platforms, compile open-source Chromium to implement random fingerprints and bypass certain anti-debugging measures at the underlying level;
[0050] (2) For the mini-program platform, patch wechatwin.dll according to the WeChat version to enable DevTools to be opened in the mini-program, which facilitates packet capture and debugging;
[0051] (3) For mobile platforms, modify the mobile kernel source code to customize the ROM, set up the packet capture environment and hook script, install the certificate of the packet capture software in the system directory, and bypass some common SSL pinning detection, VPN detection and proxy detection through Xposed or Frida script.
[0052] In this embodiment, the interface for data collection may be a webpage, a mobile app, or a mini-program. Based on the collection requirements, the platform to be used for data collection and packet capture are determined. For webpage-based collection, compiling the open-source Chromium allows for features such as random fingerprinting, skipping infinite debugging, and automatically revealing key information. For WeChat mini-program-based collection, it's necessary to patch wechatwin.dll beforehand according to your WeChat version to enable DevTools within the mini-program, allowing for fast packet capture and debugging similar to the webpage. For mobile application-based collection, a rooted phone is required, and the kernel source code needs to be modified to bypass detections such as TracePid or stat. Simultaneously, the packet capture software's certificate should be installed in the system directory, and common SSL pinning, VPN, and proxy detections should be bypassed using Xposed or Frida scripts.
[0053] Step S200 Parameter Reverse Parsing: For the captured data packets, analyze the special interface parameters in the data packets that conform to the predetermined rules to determine the generation method of the special parameters and obtain the parameter parsing results.
[0054] The special parameters include account parameters, encryption parameters, fingerprint parameters, and dynamic cookies. After reverse analysis of these special parameters, they are collected through parameter restoration or RPC remote calls.
[0055] This step involves reverse engineering the parameters based on the captured network traffic. If no special parameters are present, this step can be skipped. Special parameters include: account parameters (cookie or token); encrypted parameters, commonly a hash signature verification value, or encryption of the returned data, usually symmetric encryption followed by encoding; dynamic cookies, commonly found on web pages, such as the Ruishu solution, which is frequently used on some government websites; fingerprint parameters, typically generated based on the browser or mobile environment during registration... This reverse engineering step is also performed within the platform environment determined in the previous step. For more complex parameters, RPC remote calls or automated solutions can be considered; automated solutions will not be discussed further here.
[0056] Step S300: Customized program writing: Based on the parameter parsing results, a customized data acquisition program is written. The interface to be acquired obtains data from the target evaluation by executing the data acquisition program.
[0057] As a specific implementation of customized program writing, this step includes the following operations:
[0058] (1) For general classes, the ssl_context environment is rewritten to bypass the ja3 fingerprint detection of the underlying Python request library and to encapsulate proxy IP switching and UA switching.
[0059] (2) For the RPC solution, add an RPC remote interaction interface and add account pool reading and switching when collecting account parameters.
[0060] This step utilizes a local Python 3.7+ environment. The `ssl_context` environment is rewritten to bypass some Ja3 fingerprint detections on the underlying Python request library, and this is then encapsulated into a new request library. A proxy interface is added to bypass some IP proxy detections, and a user agent (UA) switching interface is added to bypass other UA detections. After encapsulation, various platform interfaces that require data collection can inherit from this library and be customized according to their specific needs. For example, if using remote RPC calls, an RPC remote interaction interface needs to be added; if account collection is required, an account pool reading and switching interface can be added.
[0061] Step S400 Data Acquisition and Processing: Execute the data acquisition program, collect the data specified in the predefined acquisition entry table, perform anti-crawling processing during the acquisition process, and upload the acquired data to the data server.
[0062] As a specific implementation of data acquisition and processing, this step includes the following operations:
[0063] (1) It involves a data collection entry table, which is used to determine which data to collect and the scope of collection.
[0064] (2) Traverse the entry table and execute the data acquisition program to collect the required data;
[0065] (3) Add anti-scraping logic to specific locations based on the anti-scraping situation.
[0066] In this embodiment, before implementing this step, a data collection entry table needs to be designed according to the requirements. The purpose of this table is to determine which data to collect and the scope of collection. For example, if a website collects basic information about some people, and I want to know their height, weight, and date of birth, then I need to provide the program with a data table that uniquely identifies each person. This could be a name or a combination of ID number and name. Then, the program uses this unique identifier to find each person on the website and collects the information we need. The "data collection entry table" serves this purpose; it's used by the data collection program. The program iterates through all the unique identifiers in the "data collection entry table," retrieves each piece of data, and uploads it to the data server. Because the interfaces for switching proxies, request headers, and accounts have already been designed, during the data collection process, to address anti-scraping issues, only these logics need to be added in specific locations.
[0067] Finally, the server extracts the required fields from the uploaded data and stores them in the database according to certain rules.
[0068] The method in this embodiment is designed with a relatively conventional and universal data collection process in order to simplify the analysis process each time, shorten the program development cycle, and deal with some common anti-crawling points.
[0069] Example 2:
[0070] This invention provides a general data acquisition system, including a packet capture environment configuration module, a parameter reverse parsing module, a customized program writing module, and a data acquisition and processing module.
[0071] The packet capture environment configuration module is used to perform the following: determine the platform and platform type of the interface to be collected according to the collection requirements, and configure the packet capture environment and packet capture scheme. The platform type includes web page, mobile application and mini program.
[0072] As a specific implementation of the packet capture environment configuration module, this module performs the following operations:
[0073] (1) For web-based platforms, compile open-source Chromium to implement random fingerprints and bypass certain anti-debugging measures at the underlying level;
[0074] (2) For the mini-program platform, patch wechatwin.dll according to the WeChat version to enable DevTools to be opened in the mini-program, which facilitates packet capture and debugging;
[0075] (3) For mobile platforms, modify the mobile kernel source code to customize the ROM, set up the packet capture environment and hook script, install the certificate of the packet capture software in the system directory, and bypass some common SSL pinning detection, VPN detection and proxy detection through Xposed or Frida script.
[0076] In this embodiment, the interface for data collection may be a webpage, a mobile app, or a mini-program. Based on the collection requirements, the platform to be used for data collection and packet capture are determined. For webpage-based collection, compiling the open-source Chromium allows for features such as random fingerprinting, skipping infinite debugging, and automatically revealing key information. For WeChat mini-program-based collection, it's necessary to patch wechatwin.dll beforehand according to your WeChat version to enable DevTools within the mini-program, allowing for fast packet capture and debugging similar to the webpage. For mobile application-based collection, a rooted phone is required, and the kernel source code needs to be modified to bypass detections such as TracePid or stat. Simultaneously, the packet capture software's certificate should be installed in the system directory, and common SSL pinning, VPN, and proxy detections should be bypassed using Xposed or Frida scripts.
[0077] The parameter reverse parsing module is used to perform the following: For the captured data packets, it analyzes the special interface parameters in the data packets that conform to predetermined rules to determine the generation method of the special parameters and obtain the parameter parsing results.
[0078] The special parameters include account parameters, encryption parameters, fingerprint parameters, and dynamic cookies. After reverse analysis of these special parameters, they are collected through parameter restoration or RPC remote calls.
[0079] This module performs reverse engineering on the parameters based on the captured packets. If there are no special parameters, this step can be skipped. Special parameters include: account parameters (cookie or token); encrypted parameters, commonly a hash signature verification value, or encryption of the returned data, usually symmetric encryption followed by encoding; dynamic cookies, commonly found on web pages, such as the Ruishu solution, which is frequently seen on some government websites; fingerprint parameters, generally generated based on the browser or mobile environment during registration... This reverse engineering step is also performed within the platform environment determined in the previous step. For more complex parameters, RPC remote calls or automated solutions can be considered; automated solutions will not be discussed here.
[0080] The customized program writing module is used to perform the following: write a customized data acquisition program based on the parameter parsing results, and the interface to be collected obtains data from the target evaluation by executing the data acquisition program.
[0081] As a specific implementation of the customized programming module, this module is used to perform the following operations:
[0082] (1) For general classes, the ssl_context environment is rewritten to bypass the ja3 fingerprint detection of the underlying Python request library and to encapsulate proxy IP switching and UA switching.
[0083] (2) For the RPC solution, add an RPC remote interaction interface and add account pool reading and switching when collecting account parameters.
[0084] This module uses a Python 3.7+ environment locally. It rewrites the ssl_context environment to bypass some Ja3 fingerprint detections on the underlying Python request library and encapsulates this into a new request library. It also adds a proxy interface to bypass some IP proxy detections and an UA switching interface to bypass some UA detections. After encapsulation, various platform interfaces that need to collect data can inherit from this library and be improved according to the specific data collection needs. For example, if using remote RPC calls, an RPC remote interaction interface needs to be added; if account collection is required, an account pool reading and switching function can be added.
[0085] The data acquisition and processing module is used to perform the following: execute the data acquisition program, collect data specified in the predefined acquisition entry table, perform anti-crawling processing during the acquisition process, and upload the acquired data to the data server.
[0086] As a specific implementation of the data acquisition and processing module, this module is used to perform the following operations:
[0087] (1) It involves a data collection entry table, which is used to determine which data to collect and the scope of collection.
[0088] (2) Traverse the entry table and execute the data acquisition program to collect the required data;
[0089] (3) Add anti-scraping logic to specific locations based on the anti-scraping situation.
[0090] In this embodiment, before implementing this module, a data collection entry table needs to be designed based on requirements. The purpose of this table is to determine which data to collect and the scope of collection. For example, if a website collects basic information about people, and I want to know their height, weight, and date of birth, then I need to provide the program with a data table that uniquely identifies each person. This could be a name or a combination of ID number and name. The program then uses this unique identifier to find each person on the website and collects the necessary information. The "data collection entry table" serves this purpose; it's used by the data collection program. The program iterates through all the unique identifiers in the "data collection entry table," retrieves each piece of data, and uploads it to the data server. Because the interfaces for switching proxies, request headers, and accounts have already been designed, during the data collection process, to address anti-scraping issues, only these logics need to be added in specific locations.
[0091] Finally, the server extracts the required fields from the uploaded data and stores them in the database according to certain rules.
[0092] The system in this embodiment can perform data acquisition by executing the method disclosed in Embodiment 1.
[0093] The general data acquisition method and system provided by the present invention have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.
Claims
1. A general data acquisition method, characterized in that, Includes the following steps: Packet capture environment configuration: Determine the platform and platform type of the interface to be captured based on the collection requirements, and configure the packet capture environment and packet capture scheme. The platform type includes web, mobile application and mini-program. Parameter reverse parsing: For the captured data packets, the special interface parameters in the data packets that conform to the predetermined rules are analyzed to determine the generation method of the special parameters and obtain the parameter parsing results; Customized program writing: Based on the parameter parsing results, a customized data acquisition program is written, and the interface to be collected obtains data from the target evaluation by executing the data acquisition program; Data Acquisition and Processing: Execute the data acquisition program, collect data specified in the predefined acquisition entry table, perform anti-crawling processing during the acquisition process, and upload the acquired data to the data server.
2. The general data acquisition method according to claim 1, characterized in that, The packet capture environment configuration includes the following steps: For web-based platforms, compile the open-source Chromium to implement random fingerprints and bypass certain anti-debugging measures at the underlying level. For the WeChat Mini Program platform, wechatwin.dll is patched according to the WeChat version to enable DevTools to open in the Mini Program, which facilitates packet capture and debugging. For mobile platforms, modify the mobile kernel source code to customize the ROM, set up the packet capture environment and hook scripts, install the packet capture software certificate in the system directory, and bypass some common SSL pinning detection, VPN detection and proxy detection through Xposed or Frida scripts.
3. The general data acquisition method according to claim 1, characterized in that, Packet capture environment configuration, special parameters include account parameters, encryption parameters, fingerprint parameters and dynamic cookies. After reverse analysis of the special parameters, data is collected through parameter restoration or RPC remote call.
4. The general data acquisition method according to claim 1, characterized in that, Packet capture environment configuration and customized program writing include the following operations: For general classes, the ssl_context environment is rewritten to bypass the ja3 fingerprint detection of the underlying Python request library, and proxy IP switching and UA switching are encapsulated. For the RPC solution, add an RPC remote interaction interface and add account pool reading and switching when collecting account parameters.
5. The general data acquisition method according to claim 1, characterized in that, Data acquisition and processing includes the following operations: This involves a data collection entry table, which is used to determine which data to collect and the scope of the collection. Traverse the entry table and execute the data acquisition program to collect the required data; Add anti-scraping logic to specific locations based on anti-scraping measures.
6. A universal data acquisition system, characterized in that, It includes a packet capture environment configuration module, a parameter reverse parsing module, a customized program writing module, and a data acquisition and processing module; The packet capture environment configuration module is used to perform the following: determine the platform and platform type of the interface to be collected according to the collection requirements, and configure the packet capture environment and packet capture scheme. The platform type includes web, mobile application and mini-program. The parameter reverse parsing module is used to perform the following: For the captured data packets, it analyzes the special interface parameters in the data packets that conform to the predetermined rules to determine the generation method of the special parameters and obtain the parameter parsing results; The customized program writing module is used to perform the following: write a customized data acquisition program based on the parameter parsing results, and the interface to be collected obtains data from the target evaluation by executing the data acquisition program; The data acquisition and processing module is used to perform the following: execute the data acquisition program, collect data specified in the predefined acquisition entry table, perform anti-crawling processing during the acquisition process, and upload the acquired data to the data server.
7. The general-purpose data acquisition system according to claim 6, characterized in that, The packet capture environment configuration module is used to perform the following operations: For web-based platforms, compile the open-source Chromium to implement random fingerprints and bypass certain anti-debugging measures at the underlying level. For the WeChat Mini Program platform, wechatwin.dll is patched according to the WeChat version to enable DevTools to open in the Mini Program, which facilitates packet capture and debugging. For mobile platforms, modify the mobile kernel source code to customize the ROM, set up the packet capture environment and hook scripts, install the packet capture software certificate in the system directory, and bypass some common SSL pinning detection, VPN detection and proxy detection through Xposed or Frida scripts.
8. The general-purpose data acquisition system according to claim 6, characterized in that, Special parameters include account parameters, encryption parameters, fingerprint parameters, and dynamic cookies. After reverse analysis of these special parameters, they are collected through parameter restoration or RPC remote calls.
9. The general-purpose data acquisition system according to claim 6, characterized in that, The customized programming module is used to perform the following operations: For general classes, the ssl_context environment is rewritten to bypass the ja3 fingerprint detection of the underlying Python request library, and proxy IP switching and UA switching are encapsulated. For the RPC solution, add an RPC remote interaction interface and add account pool reading and switching when collecting account parameters.
10. The general-purpose data acquisition system according to claim 6, characterized in that, The data acquisition and processing module is used to perform the following operations: This involves a data collection entry table, which is used to determine which data to collect and the scope of the collection. Traverse the entry table and execute the data acquisition program to collect the required data; Add anti-scraping logic to specific locations based on anti-scraping measures.