Platform data automatic acquisition method

Abstract

The application provides a platform data automatic collection method. Through response to collection task triggering, first, valid historical session credentials are queried and reused, and only when invalid, a browser control mode is started to perform automatic login to obtain new credentials, and then switched to a lightweight background request mode for data collection and warehousing. The core of the method is to combine the session credential cache reuse mechanism with the browser control and background request dual-mode cooperation. Credential reuse greatly reduces the high-cost repeated login operation; the browser mode is dedicated to cracking complex interactions such as verification codes; the background request mode focuses on efficient collection. With automatic verification code processing, structured data cleaning and task state management, the efficiency, stability, resource consumption and automation of multi-account platform data collection are comprehensively improved.

Description

Technical Field

[0001] This application relates to the field of data application, and in particular to a method for automated data collection on a platform. Background Technology

[0002] With the development of enterprise informatization, companies with numerous branches or stores often need to regularly obtain business data such as cash flow from each store from third-party service platforms for financial reconciliation and operational analysis. However, such platforms often do not provide standardized batch data acquisition interfaces for ordinary enterprise users.

[0003] Currently, there are three main feasible data collection methods: First, relying on technical personnel to manually log in to each store and copy and paste data, which is inefficient and prone to errors; second, simulating manual operation based on traditional browser automation tools, but such tools are difficult to effectively handle the complex CAPTCHAs deployed on the platform, and maintaining a browser instance for each store will lead to huge resource consumption, poor stability and maintainability; third, connecting through the platform's official API interface, but there are problems such as long review cycle, restricted permissions, complex management of multiple accounts and additional costs.

[0004] Therefore, it is necessary to design an automated platform data collection method that can automatically, efficiently, and stably complete multi-account platform data collection, especially one that can effectively handle CAPTCHA verification processes and reduce resource consumption, so as to improve the efficiency and effectiveness of data collection. Summary of the Invention

[0005] The purpose of this application is to address at least one of the aforementioned technical deficiencies, particularly the technical deficiency of insufficient efficiency and effectiveness in data acquisition in the prior art.

[0006] Firstly, this application provides a method for automated data collection on a platform, the method comprising:

[0007] In response to the triggering information of the data collection task, at least one target account to be collected is identified;

[0008] For each target account, query the pre-established session credential cache record and determine the validity of the session credential;

[0009] If it is determined that the historical session credential in the session credential cache record is valid, then the historical session credential is reused as the target session credential.

[0010] If it is determined that the historical session credentials in the session credential cache record are invalid, then the browser control mode is activated, and an automated login process for the target account is executed through the browser instance to obtain a new valid session credential. The new valid session credential is then used as the target session credential and updated in the session credential cache record.

[0011] Based on the target session credentials, switch to background request mode and send a request to at least one target data interface to collect the data corresponding to the target account and write it into the target database, until all target accounts have completed data collection.

[0012] As an optional implementation, the step of initiating browser control mode to execute an automated login process for the target account to obtain new valid session credentials includes:

[0013] Control the browser instance to navigate to the login page of the target platform;

[0014] The authentication information of the target account is automatically filled in on the login page;

[0015] When a verification code verification process is detected on the login page, the target verification code processing service is invoked to complete the verification code verification process.

[0016] After completing the verification code verification process, the login form submission action is triggered;

[0017] Extract the valid session credentials generated from this login session from the browser instance after successful login.

[0018] As an optional implementation, after extracting the valid session credentials generated by the current login session from the browser instance after successful login, the method further includes:

[0019] Close the browser instance;

[0020] The background request pattern is executed via a lightweight session;

[0021] The lightweight session is used to execute the data interaction process corresponding to the background request mode without loading and rendering the page resources of the target platform.

[0022] As an optional implementation, when a verification code verification process is detected on the login page, invoking the target verification code processing service to complete the verification code verification process includes:

[0023] Based on the document object model structure of the login page, identify the type of the CAPTCHA control in the login page, and extract the unique identifier parameter corresponding to the CAPTCHA control from the source code of the login page;

[0024] Send the login page access address and the unique identifier parameter to the application programming interface of the target verification code processing service;

[0025] Receive the verification token returned by the target verification code processing service;

[0026] The verification token is filled back into the corresponding verification response field on the login page, and the verification completion callback function preset by the verification code control is triggered to complete the verification code verification process.

[0027] As an optional implementation, the step of switching to background request mode based on the target session credentials and initiating a request to at least one target data interface to collect data corresponding to the target account and write it to the target database includes:

[0028] Based on the target session credentials, switch to background request mode, initiate a request to at least one target data interface, and identify the characteristic information of the target data interface;

[0029] The target data interface includes a data interface corresponding to the transaction activity module or the fund change module;

[0030] In background request mode, construct an HTTP request that conforms to the aforementioned feature information;

[0031] The HTTP request header field carries authentication information generated based on the target session credentials;

[0032] Send the HTTP request to the target data interface and receive the response data returned by it;

[0033] The response data is parsed to obtain the original data set;

[0034] Perform a cleaning operation on each data record in the original dataset;

[0035] The cleaning operation includes at least one of the following: unit conversion of monetary values, format standardization of time information, and filtering of invalid data records according to preset rules.

[0036] After performing the cleaning operation, each of the data records is written in batches to a preset data table in the target database for persistent storage;

[0037] The preset data tables include one or more of the following: a fund details table, a session credential cache record table, a task execution status table, and a system log table.

[0038] As an optional implementation, the method further includes:

[0039] Maintain a corresponding state machine for the collection task of each target account;

[0040] Update the state representation of the state machine according to the actual execution stage of the data acquisition task;

[0041] The states include a first state, a second state, a third state, a fourth state, a fifth state, and a sixth state.

[0042] The first state indicates that the data collection task is about to start; the second state indicates that the automated login process is being executed; the third state indicates that the data collection task is being executed; the fourth state indicates that the data collection task has been completed; the fifth state indicates that any step of the data collection task has encountered an error; and the sixth state indicates that a retry action will be performed after any step of the data collection task encounters an error.

[0043] Furthermore, the method further includes:

[0044] The status representation and progress information of the collection task corresponding to each target account are persistently stored;

[0045] When the acquisition task is detected to be abnormally interrupted and restarted, the status indication and progress information are read, and the corresponding task process is resumed according to the status indication and progress information.

[0046] Furthermore, before determining at least one target account to be collected in response to the triggering information of the collection task, the method further includes:

[0047] Configure a distributed task scheduler;

[0048] The distributed task scheduler generates trigger information for the collection task at a preset time point within a preset period.

[0049] When the triggering information is received, the status representation corresponding to the collection task of each target account is reset to the first status.

[0050] Secondly, this application provides a platform data automated acquisition device, the device comprising:

[0051] The processing module is used to determine at least one target account to be collected in response to the triggering information of the collection task;

[0052] The processing module is also used to query the pre-established session credential cache records for each target account and determine the validity of the session credential.

[0053] The processing module is further configured to reuse the historical session credential as the target session credential if it is determined that the historical session credential in the session credential cache record is valid.

[0054] The processing module is further configured to, if it is determined that the historical session credentials in the session credential cache record are invalid, start the browser control mode, execute the automated login process for the target account through the browser instance to obtain a new valid session credential, use the new valid session credential as the target session credential, and update the session credential cache record.

[0055] The processing module is further configured to switch to background request mode based on the target session credentials, and send a request to at least one target data interface to collect data corresponding to the target account and write it into the target database, until all target accounts have completed data collection.

[0056] As an optional implementation, the specific method by which the processing module initiates a browser control mode to execute an automated login process for the target account to obtain new valid session credentials includes:

[0057] Control the browser instance to navigate to the login page of the target platform;

[0058] The authentication information of the target account is automatically filled in on the login page;

[0059] When a verification code verification process is detected on the login page, the target verification code processing service is invoked to complete the verification code verification process.

[0060] After completing the verification code verification process, the login form submission action is triggered;

[0061] Extract the valid session credentials generated from this login session from the browser instance after successful login.

[0062] Thirdly, this application provides a computer device including one or more processors and a memory storing computer-readable instructions that, when executed by the one or more processors, perform the steps of the method described in the first aspect.

[0063] Fourthly, this application provides a storage medium storing computer-readable instructions that, when executed by one or more processors, cause the one or more processors to perform the steps of the method described in the first aspect.

[0064] As can be seen from the above technical solutions, the embodiments of this application have the following advantages:

[0065] Based on any of the above embodiments, this application, triggered by a data collection task, first queries and reuses valid historical session credentials. Only when invalid credentials are found is the browser-controlled mode activated to perform automated login to obtain new credentials. Subsequently, it switches to a lightweight background request mode for data collection and storage. The core of this method lies in combining a session credential caching and reuse mechanism with the collaborative use of browser control and background request modes. Credential reuse greatly reduces costly repetitive login operations; the browser mode is dedicated to overcoming complex interactions such as CAPTCHAs; and the background request mode focuses on efficient data collection. Coupled with automated CAPTCHA processing, structured data cleaning, and task status management, this approach comprehensively improves the efficiency, stability, resource consumption, and automation of data collection from multi-account platforms. Attached Figure Description

[0066] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0067] Figure 1 A flowchart illustrating an automated data acquisition method for a platform provided in one embodiment of this application;

[0068] Figure 2 A schematic diagram of the process architecture based on a real-world application scenario for the platform data automation acquisition method provided in one embodiment of this application;

[0069] Figure 3 A flowchart illustrating the process timing of an automated data acquisition method for a platform provided in one embodiment of this application, based on a real-world application scenario.

[0070] Figure 4 This is an internal structural diagram of a computer device provided in an embodiment of this application. Detailed Implementation

[0071] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0072] With the rapid development of cross-border e-commerce, more and more companies are opening stores on overseas third-party payment platforms to expand their businesses. For example, Clover, as a leading global POS payment solution provider, offers merchants a comprehensive service platform integrating payment processing, transaction management, and data analysis, and has a wide market share in the North American and global food and beverage retail industries.

[0073] For businesses with hundreds of overseas chain stores, daily cash flow data from each store is required from the Clover platform for financial reconciliation, accounting, and operational analysis. While Clover provides a merchant backend for store managers to query transaction data, it does not offer a standardized API interface for bulk data retrieval to ordinary enterprise users. For example, a chain restaurant operating over 300 stores on the Clover platform requires its technical staff to log into each store's account daily, manually copy the day's bank statement data, and then input it into the company's internal financial system after processing.

[0074] The aforementioned manual operation mode can be maintained when the number of stores is small, but when the number of stores expands to hundreds, its inefficiency and error-proneness become increasingly prominent, seriously restricting the timeliness and accuracy of financial accounting.

[0075] Specifically, for data acquisition tasks, there are three feasible technical approaches:

[0076] Fully manual operation solution:

[0077] Technicians accessed each store's Clover login page via a browser, manually entered their username and password, and then logged in to the financial statements page. They copied billing data using mouse selection or keyboard shortcuts, pasted it into an Excel spreadsheet, processed it, and finally imported it into the company's financial system. This solution is the standard practice for the vast majority of small and medium-sized enterprises.

[0078] The main limitation of this type of method is:

[0079] Labor costs are high. For a scale of 300 stores, dedicated personnel need to spend four to six hours a day to complete the entire process of logging in, copying, pasting, and organizing for each store, which seriously occupies the effective working time of technical personnel.

[0080] Data is prone to errors. During manual copying and pasting, problems such as data omissions, misaligned rows and columns, formatting errors, and misreading of decimal points are likely to occur. The error rate is usually between three and five percent, and additional time needs to be invested in checking and correcting them afterward.

[0081] Poor real-time performance. Due to time and efficiency constraints, manual operations often result in financial data not being processed and entered into the company's financial system until T+1 day or even later, failing to meet the timeliness requirements for same-day reconciliation.

[0082] It cannot be scaled up. As the number of stores increases further, the demand for manpower grows linearly. Three hundred stores is already the limit for manual operation, and continuing to add stores will make it impossible to complete the task.

[0083] Automation solutions based on traditional Selenium / RPA:

[0084] Some technologically advanced companies use traditional RPA (Robotic Process Automation) tools, such as Octoparse and the open-source Selenium framework, to simulate manual processes and perform batch logins and data scraping by recording or writing automated scripts. This solution completes steps such as form filling, login clicks, and data extraction by locating and manipulating browser controls.

[0085] The main limitation of this type of method is:

[0086] The CAPTCHA processing capability is weak. The Clover platform uses Google reCAPTCHA in its login process. Traditional automated scripts cannot recognize or pass this type of verification challenge, resulting in a high login failure rate and making it difficult for automated processes to run stably.

[0087] The script has poor stability. When page elements (such as the ID of an input box, the XPath path of a button, etc.) change, the pre-recorded positioning rules become invalid, requiring frequent maintenance and updates to the automated script, resulting in high operation and maintenance costs.

[0088] The resource consumption is enormous. Each automated task requires maintaining a complete browser instance, including overhead for page rendering, JavaScript execution, and image loading. Simultaneously managing logins and data collection from three hundred stores places extremely high demands on server memory and CPU, making the hardware costs prohibitive.

[0089] There is no effective session management mechanism. Each data collection requires executing the entire login process from scratch, making it impossible to reuse previously acquired login states. This results in 300 stores having to perform 300 complete browser login operations daily, leading to inefficiency.

[0090] Integration solution based on official API:

[0091] Clover provides an official RESTful API for developers to use. Businesses can obtain access tokens through OAuth 2.0 authentication and then call official interfaces to retrieve transaction and payment data. However, this solution requires businesses to submit their applications for review through the official Clover app store, and each store must complete its own OAuth authorization process.

[0092] The limitations of this type of solution are:

[0093] The implementation threshold is high. Enterprises need to go through the review process of the official Clover app store and submit materials such as application descriptions, privacy policies, and security audit reports. The review period can take several weeks to several months, making it impossible to launch and use the app quickly.

[0094] Access is restricted. The official API offers limited data access to ordinary merchants. Some fund details fields (such as original transaction records and channel fees) are not available to non-contracted merchants, making it impossible to obtain complete data required for reconciliation.

[0095] Managing multiple stores is complex. Each store needs to complete the OAuth authorization process separately and generate an independent access token. Three hundred stores means managing three hundred sets of authorization credentials, and these tokens have expiration dates and need to be refreshed regularly, resulting in high management complexity.

[0096] There are business barriers. Some high-value data interfaces require additional payment to activate, and are billed based on the number of calls, which significantly increases costs for scenarios with high-frequency daily data collection.

[0097] In summary, the technical concept of this application is to provide a method for automated platform data collection. First, responding to the trigger information of the collection task and identifying the target account, the system prioritizes querying the session credential cache record to determine its validity for each account. If the historical credential is valid, it is directly reused. This avoids executing a resource-intensive full browser login process for each account for each collection, which is key to improving efficiency. If the credential is invalid, the system initiates browser control mode, navigating to the login page, automatically filling in information, and specifically integrating calls to an external CAPTCHA processing service to complete the CAPTCHA verification process, ultimately achieving automated login and obtaining new credentials. After successful login, the system closes the browser instance and switches to background request mode using a lightweight session. In this mode, the system carries valid session credentials, initiates a structured request to the target data interface, and parses, cleans, and batch-stores the returned data.

[0098] In this application, firstly, the caching and intelligent reuse strategy of session credentials fundamentally reduces the number of executions of the most time-consuming login step in the automated data collection process, especially for scenarios with a large number of accounts, resulting in a significant efficiency improvement. Secondly, the division of labor and collaboration between browser control mode and backend request mode allows the system to handle unavoidable complex front-end interactions (such as CAPTCHAs) in browser mode, while performing pure data acquisition using lightweight HTTP request mode, thus achieving the best balance between functional completeness and resource efficiency. Furthermore, by calling a professional CAPTCHA processing service and implementing automatic token refill, the problem of complex CAPTCHAs blocking the automated process is effectively solved, improving the login success rate and process stability. In addition, state machine management of the entire lifecycle of the data collection task enables the task to have traceable state and recoverable anomalies, enhancing the system's robustness. Finally, directly obtaining structured data from the interface and performing standardized cleaning ensures the accuracy and consistency of the final data stored in the database. In summary, this method, through a series of collaborative technical features, achieves overall optimization of multi-account platform data collection in terms of automation level, execution efficiency, system resource consumption, and task reliability.

[0099] The methods provided in this application will be described in detail below based on the corresponding implementation methods in some practical application scenarios.

[0100] Figure 1 This is a flowchart illustrating an embodiment of the platform data automated collection method provided in this application, as shown below. Figure 1 As shown, this application provides a platform data automated collection method, the method comprising:

[0101] S101. In response to the triggering information of the data collection task, identify at least one target account to be collected;

[0102] S102. For each target account, query the pre-established session credential cache record and determine the validity of the session credential;

[0103] S103. If it is determined that the historical session credentials in the session credential cache record are valid, then the historical session credentials are reused as the target session credentials.

[0104] S104. If it is determined that the historical session credentials in the session credential cache record are invalid, then the browser control mode is started, and the automated login process for the target account is executed through the browser instance to obtain a new valid session credential. The new valid session credential is used as the target session credential and updated to the session credential cache record.

[0105] As an optional implementation, the step of initiating browser control mode to execute an automated login process for the target account to obtain new valid session credentials includes:

[0106] Control the browser instance to navigate to the login page of the target platform;

[0107] The authentication information of the target account is automatically filled in on the login page;

[0108] When a verification code verification process is detected on the login page, the target verification code processing service is invoked to complete the verification code verification process.

[0109] After completing the verification code verification process, the login form submission action is triggered;

[0110] Extract the valid session credentials generated from this login session from the browser instance after successful login.

[0111] The step of calling the target verification code processing service to complete the verification code verification process when a verification code verification process is detected on the login page includes:

[0112] Based on the document object model structure of the login page, identify the type of the CAPTCHA control in the login page, and extract the unique identifier parameter corresponding to the CAPTCHA control from the source code of the login page;

[0113] Send the login page access address and the unique identifier parameter to the application programming interface of the target verification code processing service;

[0114] Receive the verification token returned by the target verification code processing service;

[0115] The verification token is filled back into the corresponding verification response field on the login page, and the verification completion callback function preset by the verification code control is triggered to complete the verification code verification process.

[0116] Therefore, upon detecting the CAPTCHA verification process, the method analyzes the document object model structure of the login page to identify the CAPTCHA control type and extract its unique identifier parameter. This parameter, along with the page address, is sent to the application interface of the target CAPTCHA processing service. The returned verification token is then received and populated back onto the page to complete the verification. This method automates the identification and processing of specific types of CAPTCHAs, especially complex image CAPTCHAs. By parameterizing calls to professional CAPTCHA services, the verification challenge is transformed into a programmable interface call, thus bypassing the technical bottleneck of traditional automation scripts in CAPTCHA recognition and ensuring the smooth execution of the automated login process.

[0117] This implementation method, when executing the automated login process in browser control mode, specifically controls the browser instance to navigate to the login page and automatically fills in authentication information. When a CAPTCHA verification process is detected, the target CAPTCHA processing service is invoked to complete the verification. Subsequently, the login form is submitted, and valid session credentials are extracted from the successfully logged-in instance. This series of steps achieves a fully automated closed loop in the login process. In particular, by integrating an external CAPTCHA processing service, it effectively solves the obstacles to the automated process posed by complex human-machine verification methods such as the nine-grid CAPTCHA, significantly improving the success rate and reliability of automated login, and providing a stable and effective identity authentication foundation for subsequent data collection.

[0118] S105. Based on the target session credentials, switch to background request mode and send a request to at least one target data interface to collect the data corresponding to the target account and write it into the target database until all target accounts have completed data collection.

[0119] As an optional implementation, after extracting the valid session credentials generated by the current login session from the browser instance after successful login, the method further includes:

[0120] Close the browser instance;

[0121] The background request pattern is executed via a lightweight session;

[0122] The lightweight session is used to execute the data interaction process corresponding to the background request mode without loading and rendering the page resources of the target platform.

[0123] This implementation extracts valid session credentials from the browser instance, closes the browser instance, and executes subsequent background request modes through a lightweight session. Since the lightweight session avoids loading and rendering page resources on the target platform and only performs necessary data interactions, it significantly reduces the system's memory and CPU resource consumption. This design enables timely switching and isolation between resource-intensive and lightweight operations, releasing browser resources immediately after successfully obtaining login status, thereby greatly improving the overall resource utilization efficiency and concurrent processing capabilities of the system.

[0124] As an optional implementation, the step of switching to background request mode based on the target session credentials and initiating a request to at least one target data interface to collect data corresponding to the target account and write it to the target database includes:

[0125] Based on the target session credentials, switch to background request mode, initiate a request to at least one target data interface, and identify the characteristic information of the target data interface;

[0126] The target data interface includes a data interface corresponding to the transaction activity module or the fund change module;

[0127] In background request mode, construct an HTTP request that conforms to the aforementioned feature information;

[0128] The HTTP request header field carries authentication information generated based on the target session credentials;

[0129] Send the HTTP request to the target data interface and receive the response data returned by it;

[0130] The response data is parsed to obtain the original data set;

[0131] Perform a cleaning operation on each data record in the original dataset;

[0132] The cleaning operation includes at least one of the following: unit conversion of monetary values, format standardization of time information, and filtering of invalid data records according to preset rules.

[0133] After performing the cleaning operation, each of the data records is written in batches to a preset data table in the target database for persistent storage;

[0134] The preset data tables include one or more of the following: a fund details table, a session credential cache record table, a task execution status table, and a system log table.

[0135] This implementation, in background request mode, constructs a compliant HTTP request by identifying the characteristic information of the target data interface and includes authentication information generated based on the target session credentials in the request header. This allows the system to initiate a request to the interface and receive response data. Subsequently, the parsed raw data set undergoes cleaning operations, including monetary unit conversion, time format standardization, and invalid data filtering. The cleaned data is then written in batches to a designated table in the database. This process achieves structured and automated processing from interface request to data storage. Precise request construction ensures the accuracy of data acquisition, and systematic data cleaning ensures the standardization and consistency of the data entering the database, thereby improving the data quality and processing efficiency of the entire data acquisition chain.

[0136] This application identifies the target account to be collected by responding to the trigger information of the data collection task and queries the pre-established session credential cache records to determine the validity of the credentials. If the historical session credentials are valid, they are reused directly, avoiding the high-cost browser login process that is repeatedly executed for each account; if invalid, the browser control mode is activated to perform automated login to obtain new credentials and update the cache. Based on the obtained target session credentials, the system switches to background request mode to send a request to the target data interface to collect data and write it to the database. This method, through the intelligent caching and reuse mechanism of session credentials, combined with the dual-mode collaboration of browser control mode for handling complex interactions and background request mode for efficient data collection, minimizes resource-intensive browser instance operations while ensuring that it can handle complex login processes such as platform verification, thereby achieving automation, efficiency, and stability of multi-account platform data collection.

[0137] As an optional implementation, the method further includes:

[0138] Maintain a corresponding state machine for the collection task of each target account;

[0139] Update the state representation of the state machine according to the actual execution stage of the data acquisition task;

[0140] The states include a first state, a second state, a third state, a fourth state, a fifth state, and a sixth state.

[0141] The first state indicates that the data collection task is about to start; the second state indicates that the automated login process is being executed; the third state indicates that the data collection task is being executed; the fourth state indicates that the data collection task has been completed; the fifth state indicates that any step of the data collection task has encountered an error; and the sixth state indicates that a retry action will be performed after any step of the data collection task encounters an error.

[0142] Furthermore, the method further includes:

[0143] The status representation and progress information of the collection task corresponding to each target account are persistently stored;

[0144] When the acquisition task is detected to be abnormally interrupted and restarted, the status indication and progress information are read, and the corresponding task process is resumed according to the status indication and progress information.

[0145] Furthermore, before determining at least one target account to be collected in response to the triggering information of the collection task, the method further includes:

[0146] Configure a distributed task scheduler;

[0147] The distributed task scheduler generates trigger information for the collection task at a preset time point within a preset period.

[0148] When the triggering information is received, the status representation corresponding to the collection task of each target account is reset to the first status.

[0149] This implementation maintains a corresponding state machine for each target account's data collection task and updates the state representation according to the actual execution stage of the task, while persistently storing these states and progress information. When a task is abnormally interrupted and restarted, the system can read the stored state and progress information to resume execution. This mechanism provides data collection tasks with refined process management and monitoring capabilities, enabling the task execution process to have traceable states, detectable anomalies, and resume interrupted tasks. This greatly enhances the system's robustness and self-recovery capabilities in the face of abnormal situations such as network fluctuations and platform service instability, ensuring the completion rate and reliability of large-scale batch data collection tasks.

[0150] Figure 2 A schematic diagram of the process architecture based on a real-world application scenario, corresponding to a platform data automation acquisition method provided in one embodiment of this application; and, Figure 3 This is a flowchart illustrating the process timing of an automated data acquisition method for a platform provided in one embodiment of this application, based on a real-world application scenario. The following is a summary of the process. Figure 2 and Figure 3 The specific implementation of each embodiment of this application will be described in detail based on actual application scenarios.

[0151] It should be noted that when specific commercial platforms or frameworks are mentioned below, the relevant implementation methods must be executed under legal authorization.

[0152] Scheduled task and task status management module:

[0153] This application employs a distributed scheduled task framework (such as APScheduler), which can be configured to automatically trigger all tasks at 17:00 every calendar day. The choice of 17:00 as the trigger time is based on the fact that transaction settlements on the Clover platform are typically completed by midnight Beijing time the following day; triggering at 17:00 ensures that the day's data has been completely stored in the database. After the task is triggered, the system first performs a reset operation on the task status of all stores, updating the processing status field of all stores to 0, representing "pending processing." This mechanism ensures the idempotency of daily tasks—even if a task fails to execute on a certain day, re-execution after resetting the next day will not generate duplicate data.

[0154] The store task state machine is designed as follows:

[0155] Pending: Initial state, indicating that the store has not yet started processing the request for the day.

[0156] Logging in: The system is performing a login operation for this store.

[0157] Data collection in progress: Login successful, retrieving financial data.

[0158] Completed: Data collection successful and data has been entered into the database.

[0159] Error / Login Failure / Data Collection Failure: Status of each step when an error occurs.

[0160] Retry: After failure, the item is added to the retry queue. Once the maximum number of retries is reached, it is marked as an exception.

[0161] This state machine supports breakpoint resumption capability: when a task is interrupted due to network fluctuations or other reasons, the system can resume execution from the point of interruption based on the current state, without having to start from the beginning.

[0162] DrissionPage automated login module:

[0163] This application selects DrissionPage as the core framework for browser automation. Compared to the traditional Selenium, this framework has the following technical advantages: it integrates both browser control mode and pure HTTP request mode capabilities, allowing seamless switching within the same session. Browser control mode is used to handle complex login interactions (especially CAPTCHA challenges), while data collection after obtaining cookies switches to HTTP request mode, significantly reducing resource consumption.

[0164] The cookie-first strategy is one of the core innovations of this application, and its process is as follows:

[0165] When the system needs to collect financial data from a specific store, it first queries the cookie cache table in the database to determine if a valid login cookie exists for that store. The criteria for determining validity include: whether the cookie is within its expiration period and whether it has been marked as invalid by the Clover platform.

[0166] If the cookie is valid, the system skips the full browser login process, reuses the cookie to create an HTTP session, and enters the data collection phase. This design means that for the 300 stores, browser login only needs to be performed once when the cookie is first obtained or expires; all other daily data collection directly reuses the cache, greatly improving execution efficiency.

[0167] If the cookie is missing or expired, the system will initiate browser control mode to execute the complete login process: launch a ChromiumPage browser instance, access the Clover login page, fill in the store account and password, and wait for the page to load before proceeding to the verification code processing stage.

[0168] After successful login, the system extracts cookie information from the browser instance, persists it to the cookie cache table in the database, and records the cookie's expiration time. The browser instance is then closed, and the obtained cookie is injected into the SessionPage (HTTP session mode) to continue subsequent data collection.

[0169] This "login once in the browser, use HTTP mode continuously" design means that the daily data collection task for 300 stores only requires maintaining a very low HTTP request overhead, without having to run 300 browser instances simultaneously.

[0170] CAPTCHA cracking module:

[0171] The Clover platform deploys Google reCAPTCHAv2 (a CAPTCHA checkbox), which is a major technical obstacle to automated login. The reCAPTCHAv2 verification mechanism requires users to click the "I am not a robot" checkbox and sometimes pops up image selection challenges (such as "select an image containing traffic lights"). Traditional OCR technology and simple image recognition algorithms cannot solve these problems.

[0172] This application uses the third-party captcha platform ez-captcha as the captcha cracking solution, but the implementation method must still be carried out within the scope of compliance and authorization. ez-captcha is an automated captcha solving service based on deep learning technology, supporting multiple captcha types such as reCAPTCHAv2 / v3, FunCaptcha, and hCaptcha. Its working principle is as follows: the user submits the target website's captcha page URL and sitekey parameter to the ez-captcha API. The ez-captcha server simulates human operation, calling its trained neural network model to complete the captcha recognition and challenge, ultimately returning a valid verification token.

[0173] The specific process for cracking CAPTCHAs is as follows:

[0174] The first step is CAPTCHA type detection. The system analyzes the DOM structure of the Clover login page to detect the presence of a specific element identifier for reCAPTCHA (such as a g-recaptcha container), thereby determining that the CAPTCHA type is reCAPTCHAv2.

[0175] The second step is to extract the sitekey parameter. Extract the data-sitekey attribute value of the reCAPTCHA control from the page source code. This parameter is a unique credential identifying the current website's CAPTCHA instance; the CAPTCHA solving platform needs to use this parameter to identify and process the corresponding CAPTCHA challenge.

[0176] The third step is to call the ez-captcha API. Submit the current page URL and the extracted sitekey parameter to the ez-captcha server. The server typically cracks the CAPTCHA within 3-5 seconds and returns a verification token valid for approximately 120 seconds.

[0177] The fourth step is token filling and verification submission. The system fills the obtained verification token into the hidden field `g-recaptcha-response` on the login page and triggers the Google reCAPTCHA callback function. At this point, the page considers the user to have completed the verification challenge, and the system calls the login interface to complete the final submission.

[0178] Funds data collection module:

[0179] After successful login and obtaining a valid cookie, the system switches the browser mode to SessionPage (an HTTP request mode based on the requests library). Compared to maintaining a full browser instance, SessionPage mode only sends lightweight HTTP requests, without loading and rendering resources such as images, CSS, and JavaScript on the page, which can reduce memory consumption and CPU usage by about 90%, while reducing the time per request from seconds to milliseconds.

[0180] The system calls Clover platform's backend interfaces (or internal APIs) to collect data. These interfaces are backend interfaces called by the Clover platform's frontend pages themselves and can be analyzed and captured using browser developer tools. The main interfaces involved include:

[0181] Payment Transaction Interface: Retrieves details of each payment transaction from the store, including transaction ID, amount, currency, transaction time, payment method, and transaction status.

[0182] Order API: Retrieves store order information and verifies it against payment data.

[0183] Refund API: Retrieves refund records, used to deduct from income to calculate net cash balance.

[0184] Funds Summary Interface: Retrieves daily summaries of store funds, including total revenue, total refunds, and net revenue.

[0185] During data collection, the system sends a GET request to the aforementioned interface, including previously acquired cookies and the simulated browser's User-Agent identifier in the HTTP request header, along with query parameters such as the time range and page size. The JSON data returned by the interface is parsed by the system, and then subjected to field mapping and data cleaning, such as converting the monetary unit from "cents" to "yuan," standardizing the time format, and filtering test transactions.

[0186] Data storage module:

[0187] The collected financial data, after being cleaned and formatted, is written to the database in batches. The database mainly consists of the following core tables:

[0188] Store Funds Details: This table stores detailed information for each transaction daily, including fields such as the store's unique identifier, transaction number, transaction amount, currency, transaction type, transaction time, and transaction status. It is the core data source for financial reconciliation.

[0189] Store Cookie Cache Table: Stores cookie information after successful store login, including fields such as cookie string, expiration time, and last used time. The system queries this table first before each data collection to enable cookie reuse.

[0190] Task Execution Status Table: Records the execution status of daily tasks, including fields such as store identifier, task date, current status (pending / login in / collecting / completed / abnormal), number of retries, reason for failure, start time, and end time. This table supports task monitoring, tracing, and resuming interrupted tasks.

[0191] Execution Log: Records key events and anomalies during system operation for troubleshooting and auditing.

[0192] Therefore, the key points of this application are mainly as follows:

[0193] Key Point 1: DrissionPage Dual-Mode Collaborative Architecture

[0194] This application organically combines the browser control mode (ChromiumPage) with the HTTP request mode (SessionPage) to form a technical solution of "browser login once, HTTP mode for continuous use". The key to this architecture is: using the browser mode to bypass complex CAPTCHA protection mechanisms to obtain cookies, and then switching to the lightweight HTTP mode to complete large-scale data collection.

[0195] Key Point Two: Cookie-Prioritized Multi-Store Session Management Strategy

[0196] This application designs a dedicated store cookie cache table, which prioritizes querying and verifying the validity of cached cookies before each data collection. The full browser login process is only triggered when the cookie is invalid or expired. This strategy eliminates the need for repetitive, costly login operations for the daily data collection task of 300 stores.

[0197] Key Point 3: Automated Cracking Solution for Nine-Grid CAPTCHAs

[0198] This application deeply integrates the ez-captcha captcha platform with the DrissionPage browser automation framework, realizing a fully automated closed loop from captcha detection, sitekey extraction, API calls to token refilling, and solving the recognition problem of reCAPTCHAv2 type captchas in automated scenarios.

[0199] Key Point 4: Resuming interrupted downloads and ensuring idempotency based on task state machines:

[0200] This application designs a complete multi-state task management mechanism. When the daily task starts, the status of all stores is reset. During the execution process, detailed status information of each store is recorded, and automatic retry and breakpoint resume are supported.

[0201] Key Point 5: Reverse Calling and Data Collection of Backend APIs:

[0202] This application analyzes the backend interfaces called by the frontend pages of the Clover platform and simulates HTTP requests to directly call these backend interfaces to obtain financial data. Compared with the frontend page crawling method, it is more efficient and the data is more structured.

[0203] Therefore, this application achieves the following multiple technical effects:

[0204] The CAPTCHA processing capability has been significantly enhanced.

[0205] Existing manual solutions rely on human verification of CAPTCHAs, and existing Selenium / RPA solutions are completely unable to bypass the Google reCAPTCHAv2 nine-square grid CAPTCHA, resulting in an extremely high failure rate for automated login. This application achieves a closed-loop process of automatic CAPTCHA detection, sitekey extraction, API calls, and token refilling by deeply integrating the DrissionPage framework with the ez-captcha CAPTCHA solving platform, achieving a success rate of over 95% and fundamentally solving the core obstacle of CAPTCHA-based automated login.

[0206] System resource consumption has been significantly reduced.

[0207] Existing Selenium / RPA solutions require maintaining a complete browser instance for each store. Managing 300 stores simultaneously consumes tens of gigabytes of memory and significant CPU resources, resulting in high hardware costs. This application adopts a dual-mode collaborative architecture of "one-time browser login and continuous use in HTTP mode." The browser instance is only temporarily started when the cookie expires, while daily data collection utilizes a lightweight HTTP request mode. Daily data collection tasks for 300 stores consume less than 2GB of memory, reducing resource consumption by approximately 90% compared to existing solutions.

[0208] Data collection and execution efficiency has been significantly improved.

[0209] Existing manual solutions for processing billing data from 300 stores require dedicated technical personnel for four to six hours, and are limited by available time and energy. Existing Selenium / RPA solutions, due to the overhead of maintaining browser instances, take approximately two to three minutes per store login for data collection, resulting in over ten hours for all 300 stores. This application, with valid cookies, requires only a few HTTP requests per store, keeping the total time under ten seconds. Data collection for all 300 stores can be completed in under an hour, improving efficiency by several to tens of times.

[0210] The system has significant advantages in terms of stability and maintenance costs.

[0211] Existing Selenium / RPA solutions heavily rely on page element location (such as input box IDs, button XPath, etc.). Once the Clover platform's front-end page changes, the scripts become ineffective, requiring frequent maintenance and updates. This application's data collection is based on HTTP requests to the backend interface, completely decoupled from the front-end page structure. Changes to page style and layout do not affect the collection process. Simultaneously, cookie pool management and state machine mechanisms ensure the task's fault tolerance and self-healing capabilities, significantly reducing maintenance costs.

[0212] The accuracy of data collection is fundamentally guaranteed.

[0213] In existing fully manual solutions, technicians copy and paste billing data store by store, which easily leads to problems such as data omissions, misaligned rows and columns, misreading of monetary units, and formatting errors. The error rate is typically between 3% and 5%, requiring additional time for verification and correction. This application automates the process by directly calling an API to obtain structured data. After field mapping, data cleaning, and format standardization, the data is batch-written into the database. The entire process requires no manual intervention, resulting in a near-zero data error rate and fundamentally guaranteeing data quality.

[0214] It has good scalability.

[0215] In existing fully manual solutions, the demand for human resources increases linearly with the number of stores. Three hundred stores are already close to the limit of manual operation; further additions will make the task impossible to complete. This application adopts a horizontally scalable architecture. Adding stores only requires adding store account information to the configuration table, without adding hardware resources or modifying code logic. It has excellent scalability and can easily handle scenarios where the number of stores grows to several thousand.

[0216] Supports resume download and self-healing from errors.

[0217] Existing Selenium / RPA solutions lack a robust task state management mechanism. If execution is interrupted due to network fluctuations, platform rate limiting, or other reasons, the entire task must be re-executed from the beginning, wasting time and resources. This application designs a complete task state machine. Before the start of each daily task, the state of all stores is reset, and the status and progress of each store are recorded in detail during execution. When a task is interrupted, execution can resume from the point of interruption based on the status records, without repeating the data collection from already completed stores. Simultaneously, failed stores automatically enter a retry queue and undergo a limited number of retries according to a preset strategy, possessing self-healing capabilities.

[0218] It operates fully automatically and requires no human intervention.

[0219] Existing fully manual solutions require technicians to dedicate dedicated time each day to operation. While existing alternatives using "Selenium plus manual CAPTCHA processing" achieve partial automation, they still require manual intervention upon encountering a CAPTCHA, making it impossible to operate without human intervention. This application achieves full automation of the entire process, from scheduled triggering, batch login, CAPTCHA cracking, data collection to database storage, enabling unattended daily task completion and fundamentally freeing up manpower.

[0220] The efficiency and effectiveness of data acquisition are improved. For example, the API solution is limited by the data permissions granted to ordinary merchants by the Clover platform, and some detailed fund transaction fields (such as original transaction records, channel fees, terminal identifiers, etc.) cannot be obtained, resulting in incomplete reconciliation data. This application, by reverse engineering the backend interface called by the frontend, can obtain complete data fields that are completely consistent with the platform's backend management page. Under the condition of legal authorization, it can meet all the data needs of enterprises for reconciliation.

[0221] This application also provides a platform data automated acquisition device, the device comprising:

[0222] The processing module is used to determine at least one target account to be collected in response to the triggering information of the collection task;

[0223] The processing module is also used to query the pre-established session credential cache records for each target account and determine the validity of the session credential.

[0224] The processing module is further configured to reuse the historical session credential as the target session credential if it is determined that the historical session credential in the session credential cache record is valid.

[0225] The processing module is further configured to, if it is determined that the historical session credentials in the session credential cache record are invalid, start the browser control mode, execute the automated login process for the target account through the browser instance to obtain a new valid session credential, use the new valid session credential as the target session credential, and update the session credential cache record.

[0226] The processing module is further configured to switch to background request mode based on the target session credentials, and send a request to at least one target data interface to collect data corresponding to the target account and write it into the target database, until all target accounts have completed data collection.

[0227] This implementation determines the target account to be collected by responding to the trigger information of the data collection task and queries the pre-established session credential cache records to determine the validity of the credentials. If the historical session credentials are valid, they are directly reused, avoiding the high-cost browser login process that is repeatedly executed for each account; if invalid, the browser control mode is activated to perform automated login to obtain new credentials and update the cache. Based on the obtained target session credentials, the system switches to background request mode to send a request to the target data interface to collect data and write it to the database. This method, through the intelligent caching and reuse mechanism of session credentials, combined with the dual-mode collaboration of browser control mode for handling complex interactions and background request mode for efficient data collection, minimizes resource-intensive browser instance operations while ensuring that it can handle complex login processes such as platform verification, thereby achieving automation, efficiency, and stability of multi-account platform data collection.

[0228] As an optional implementation, the specific method by which the processing module initiates a browser control mode to execute an automated login process for the target account to obtain new valid session credentials includes:

[0229] Control the browser instance to navigate to the login page of the target platform;

[0230] The authentication information of the target account is automatically filled in on the login page;

[0231] When a verification code verification process is detected on the login page, the target verification code processing service is invoked to complete the verification code verification process.

[0232] After completing the verification code verification process, the login form submission action is triggered;

[0233] Extract the valid session credentials generated from this login session from the browser instance after successful login.

[0234] This implementation method, when executing the automated login process in browser control mode, specifically controls the browser instance to navigate to the login page and automatically fills in authentication information. When a CAPTCHA verification process is detected, the target CAPTCHA processing service is invoked to complete the verification. Subsequently, the login form is submitted, and valid session credentials are extracted from the successfully logged-in instance. This series of steps achieves a fully automated closed loop in the login process. In particular, by integrating an external CAPTCHA processing service, it effectively solves the obstacles to the automated process posed by complex human-machine verification methods such as the nine-grid CAPTCHA, significantly improving the success rate and reliability of automated login, and providing a stable and effective identity authentication foundation for subsequent data collection.

[0235] As an optional implementation, the processing module is further configured to, after extracting the valid session credentials generated by the current login session from the browser instance after successful login, execute:

[0236] Close the browser instance;

[0237] The background request pattern is executed via a lightweight session;

[0238] The lightweight session is used to execute the data interaction process corresponding to the background request mode without loading and rendering the page resources of the target platform.

[0239] This implementation extracts valid session credentials from the browser instance, closes the browser instance, and executes subsequent background request modes through a lightweight session. Since the lightweight session avoids loading and rendering page resources on the target platform and only performs necessary data interactions, it significantly reduces the system's memory and CPU resource consumption. This design enables timely switching and isolation between resource-intensive and lightweight operations, releasing browser resources immediately after successfully obtaining login status, thereby greatly improving the overall resource utilization efficiency and concurrent processing capabilities of the system.

[0240] As an optional implementation, when the processing module detects that a verification code verification process exists on the login page, the specific method by which it calls the target verification code processing service to complete the verification code verification process includes:

[0241] Based on the document object model structure of the login page, identify the type of the CAPTCHA control in the login page, and extract the unique identifier parameter corresponding to the CAPTCHA control from the source code of the login page;

[0242] Send the login page access address and the unique identifier parameter to the application programming interface of the target verification code processing service;

[0243] Receive the verification token returned by the target verification code processing service;

[0244] The verification token is filled back into the corresponding verification response field on the login page, and the verification completion callback function preset by the verification code control is triggered to complete the verification code verification process.

[0245] This implementation method, upon detecting a CAPTCHA verification process, identifies the CAPTCHA control type by analyzing the document object model structure of the login page and extracts its unique identifier parameter. This parameter, along with the page address, is sent to the application programming interface (API) of the target CAPTCHA processing service. The returned verification token is then received and populated back onto the page to complete the verification. This method automates the identification and processing of specific types of CAPTCHAs, especially complex graphic CAPTCHAs. By parameterizing calls to professional CAPTCHA services, the verification challenge is transformed into a programmable API call, thus bypassing the technical bottlenecks of traditional automated scripts in CAPTCHA recognition and ensuring the smooth execution of the automated login process.

[0246] As an optional implementation, the processing module, based on the target session credentials, switches to a background request mode and initiates a request to at least one target data interface to collect data corresponding to the target account and write it to the target database. The specific methods include:

[0247] Based on the target session credentials, switch to background request mode, initiate a request to at least one target data interface, and identify the characteristic information of the target data interface;

[0248] The target data interface includes a data interface corresponding to the transaction activity module or the fund change module;

[0249] In background request mode, construct an HTTP request that conforms to the aforementioned feature information;

[0250] The HTTP request header field carries authentication information generated based on the target session credentials;

[0251] Send the HTTP request to the target data interface and receive the response data returned by it;

[0252] The response data is parsed to obtain the original data set;

[0253] Perform a cleaning operation on each data record in the original dataset;

[0254] The cleaning operation includes at least one of the following: unit conversion of monetary values, format standardization of time information, and filtering of invalid data records according to preset rules.

[0255] After performing the cleaning operation, each of the data records is written in batches to a preset data table in the target database for persistent storage;

[0256] The preset data tables include one or more of the following: a fund details table, a session credential cache record table, a task execution status table, and a system log table.

[0257] This implementation, in background request mode, constructs a compliant HTTP request by identifying the characteristic information of the target data interface and includes authentication information generated based on the target session credentials in the request header. This allows the system to initiate a request to the interface and receive response data. Subsequently, the parsed raw data set undergoes cleaning operations, including monetary unit conversion, time format standardization, and invalid data filtering. The cleaned data is then written in batches to a designated table in the database. This process achieves structured and automated processing from interface request to data storage. Precise request construction ensures the accuracy of data acquisition, and systematic data cleaning ensures the standardization and consistency of the data entering the database, thereby improving the data quality and processing efficiency of the entire data acquisition chain.

[0258] As an optional implementation, the processing module is further configured to:

[0259] Maintain a corresponding state machine for the collection task of each target account;

[0260] Update the state representation of the state machine according to the actual execution stage of the data acquisition task;

[0261] The states include a first state, a second state, a third state, a fourth state, a fifth state, and a sixth state.

[0262] The first state indicates that the data collection task is about to start; the second state indicates that the automated login process is being executed; the third state indicates that the data collection task is being executed; the fourth state indicates that the data collection task has been completed; the fifth state indicates that any step of the data collection task has encountered an error; and the sixth state indicates that a retry action will be performed after any step of the data collection task encounters an error.

[0263] Furthermore, the processing module is also used for:

[0264] The status representation and progress information of the collection task corresponding to each target account are persistently stored;

[0265] When the acquisition task is detected to be abnormally interrupted and restarted, the status indication and progress information are read, and the corresponding task process is resumed according to the status indication and progress information.

[0266] Furthermore, the processing module is also configured to perform the following before determining at least one target account to be collected in response to the triggering information of the collection task:

[0267] Configure a distributed task scheduler;

[0268] The distributed task scheduler generates trigger information for the collection task at a preset time point within a preset period.

[0269] When the triggering information is received, the status representation corresponding to the collection task of each target account is reset to the first status.

[0270] This implementation maintains a corresponding state machine for each target account's data collection task and updates the state representation according to the actual execution stage of the task, while persistently storing these states and progress information. When a task is abnormally interrupted and restarted, the system can read the stored state and progress information to resume execution. This mechanism provides data collection tasks with refined process management and monitoring capabilities, enabling the task execution process to have traceable states, detectable anomalies, and resume interrupted tasks. This greatly enhances the system's robustness and self-recovery capabilities in the face of abnormal situations such as network fluctuations and platform service instability, ensuring the completion rate and reliability of large-scale batch data collection tasks.

[0271] It should be noted that the division of the various modules in the above device is merely a logical functional division. In actual implementation, they can be fully or partially integrated into a single physical entity, or they can be physically separated. Furthermore, these modules can be implemented entirely in software via processing element calls; they can be fully implemented in hardware; or some modules can be implemented by processing element calls to software, while others are implemented in hardware. For example, a processing module can be a separate processing element, or it can be integrated into a chip within the device. Alternatively, it can be stored as program code in the device's memory, and its functions can be called and executed by a processing element. The implementation of other modules is similar. Moreover, these modules can be fully or partially integrated together, or they can be implemented independently. The processing element here can be an integrated circuit with signal processing capabilities. During implementation, each step of the above method or each of the above modules can be completed through integrated logic circuits in the hardware of the processor element or through software instructions.

[0272] Indicatively, such as Figure 4 As shown, Figure 4This is a schematic diagram of the internal structure of a computer device 300 provided in an embodiment of this application. The computer device 300 can be provided as a server. (Refer to...) Figure 4 The computer device 300 includes a processing component 302, which further includes one or more processors, and memory resources represented by memory 301 for storing instructions, such as application programs, that can be executed by the processing component 302. The application programs stored in memory 301 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processing component 302 is configured to execute instructions to perform the methods of any of the embodiments described above.

[0273] The computer device 300 may also include a power supply component 303 configured to perform power management of the computer device 300, a wired or wireless network interface 304 configured to connect the computer device 300 to a network, and an input / output (I / O) interface 305. The computer device 300 may operate on an operating system stored in memory 301, such as Windows Server™, MacOSX™, Unix™, Linux™, FreeBSD™, or similar.

[0274] Those skilled in the art will understand that Figure 4 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0275] This application provides a storage medium storing computer-readable instructions that, when executed by one or more processors, cause the one or more processors to perform the method provided in any embodiment.

[0276] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0277] The various embodiments in this specification are described in a progressive manner. Each embodiment focuses on the differences from other embodiments. The various embodiments can be combined as needed, and the same or similar parts can be referred to each other.

[0278] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for automated data collection on a platform, characterized in that, The method includes: In response to the triggering information of the data collection task, at least one target account to be collected is identified; For each target account, query the pre-established session credential cache record and determine the validity of the session credential; If it is determined that the historical session credential in the session credential cache record is valid, then the historical session credential is reused as the target session credential. If it is determined that the historical session credentials in the session credential cache record are invalid, then the browser control mode is activated, and an automated login process for the target account is executed through the browser instance to obtain a new valid session credential. The new valid session credential is then used as the target session credential and updated in the session credential cache record. Based on the target session credentials, switch to background request mode and send a request to at least one target data interface to collect the data corresponding to the target account and write it into the target database, until all target accounts have completed data collection.

2. The method according to claim 1, characterized in that, The step of initiating browser control mode to execute an automated login process for the target account to obtain new valid session credentials includes: Control the browser instance to navigate to the login page of the target platform; The authentication information of the target account is automatically filled in on the login page; When a verification code verification process is detected on the login page, the target verification code processing service is invoked to complete the verification code verification process. After completing the verification code verification process, the login form submission action is triggered; Extract the valid session credentials generated from this login session from the browser instance after successful login.

3. The method according to claim 2, characterized in that, After extracting the valid session credentials generated by the current login session from the browser instance after successful login, the method further includes: Close the browser instance; The background request pattern is executed via a lightweight session; The lightweight session is used to execute the data interaction process corresponding to the background request mode without loading and rendering the page resources of the target platform.

4. The method according to claim 2, characterized in that, When a verification code verification process is detected on the login page, the target verification code processing service is invoked to complete the verification code verification process, including: Based on the document object model structure of the login page, identify the type of the CAPTCHA control in the login page, and extract the unique identifier parameter corresponding to the CAPTCHA control from the source code of the login page; Send the login page access address and the unique identifier parameter to the application programming interface of the target verification code processing service; Receive the verification token returned by the target verification code processing service; The verification token is filled back into the corresponding verification response field on the login page, and the verification completion callback function preset by the verification code control is triggered to complete the verification code verification process.

5. The method according to claim 1, characterized in that, Based on the target session credentials, the process of switching to background request mode and initiating a request to at least one target data interface to collect data corresponding to the target account and write it to the target database includes: Based on the target session credentials, switch to background request mode, initiate a request to at least one target data interface, and identify the characteristic information of the target data interface; The target data interface includes a data interface corresponding to the transaction activity module or the fund change module; In background request mode, construct an HTTP request that conforms to the aforementioned feature information; The HTTP request header field carries authentication information generated based on the target session credentials; Send the HTTP request to the target data interface and receive the response data returned by it; The response data is parsed to obtain the original data set; Perform a cleaning operation on each data record in the original dataset; The cleaning operation includes at least one of the following: unit conversion of monetary values, format standardization of time information, and filtering of invalid data records according to preset rules. After performing the cleaning operation, each of the data records is written in batches to a preset data table in the target database for persistent storage; The preset data tables include one or more of the following: a fund details table, a session credential cache record table, a task execution status table, and a system log table.

6. The method according to any one of claims 1-5, characterized in that, The method further includes: Maintain a corresponding state machine for the collection task of each target account; Update the state representation of the state machine according to the actual execution stage of the data acquisition task; The states include a first state, a second state, a third state, a fourth state, a fifth state, and a sixth state. The first state indicates that the data collection task is about to start; the second state indicates that the automated login process is being executed; the third state indicates that the data collection task is being executed; the fourth state indicates that the data collection task has been completed; the fifth state indicates that any step of the data collection task has encountered an error; and the sixth state indicates that a retry action will be performed after any step of the data collection task encounters an error. Furthermore, the method further includes: The status representation and progress information of the collection task corresponding to each target account are persistently stored; When the abnormal interruption and restart of the data acquisition task is detected, the status representation and the progress information are read, and the corresponding task process is resumed according to the status representation and the progress information. Furthermore, before determining at least one target account to be collected in response to the triggering information of the collection task, the method further includes: Configure a distributed task scheduler; The distributed task scheduler generates trigger information for the collection task at a preset time point within a preset period. When the triggering information is received, the status representation corresponding to the collection task of each target account is reset to the first status.

7. A platform data automated acquisition device, characterized in that, The device includes: The processing module is used to determine at least one target account to be collected in response to the triggering information of the collection task; The processing module is also used to query the pre-established session credential cache records for each target account and determine the validity of the session credential. The processing module is further configured to reuse the historical session credential as the target session credential if it is determined that the historical session credential in the session credential cache record is valid. The processing module is further configured to, if it is determined that the historical session credentials in the session credential cache record are invalid, start the browser control mode, execute the automated login process for the target account through the browser instance to obtain a new valid session credential, use the new valid session credential as the target session credential, and update the session credential cache record. The processing module is further configured to switch to background request mode based on the target session credentials, and send a request to at least one target data interface to collect data corresponding to the target account and write it into the target database, until all target accounts have completed data collection.

8. The apparatus according to claim 7, characterized in that, The specific methods by which the processing module initiates browser control mode to execute an automated login process for the target account to obtain new valid session credentials include: Control the browser instance to navigate to the login page of the target platform; The authentication information of the target account is automatically filled in on the login page; When a verification code verification process is detected on the login page, the target verification code processing service is invoked to complete the verification code verification process. After completing the verification code verification process, the login form submission action is triggered; Extract the valid session credentials generated from this login session from the browser instance after successful login.

9. A computer device, characterized in that, The method includes one or more processors and a memory storing computer-readable instructions that, when executed by the one or more processors, perform the steps of the method as described in any one of claims 1-6.

10. A storage medium, characterized in that, The storage medium stores computer-readable instructions that, when executed by one or more processors, cause the one or more processors to perform the steps of the method as described in any one of claims 1-6.

Images