Adaptive data capture methods, devices, storage media and apparatus

Abstract

This application discloses an adaptive data crawling method, device, storage medium, and apparatus, relating to the field of Internet technology. The method includes: configuring request parameters based on an interface request initiated by a target user, a preset configuration file, and a preset custom mapping rule table to obtain data configuration information; performing pre-crawling verification on the data configuration information to obtain pre-verification results; and matching a target-level thread pool from a preset-level thread pool based on the pre-verification results and a preset sliding window algorithm, and activating threads in the target-level thread pool to complete data crawling. Compared to traditional data crawling methods that require manual intervention for system updates and maintenance, affecting the accuracy and success rate of data crawling, this application can automatically map request parameters, notify the custom mapping rule table to assemble complex parameters and perform pre-verification of parameters, and filter thread priorities to complete data crawling, thereby improving the accuracy and success rate of data crawling.

Description

Technical Field

[0001] This application relates to the field of Internet technology, and in particular to an adaptive data capture method, device, storage medium and apparatus. Background Technology

[0002] In recent years, with the rapid development of internet finance, consumer finance, as a core component, has also experienced a surge in growth. Against this backdrop, the demand for basic risk assessment data, such as customer credit information, in key business processes like approval decisions within the consumer finance sector is growing at an alarming rate, posing unprecedented challenges to data stability and timeliness. Therefore, it has become crucial for consumer finance companies to leverage automated and intelligent data acquisition solutions to meet the data needs of their business processes and ensure the continuous and stable operation of their businesses.

[0003] In traditional data scraping solutions, data processing is often pre-programmed into a fixed pattern, lacking necessary intelligent and automated workflows. These systems typically lack adaptability and self-correction capabilities. Once the data source changes or new data formats emerge, manual intervention is required for system updates and maintenance. This not only increases long-term operating and maintenance costs but may also lead to delays in data processing. Furthermore, the system may fail to identify and process data in a timely manner, impacting the accuracy and success rate of data scraping. Summary of the Invention

[0004] The main purpose of this application is to provide an adaptive data crawling method, device, storage medium and apparatus, which aims to solve the technical problem that traditional data crawling methods cannot adapt to system updates in a timely manner, require manual intervention for system updates and maintenance, resulting in data processing delays and affecting the accuracy and success rate of data crawling.

[0005] To achieve the above objectives, this application proposes an adaptive data crawling method, which includes:

[0006] Based on the interface request initiated by the target user, the preset configuration file, and the preset custom mapping rule table, the request parameters are configured to obtain data configuration information;

[0007] Perform a pre-data capture verification on the data configuration information to obtain the pre-data capture verification result;

[0008] Based on the aforementioned pre-verification results, a target level thread pool is matched from the preset level thread pool using a preset sliding window algorithm.

[0009] Threads from the target level thread pool are used to complete data fetching.

[0010] Optionally, the step of configuring request parameters based on the interface request initiated by the target user, a preset configuration file, and a preset custom mapping rule table to obtain data configuration information includes:

[0011] Based on the interface request initiated by the target user and the preset configuration file, the request parameters are mapped to obtain the display parameters. The preset configuration file includes at least one parameter, such as data name, data configuration file path, node response code mapping relationship, response parameter mapping relationship, request header node attribute information, and request parameter mapping relationship.

[0012] The display parameters are configured based on a preset custom mapping rule table, and a request message is generated.

[0013] The data configuration information is determined based on the request message.

[0014] Optionally, the step of performing pre-data capture verification on the data configuration information and obtaining the pre-verification result includes:

[0015] Based on the Aviator expression evaluation engine, the custom logical expressions of the break and cando attributes in the request header of the data configuration information are executed to determine whether the data request meets the data capture conditions and obtain the condition judgment result.

[0016] If the condition judgment result is that the data request meets the data capture condition, then perform a pre-capture verification on each request parameter in the data configuration information to obtain the pre-capture verification result.

[0017] Optionally, the step of matching the target level thread pool from the preset level thread pool based on the pre-verification result and the preset sliding window algorithm includes:

[0018] If the pre-verification result is successful, the average request processing time and average processing success rate of each interface in the latest time window are statistically analyzed in real time based on the preset sliding window algorithm.

[0019] The target level thread pool is matched from the preset level thread pool based on the average request processing time and the average processing success rate.

[0020] Optionally, before the step of calculating the average request processing time and average success rate of each interface in the latest time window in real time based on a preset sliding window algorithm if the pre-verification result is successful, the method further includes:

[0021] If the pre-verification result is successful, the successfully verified request parameters are marked based on a preset timed task, and the successfully verified request parameters are recorded in a preset request log table as format sample data.

[0022] If the pre-verification result is a verification failure, the parameter format of the failed request parameter is corrected based on the preset supervised machine learning algorithm and the format sample data, and it is determined whether the corrected request parameter conforms to the parameter format corresponding to the format sample data.

[0023] If the corrected request parameters conform to the parameter format, the pre-verification result is determined to be successful, and the step of performing the step of real-time statistics of the average request processing time and average processing success rate of each interface in the latest time window based on the preset sliding window algorithm is executed.

[0024] If the corrected request parameters do not conform to the parameter format, the pre-verification result is determined to be a verification failure, and the pre-verification process ends.

[0025] Optionally, the preset level thread pool includes a high-level thread pool, a medium-level thread pool, and a low-level thread pool. The step of matching a target level thread pool from the preset level thread pool based on the average request processing time and the average processing success rate includes:

[0026] If the average request processing time and the average processing success rate are within a first preset range, then a higher-level thread pool is matched from the preset-level thread pools as the target-level thread pool.

[0027] If the average request processing time and the average processing success rate are within the second preset range, then the thread pool of the preset level is matched as the target level thread pool.

[0028] If the average request processing time and the average processing success rate are within a third preset range, then a lower-level thread pool is matched from the preset-level thread pool as the target-level thread pool.

[0029] Optionally, the step of enabling threads in the target level thread pool to complete data fetching includes:

[0030] When a request thread is obtained from the target level thread pool, a data capture request is initiated according to the request protocol type corresponding to the request message. The request protocol type includes HTTP protocol, DUBBO protocol and KAFKA protocol.

[0031] Determine whether the data capture request was successfully responded to, and obtain the response result;

[0032] Based on the response result, the priority of the request thread is adjusted, and the data is fetched according to the request thread with the adjusted priority.

[0033] In addition, to achieve the above objectives, this application also proposes an adaptive data capture device, the device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the adaptive data capture method as described above.

[0034] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of the adaptive data fetching method described above.

[0035] Furthermore, to achieve the above objectives, this application also proposes an adaptive data crawling device, which includes:

[0036] The data interface configuration management module is used to configure request parameters based on the interface request initiated by the target user, the preset configuration file, and the preset custom mapping rule table, and obtain data configuration information.

[0037] The data request parameter dynamic assembly module is used to perform pre-data capture verification on the data configuration information and obtain the pre-verification result.

[0038] The data capture and monitoring adjustment module is used to match the target level thread pool from the preset level thread pool based on the pre-verification result and the preset sliding window algorithm.

[0039] The data capture and monitoring adjustment module is also used to enable threads in the target level thread pool to complete data capture.

[0040] One or more technical solutions proposed in this application have at least the following technical effects:

[0041] This application obtains data configuration information by configuring request parameters based on interface requests initiated by the target user, a preset configuration file, and a preset custom mapping rule table. It then performs pre-data scraping validation on the data configuration information to obtain the validation results. Based on the validation results and a preset sliding window algorithm, it matches the target level thread pool from a preset level thread pool and activates threads in the target level thread pool to complete data scraping. Compared to traditional data scraping methods that cannot adapt to system updates in a timely manner and require manual intervention for system updates and maintenance, leading to data processing delays and affecting the accuracy and success rate of data scraping, this application can automatically map request parameters, notify the custom mapping rule table to assemble complex parameters and perform parameter pre-validation, and select thread priorities to complete data scraping, thus improving the accuracy and success rate of data scraping. Attached Figure Description

[0042] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0043] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0044] Figure 1 This is a flowchart illustrating an embodiment of the adaptive data capture method of this application.

[0045] Figure 2 This is a schematic diagram of a data crawling system provided in Embodiment 1 of the adaptive data crawling method of this application;

[0046] Figure 3 This is a representation of the data configuration version information provided in Embodiment 1 of the adaptive data crawling method of this application;

[0047] Figure 4 This is an illustration of the data retrieval request pipeline provided in Embodiment 1 of the adaptive data retrieval method of this application;

[0048] Figure 5 This application provides a schematic diagram of the data version concurrency control method processing flow for Embodiment 1 of the adaptive data fetching method;

[0049] Figure 6 This is a representation of the formula-custom function mapping rule provided in Embodiment 1 of the adaptive data crawling method of this application;

[0050] Figure 7 This is a flowchart illustrating Embodiment 2 of the adaptive data capture method of this application;

[0051] Figure 8 This is a schematic representation of the data request parameter format provided in Embodiment 2 of the adaptive data crawling method of this application;

[0052] Figure 9 This is a flowchart illustrating Embodiment 3 of the adaptive data capture method of this application;

[0053] Figure 10 This is a schematic diagram of the data capture and monitoring adjustment process provided in Embodiment 3 of the adaptive data capture method of this application;

[0054] Figure 11 A schematic diagram of the overall process of the adaptive data crawling method provided in Embodiment 3 of this application;

[0055] Figure 12 This is a schematic diagram of the module structure of the adaptive data capture method apparatus according to an embodiment of this application;

[0056] Figure 13 This is a schematic diagram of the device structure of the hardware operating environment involved in the adaptive data capture method in this application embodiment.

[0057] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0058] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0059] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0060] The main solution of this application embodiment is as follows: Based on the interface request initiated by the target user, the preset configuration file and the preset custom mapping rule table, the request parameters are configured to obtain data configuration information; the data configuration information is subjected to pre-data crawling verification to obtain the pre-verification result; based on the pre-verification result and the preset sliding window algorithm, the target level thread pool is matched from the preset level thread pool; and the thread in the target level thread pool is enabled to complete the data crawling.

[0061] In this embodiment, for ease of description, the following description uses a computing service device as the execution subject.

[0062] Traditional data scraping methods cannot adapt to system updates in a timely manner, requiring manual intervention for system updates and maintenance. This leads to data processing delays and affects the accuracy and success rate of data scraping.

[0063] This application provides a solution that can automatically map request parameters, notify a custom mapping rule table to assemble complex parameters and perform pre-parameter validation, and filter thread priorities to complete data scraping, thereby improving the accuracy and success rate of data scraping.

[0064] As can be seen from the above embodiments, this application configures request parameters based on the interface request initiated by the target user, a preset configuration file, and a preset custom mapping rule table to obtain data configuration information; performs pre-data scraping verification on the data configuration information to obtain pre-verification results; and matches the target level thread pool from the preset level thread pool based on the pre-verification results and a preset sliding window algorithm, and enables threads in the target level thread pool to complete data scraping. Compared with traditional data scraping methods that cannot adapt to system updates in a timely manner and require manual intervention for system updates and maintenance, resulting in data processing delays and affecting the accuracy and success rate of data scraping, this application can automatically map request parameters, notify the custom mapping rule table to assemble complex parameters and perform parameter pre-verification, and filter thread priorities to complete data scraping, thereby improving the accuracy and success rate of data scraping.

[0065] It should be noted that the executing entity in this embodiment can be a computing service device with data processing, network communication, and program execution functions, such as a tablet computer, personal computer, or mobile phone, or an electronic device capable of performing the above functions, including an adaptive data capture system. The following description uses a computer as an example to illustrate this embodiment and the subsequent embodiments.

[0066] Based on this, embodiments of this application provide an adaptive data crawling method, referring to... Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the adaptive data capture method of this application.

[0067] In this embodiment, the adaptive data capture method includes steps S10 to S40:

[0068] Step S10: Configure the request parameters based on the interface request initiated by the target user, the preset configuration file, and the preset custom mapping rule table to obtain data configuration information;

[0069] It should be noted that the adaptive data acquisition system mentioned above can be referenced. Figure 2The data capture system diagram shown primarily comprises four modules: a data interface configuration management module, a data request parameter dynamic assembly module, a data capture and monitoring adjustment module, and a data usage and custom extension module. The data interface configuration management module includes a data configuration unit and a data version control unit. The data request parameter dynamic assembly module includes an explicit parameter assembly unit, a parameter pre-validation unit, and an implicit parameter auto-correction unit. The data capture and monitoring adjustment module includes a data capture unit and a monitoring adjustment unit. After the user configures the corresponding data interface request and responds with relevant configuration information through the data interface configuration management module, the data request parameter dynamic assembly module performs pre-validation based on the data configuration information. Upon successful validation, it automatically maps and assembles the request parameters. Then, the data capture and monitoring adjustment module captures the data, and the data usage and custom extension module sends a callback notification to the interface caller to process the data. Target users can be any users within an enterprise who need to access data, such as marketing staff who need to access customer information and transmit this information to the marketing system using the data capture interface. The interface request can be a request from a user to configure a data interface through a visual interface. The preset configuration file can be a pre-configured XML file containing at least one parameter, including the data name, the path to the data configuration file, node response code mapping relationships, response parameter mapping relationships, request header node attribute information, and request parameter mapping relationships. The preset custom mapping rule table can be a pre-set mapping rule table that obtains specific parameter values ​​using custom functions. For example, setting values ​​for complex parameters using a formula custom function.

[0070] Understandably, by having the target user configure the data interface request through a visual interface, the request parameters are configured according to a pre-set configuration file and a preset custom mapping rule table to determine the data configuration information. For further explanation of the data configuration information in this application, please refer to... Figure 3 The data configuration version information shown is a representation of the data configuration version information. Field names include DATA_ID, DATA_NAME, DATA_FILE_PATH, VERSION, and REFERENCE; data types include VARACHAR2 and INT; lengths include 32, 64, and 256; and remarks include data ID, data name, data configuration file path, data version, and data reference count. After configuration, the system automatically converts the configuration information into an XML file and saves it. Each time a data configuration is added or updated, the system saves the data configuration information to the data configuration version information table. When a new configuration is added, the data version field is set to 1. For each subsequent update, the data version field is incremented by the historical maximum version number. For example, the second update sets the field value to 3, and so on. The data reference number field is initially set to 0.

[0071] In practice, the data version control unit primarily employs a version concurrency control method to systematically manage different versions of data interface configurations, ensuring compatibility and continuity of data capture when data interfaces change. Each time a data caller references a specific data configuration for a data capture request, the system automatically adds a new record of the data configuration version information. The structure of the data configuration version information record table is shown below. Figure 4 The data request pipeline shown below represents the intent. Field names include REQ_PRO_ID, DATA_ID, DATA_NAME, REQ_BODY, RESP_BODY, VALIDATE_LABEL, REQ_STATUS, PRO_USE_TIME, CREATE_TIME, and UPDATE_TIME; types include VARACHAR2, JSON, INT, and TIMESTAMP; lengths include 1, 2, 32, and 64; and remarks include request process ID, data ID, data name, request body information, response body information, and a request parameter validation success tag, where '1' indicates successful validation. '0' - Validation failed; Request processing status: '00' - Incomplete; '01' - Completed; '99' - Abnormal; Request processing time (unit: milliseconds); Creation time; Update time; The REQ_STATUS field is initialized to '00' - Incomplete status. When adding or updating data configuration, a new data version information record is saved, and the data reference count is set to 0. When the data caller references a certain data request to retrieve data, a new data call request log is added. The DATA_ID field records the data ID corresponding to the largest data version number with the same data name in the data configuration version information table, and at the same time, the data reference count of the same data version record in the data configuration version information table is incremented by 1. After the aforementioned data version information and call request pipeline are saved, the subsequent data request parameter dynamic assembly module queries the data information based on DATA_ID, retrieves the data configuration file path from the data version information, obtains the file, and performs parsing and assembly processing. Meanwhile, the data capture and monitoring adjustment module updates the REQ_STATUS field of the data call request pipeline table to '01' - completed when data capture is complete, and decrements the data reference count of the data version information by 1 using an optimistic locking CAS method. When the data reference count reaches 0, the data configuration record for the current data version is deleted. See the schematic diagram of the above data version concurrency control method for reference. Figure 5 As shown. Through this version of concurrency control method, the system can reliably handle the problem that when a data configuration changes, the request information cannot be matched when the request parameter mapping relationship changes in the in-transit business processes that use the historical version of the data configuration for parameter assembly and data retrieval requests. When the historical version of the data configuration is no longer referenced by any in-transit business processes, the system automatically clears the historical version of the data configuration information, thus saving system storage space.

[0072] Furthermore, step S10 also includes: mapping request parameters based on the interface request initiated by the target user and a preset configuration file to obtain display parameters. The preset configuration file includes at least one parameter of data name, data configuration file path, node response code mapping relationship, response parameter mapping relationship, request header node attribute information, and request parameter mapping relationship. The display parameters are configured based on a preset custom mapping rule table, and a request message is generated; data configuration information is determined based on the request message.

[0073] It should be noted that the displayed parameters can refer to the parameters obtained by conventionally mapping the request parameters through a preset configuration file. The data configuration unit provides conventional mapping configuration for request and response parameters, and also provides preset formula custom functions to set values ​​for complex parameters. For example, setting the formula custom function attribute to '{time:yyyyMMdd}' means setting the parameter to the system date in the format "yyyyMMdd".

[0074] Understandably, the custom mapping rule table can be referenced for further explanation. Figure 6The diagram illustrates the mapping rules for custom functions. Custom function types include four categories: timestamp, unique ID, hash, and custom function. The formula expression for timestamp is `formula={time: yyyMMdd}`, indicating that the parameter is set to the system date in the format "yyyMMdd". The formula expression for unique ID is `formula={uuid22}`, indicating that the parameter is set to a UUID and its first 22 digits are truncated as a unique serial number. The formula expression for hash is `formula={'01': 'A', '02': 'B', '03': 'C'}`, indicating which parameters are hashed; for example, a parameter value of '01' will be automatically mapped to 'A'. The custom function type is `fo`. The formula expression is `formula={custom function name(@parameter1, @parameter2)}`, indicating that a complex parameter value is set by calling a custom function. Users can use one or more function types individually to set parameter values. For example, combining timestamp and unique ID types to set the formula attribute to '{time:yyyyMMdd_uuid23}' will automatically generate a 32-bit unique serial number starting with the current date format when the explicit parameter assembly unit assembles this parameter. When request parameters need to be set after complex logic judgment or calculation, users can use the custom function type to set the value after implementing the custom function. If the custom function implementation logic is a complex encryption operation, the explicit parameter assembly unit will execute the custom function to set the parameter to a specific ciphertext when assembling this parameter. A request message is generated based on the specific ciphertext, and data configuration information is determined based on the request message.

[0075] Step S20: Perform pre-data capture verification on the data configuration information to obtain the pre-data capture verification result.

[0076] It should be noted that pre-data scraping validation can be performed by configuring custom logical expressions to set data scraping conditions and determining whether the data interface meets the scraping conditions, thus providing flexible pre-validation for data scraping. Logical expressions can flexibly combine various arithmetic operators, relational operators, and logical operators.

[0077] Understandably, flexible pre-validation of data scraping includes parameter format validation, and the results of pre-validation include validation success and validation failure.

[0078] In the specific implementation, during the pre-data scraping validation, if the value of the paramA request parameter is an odd number less than 100, the system will automatically skip the data scraping step. When the data request meets the validation conditions for data scraping, the parameter pre-validation unit will perform format validation on each request parameter. When all parameters are successfully validated, step S30 is executed.

[0079] Step S30: Based on the pre-verification results and the preset sliding window algorithm, match the target level thread pool from the preset level thread pool.

[0080] It should be noted that the preset sliding window algorithm can be a pre-set algorithm used to calculate the average request processing time and success rate of the current interface data in the latest time window in real time. The preset thread pool levels refer to pre-set thread pools of different levels, including: high-level thread pool, medium-level thread pool, and low-level thread pool.

[0081] In specific implementation, when the current verification result is successful, the average request processing time and average success rate of the current interface data in the latest time window are statistically analyzed in real time based on the preset sliding window algorithm. According to the average request processing time and average success rate, if the interface data response is slowed down or the success rate is reduced, the system can automatically adjust the request priority of the interface data and select the target thread from the high-level thread pool, medium-level thread pool and low-level thread pool to reduce the request rate.

[0082] Step S40: Activate the threads in the target level thread pool to complete data capture.

[0083] It should be noted that data fetching is performed by using threads from the target-level thread pool. For example, data fetching can be performed by using threads from the high-level, medium-level, or low-level thread pool. If the data response becomes slow or the success rate decreases, the system can automatically adjust the request priority of the interface data, thereby selecting a new level thread for data fetching.

[0084] In practical implementation, after enabling threads in the target-level thread pool to complete data fetching, the fetched data can be used by a custom extension module to notify the caller (corresponding to the data fetching process node in the business process) to use the response data via real-time callback methods. Simultaneously, assuming the system needs to synchronously transmit the fetched customer information to the marketing system, the system can asynchronously listen to the Kafka message queue that stores the interface response information, adapt the response information to the data structure specified by the marketing system, and then synchronously push the customer information to the marketing system. Alternatively, if the system also needs customer information data analysis, it can simultaneously pull data from this message queue, break it down, and store it in a database for data analysis.

[0085] This embodiment provides an adaptive data crawling method. Based on the interface request initiated by the target user, a preset configuration file, and a preset custom mapping rule table, the method configures the request parameters to obtain data configuration information. It then performs pre-crawling validation on the data configuration information to obtain the validation results. Based on the validation results and a preset sliding window algorithm, it matches a target-level thread pool from a preset-level thread pool and activates threads in the target-level thread pool to complete data crawling. Compared to traditional data crawling methods that cannot adapt to system updates in a timely manner and require manual intervention for system updates and maintenance, leading to data processing delays and affecting the accuracy and success rate of data crawling, this method can automatically map request parameters, notify the custom mapping rule table to assemble complex parameters and perform parameter pre-validation, and filter thread priorities to complete data crawling, thus improving the accuracy and success rate of data crawling.

[0086] Based on the above Figure 1 The first embodiment shown illustrates a second embodiment of the adaptive data scraping method of this application; see also... Figure 7 , Figure 7 This is a flowchart illustrating the second embodiment of the adaptive data acquisition method of this application. Based on the first embodiment of this application, the same or similar content as the first embodiment described above can be referred to the above description and will not be repeated hereafter.

[0087] In this embodiment, as Figure 7 As shown, step S20 further includes:

[0088] Step S201: Based on the Aviator expression evaluation engine, execute the custom logical expression of the break and cando attributes in the request header of the data configuration information to determine whether the data request meets the data capture conditions and obtain the condition judgment result.

[0089] It should be noted that the Aviator expression evaluation engine refers to expression strings written in Aviator. The system uses the Aviator engine to execute these expressions and dynamically determines the program's execution flow based on the evaluation results. The `cando` attribute in the request header indicates whether it can be done or executed, a conditional expression. It determines whether the current rule should be executed. The `break` attribute indicates whether to interrupt or exit the execution; it determines whether the entire rule chain should be terminated immediately after the current rule is executed, preventing the execution of all subsequent rules. The condition evaluation result includes two outcomes: meeting the data collection conditions and not meeting the data collection conditions.

[0090] The parameter pre-validation unit uses the Aviator expression evaluation engine to execute custom logical expressions in the `break` and `cando` attributes of the data interface configuration request header to determine whether the data scraping conditions are met, performing flexible pre-validation for data scraping. The logical expressions can be flexibly combined with various arithmetic, relational, and logical operators. When the logical expression in the `break` attribute evaluates to `true`, it indicates that the data scraping conditions are not met; the logical expression in the `cando` attribute only meets the data scraping conditions if it evaluates to `true`; if neither the `break` nor `cando` attributes are set, the system defaults to not performing pre-validation for data scraping.

[0091] Step S202: If the condition judgment result is that the data request meets the data capture condition, then perform a data capture pre-verification on each request parameter in the data configuration information to obtain the pre-verification result.

[0092] It should be noted that when the condition judgment result is that the data request meets the data capture conditions, a pre-capture verification is performed on each request parameter in the data configuration information. When the data request meets the above data capture conditions, the parameter pre-verification unit will perform format verification on each request parameter and determine the pre-verification result based on the format verification result. The pre-verification result includes two results: verification success and verification failure.

[0093] In the specific implementation, after all parameters have been successfully verified, the VALIDATE_LABEL field in the data call request log table will be set to '1' - verification successful, so that the parameters marked as successfully verified can be used as verification sample data for format verification later.

[0094] In this embodiment, step S30 further includes:

[0095] Step S301: If the pre-verification result is successful, then based on the preset sliding window algorithm, the average request processing time and average processing success rate of each interface in the latest time window are statistically analyzed in real time.

[0096] It should be noted that if the pre-validation result is successful, the system will, by default, retrieve a request thread from the high-level thread pool to process the data scraping request. The monitoring and adjustment unit can monitor the performance indicators of data scraping in real time. These performance indicators can be the average processing time and average success rate of each interface in the latest time window, calculated in real time using a preset sliding window algorithm. Furthermore, the data scraping strategy can be adaptively adjusted based on the query mode configured in the data configuration settings.

[0097] Step S302: Match the target level thread pool from the preset level thread pool based on the average request processing time and the average processing success rate.

[0098] Understandably, the system will use a sliding window algorithm to calculate the average request processing time and average success rate of the current interface data in the latest time window in real time. When the response of a certain interface data slows down or the success rate drops, the system can automatically adjust the request priority of that interface data to reduce the request rate, thereby reducing the processing pressure on the data provider. When the response processing time or success rate increases in subsequent statistical periods, the request rate will be continuously increased.

[0099] Furthermore, step S302 further includes: if the average request processing time and the average processing success rate are within a first preset range, then a high-level thread pool is matched from the preset-level thread pool as the target-level thread pool; if the average request processing time and the average processing success rate are within a second preset range, then a medium-level thread pool is matched from the preset-level thread pool as the target-level thread pool; if the average request processing time and the average processing success rate are within a third preset range, then a low-level thread pool is matched from the preset-level thread pool as the target-level thread pool.

[0100] It should be noted that the first preset interval can be a pre-set interval used to distinguish the average processing time and average success rate of high-level threads, the second preset interval can be a pre-set interval used to distinguish the average processing time and average success rate of high-level threads, and the third preset interval can be a pre-set interval used to distinguish the average processing time and average success rate of high-level threads.

[0101] In practical implementation, if the data configuration is set to use caching mode, when a data request response from a certain interface is abnormal, it can automatically degrade to using historical cached data; if the data configuration is set to direct lookup mode, the average processing time can be set to AVGPT (Average Process Time), and the average success rate can be set to AVGSR (Average Success Rate). The following is the data scraping request strategy for direct lookup mode:

[0102] (1) (AVGPT<200ms)&&(AVGSR>=95%): Match a high-level thread pool to obtain threads and quickly perform data grabbing.

[0103] (2) (200ms<= AVGPT<500ms)&&(AVGSR>= 90%): Match the medium-level thread pool to obtain threads and perform data grabbing at a medium speed.

[0104] (3) (AVGPT>= 500ms) || (AVGSR<90%): Match the low-level thread pool to obtain threads and perform data grabbing at a low speed.

[0105] Further, before step S301, the method includes: if the pre-verification result is successful, then marking the successfully verified request parameters based on a preset timed task and recording the successfully verified request parameters in a preset request log table as format sample data; if the pre-verification result is unsuccessful, then correcting the parameter format of the unsuccessful request parameters based on a preset supervised machine learning algorithm and the format sample data, and determining whether the corrected request parameters conform to the parameter format corresponding to the format sample data; if the corrected request parameters conform to the parameter format, then determining that the pre-verification result is successful, and executing the step of real-time statistics of the average request processing time and average processing success rate of each interface in the latest time window based on a preset sliding window algorithm; if the corrected request parameters do not conform to the parameter format, then determining that the pre-verification result is unsuccessful, and ending the pre-verification process.

[0106] It should be noted that the formatted sample data is sample data constructed based on the parameter formats corresponding to the request parameters marked as successful by the pre-validation result. The preset supervised machine learning algorithm can be a pre-set algorithm for automatically correcting request parameters. The implicit parameter automatic correction unit can implicitly and automatically correct request parameters based on the parameter pre-validation unit's automatically marked sample data of successful parameter validation, using the supervised machine learning algorithm. If the pre-validation result is successful, the successfully validated request parameters are marked based on a preset timed task and recorded in a preset request log table as formatted sample data. If the pre-validation result is unsuccessful, the parameter formats of the unsuccessful request parameters are corrected based on the preset supervised machine learning algorithm and the formatted sample data. It is then determined whether the corrected request parameters conform to the parameter formats corresponding to the formatted sample data. If the corrected request parameters conform to the parameter formats, the pre-validation result is determined to be successful, and the step of real-time statistics of the average request processing time and average processing success rate of each interface in the latest time window based on a preset sliding window algorithm is executed. If the corrected request parameters do not conform to the parameter formats corresponding to the formatted sample data, the pre-validation result is determined to be unsuccessful, and the pre-validation process ends.

[0107] Understandably, this unit will start a background scheduled task to continuously poll and scan the data call request log table for records where the VALIDATE_LABEL field value is '1' - records that have successfully validated. It will count the number of successfully validated records for the day, using "data name - parameter name" as the dimension, and continuously iterate and update the correct data format corresponding to different parameters for different data names. The parameter data format will then be saved in the attached file. Figure 8The data request parameter format shown is illustrated in the diagram. This includes field names: DATA_NAME, PARAM_NAME, PARAM_FORMAT, with data types including VARCHAR2 and lengths of 32 or 64. Remarks include data name, parameter name, and parameter format. For example, if the date format of request parameter b for data A is incorrectly validated, the system will retrieve the data format of that request parameter from the data request parameter format table and correct it to the correct format. Through this automatic parameter correction mechanism, the system can significantly reduce request anomalies caused by inconsistent parameter formats, thereby improving the success rate of data retrieval.

[0108] Understandably, when a data request meets the above data capture conditions, the parameter pre-validation unit will perform format validation on each request parameter. Once all parameters have been successfully validated, the attached... Figure 4 The data call request log record shown has its VALIDATE_LABEL field set to '1' - verification successful. These successfully verified records will serve as reference sample data for the implicit parameter autocorrection unit during correction. If some parameters are found to be incorrectly formatted during verification, the request parameter information will be sent to the implicit parameter autocorrection unit for attempted correction. If the unit ultimately succeeds in correction, the parameter verification is still considered successful; otherwise, it is considered a parameter verification failure. In this case, the VALIDATE_LABEL field is set to '0' - verification failed, and a parameter verification exception is returned directly to the data caller, ending the pre-verification process.

[0109] This embodiment provides an adaptive data crawling method. Based on the interface request initiated by the target user, a preset configuration file, and a preset custom mapping rule table, request parameters are configured to obtain data configuration information. The method then uses the Aviator expression evaluation engine to execute custom logical expressions of the `break` and `cando` attributes in the request header of the data configuration information to determine whether the data request meets the data crawling conditions, obtaining a condition judgment result. If the condition judgment result indicates that the data request meets the data crawling conditions, a pre-crawling validation is performed on each request parameter in the data configuration information to obtain a pre-validation result. If the pre-validation result is successful, a preset sliding window algorithm is then used... This system provides real-time statistics on the average request processing time and average success rate of each interface within the latest time window. Based on these average processing times and success rates, it matches a target-level thread pool from a preset-level thread pool and activates threads from the target-level thread pool to complete data scraping. Compared to traditional data scraping methods that cannot adapt to system updates in a timely manner and require manual intervention for system updates and maintenance, leading to data processing delays and affecting the accuracy and success rate of data scraping, this application can automatically map request parameters, notify a custom mapping rule table to assemble complex parameters and perform parameter pre-validation, and filter thread priorities to complete data scraping, thereby improving the accuracy and success rate of data scraping.

[0110] Based on the above Figure 1 The first embodiment shown illustrates a third embodiment of the adaptive data scraping method of this application; see also... Figure 9 , Figure 9 This is a flowchart illustrating the third embodiment of the adaptive data acquisition method of this application. Based on the first embodiment of this application, the same or similar content as the first embodiment described above can be referred to the above description in the third embodiment of this application, and will not be repeated hereafter.

[0111] In this embodiment, as Figure 9 As shown, step S40 further includes:

[0112] Step S401: When a request thread is obtained from the target level thread pool, a data capture request is initiated according to the request protocol type corresponding to the request message. The request protocol type includes HTTP protocol, DUBBO protocol and KAFKA protocol.

[0113] It should be noted that the data capture unit comprises two parts: request processing and response processing. For further explanation of the data capture and monitoring / adjustment process, please refer to [link / reference]. Figure 10The diagram illustrates the data capture and monitoring / adjustment process. For request processing, the system divides the system into three different thread pools: high, medium, and low, configured with 500, 200, and 50 threads respectively. When processing data capture requests, the system by default retrieves a request thread from the high-level thread pool for processing. The system supports multiple request protocol types, including HTTP, Dubbo, and Kafka (Asynchronous Kafka), and can flexibly extend to new request protocol types by implementing protocol type interfaces. Furthermore, the monitoring and adjustment unit can monitor data capture performance metrics in real time, such as average response time and success rate, and can adaptively adjust the data capture strategy based on the query mode configured in the data configuration. If the data configuration is set to use caching mode, when a data request response from a certain interface is abnormal, it can automatically degrade to using historical cached data. If the data configuration is set to direct query mode, the system will use a sliding window algorithm to calculate the average request processing time and average processing success rate of the current interface data in the latest time window in real time. When the response of a certain interface data slows down or the success rate drops, the system can automatically adjust the request priority of the interface data to reduce the request rate, thereby reducing the processing pressure on the data provider. When the response processing time or success rate increases in subsequent statistical periods, the request rate will be continuously increased. Let the average request processing time be AVGPT (Average Process Time) and the average success rate be AVGSR (Average Success Rate). The following are the data grabbing request strategies for the direct lookup mode: (1) (AVGPT<200ms)&&(AVGSR>=95%): Match a high-level thread pool to get a thread and quickly perform data grabbing; (2) (200ms<=AVGPT<500ms)&&(AVGSR>=90%): Match a medium-level thread pool to get a thread and perform data grabbing at a medium speed; (3) (AVGPT>=500ms) || (AVGSR<90%): Match a low-level thread pool to get a thread and perform data grabbing at a low speed.

[0114] Step S402: Determine whether the data capture request was successfully responded to, and obtain the response result.

[0115] Understandably, after successful data retrieval (i.e., when the response code is "200"), the system will automatically convert and map the response parameters based on the interface configuration information and response message, and generate a response result. This response result includes the response message and response parameters, and will update the attached... Figure 4 The data call request log table shows the values ​​of the "Request Processing Status" and "Request Processing Time" fields for the corresponding records.

[0116] Step S403: Adjust the priority of the request thread based on the response result, and complete the data retrieval according to the request thread with the adjusted priority.

[0117] It should be understood that since the thread priority in this solution is adjusted in real time, it is necessary to automatically convert and map the response parameters based on the interface configuration information and response message in the response result to update the processing time and success rate, thereby adjusting the thread priority.

[0118] For a detailed explanation of the complete execution flow of this solution, please refer to [link / reference]. Figure 11 The diagram shown illustrates the overall workflow of the solution, including each module and processing stage. This solution is divided into a configuration stage, an assembly and verification stage, a data capture and monitoring stage, and a data usage and expansion stage. For detailed explanation of the overall process, it is based on a reference... Figure 11 The overall process diagram illustrates how users can configure data request and response parameter mapping relationships, request methods, and request validation conditions through a visual interface configuration management module. The explicit parameter assembly unit of the dynamic data request parameter assembly module will automatically and explicitly map and assemble the request parameters based on the interface configuration information in step one. The parameter pre-validation unit executes the custom logical expression of the `break` attribute in the data interface configuration request header based on the Aviator expression evaluation engine to determine if it meets the data capture conditions. When the data request meets the validation conditions explicitly specified in step one, the parameter pre-validation unit will perform format validation on each request parameter. When all parameters are successfully validated, the successfully validated records will serve as reference sample data for the implicit parameter auto-correction unit during correction. The implicit parameter auto-correction unit will then retrieve the data from the attached... Figure 8 The system retrieves the correct data format for the parameter name from the data request parameter format table shown and attempts to correct its parameter value to the correct value. Through this automatic parameter correction mechanism, when the system fetches data from the customer information query interface of the business process, it will obtain the request message and request protocol type from the data request parameter dynamic assembly module and, by default, obtain a request thread from the high-level thread pool for data fetching. This customer information query interface will perform data fetching based on the DUBBO request protocol. After successful data fetching, the system will automatically convert and map the response parameters according to the interface configuration information and response message. After the data fetching and monitoring adjustment module completes the data response parameter conversion and mapping, the data usage and custom extension module will call the callback method in real time to notify the caller (the corresponding business process data fetching process node) to use the response data. The system can asynchronously listen to the Kafka message queue that simultaneously stores the interface response information and push the customer information synchronously to the business system; or, if the system also has a need for customer information data analysis, it can simultaneously pull data from this message queue, break it down, and store it in the database for data analysis. Based on the above, this solution is illustrated as follows:

[0119] Step 1: Users configure data request and response parameter mappings, request methods, and request validation conditions in the data interface configuration management module using a visual interface. This step uses the "Customer Information Query Interface" data configuration as an example to further explain the configuration of various data attributes;

[0120] First, users need to configure the data name (corresponding to the `dataBaseName` attribute), the data configuration file path (corresponding to the `dataBaseClassPath` attribute), the `respCode` node response code mapping (the successful query response code can be explicitly configured, while the others are the failed query response codes), and the response parameter mapping (corresponding to the `parameterOut` node). Second, the `connector` node attribute information in the request header needs to be configured. For example, this interface is configured using the DUBBO protocol, and the request conditions are specified through its `break` attribute. In this interface, configuring it as "customer basic information is not empty, or customer age is less than 18 years old" does not meet the query conditions, meaning data scraping is skipped. Furthermore, the request parameter mapping needs to be configured. In this example, the request serial number parameter uses a combination of timestamp and unique ID type, setting the `formula` attribute to `{time:yyyyMMdd_uuid23}`, which allows the system to randomly generate a unique serial number with a length of 32 characters starting with the current date format. The ID card number parameter is configured to be mapped using the general parameter `idCardNo`. After configuration, the system automatically converts the configuration information into an XML file format for saving. The converted data configuration XML file content format is shown below:

[0121] <?xml version="1.0" encoding="UTF-8">

[0122] <databasexml>

[0123] <databasename> Customer Information Inquiry< / databasename>

[0124] <databaseclasspath> youcash.test.queryCustInfo< / databaseclasspath>

[0125] <connecter type="dubbo">

[0126] <param name="interface"> com.test.interface

[0127] <param name="method"> queryCustInfo

[0128] <param name="param.classtype"> com.test.rpc.req.QueryCustInfoReq

[0129] <param name="timeout"> 60000

[0130] <param name="break"> ${custInfo!=null || age<18}

[0131] < / connecter>

[0132] <respcode>

[0133] <codekey> code< / codekey>

[0134] <successcode> 200< / successcode>

[0135] < / respcode>

[0136] <parameterin>

[0137] <parameter parameterName="requestNo" description="请求流水号" parameterClass="String" formula="{time:yyyyMMdd_uuid23}" / >

[0138] <parameter parameterName="idNo" description="身份证号" parameterClass="String" mapper="{idCardNo}" / >

[0139] <parameter parameterName="systemId" description="系统编号" parameterClass="String" mapper="{systemNo}" / >

[0140] <parameter parameterName="saltEncryption" description="加密盐" parameterClass="String" formula="{getSaltEncryption(@idNo)}" / >

[0141] < / parameterin>

[0142] <parameterout>

[0143] <parameter parameterName="code" description="响应码" parameterClass="string" / >

[0144] <parameter parameterName="msg" description="响应信息" parameterClass="string" / >

[0145] <parameter parametername="data" description="响应数据" parameterclass="object">

[0146] <parameter parametername="gender" description="性别" parameterclass="string">

[0147] <parameter parametexName="phoneNo" description="手机号" parameterClass="string" / >

[0148] <parameter parameterName="urgentCont" description="紧急联系人" parameterClass="String" / >

[0149] < / parameter>

[0150] < / parameter> < / parameterout>

[0151] < / databasexml> ;

[0152] Step Two: The explicit parameter assembly unit of the dynamic parameter assembly module for data requests will explicitly and automatically map and assemble the request parameters according to the interface configuration information in Step One. Since the formula attribute of the requestNo parameter is set to '{time:yyyyMMdd_uuid23}', the explicit parameter assembly unit will automatically generate a unique serial number with a length of 32 characters and starting with the current date format when assembling this parameter. The idNo and systemId parameters will automatically retrieve the idCardNo and systemNo parameter values ​​from the global parameter information for mapping. Since this interface requires encryption of the ID number when querying information, an encryption salt is needed to increase encryption strength. Therefore, the saltEncryption parameter will call the configured "getSaltEncryption()" function to automatically generate the encryption salt. Then, the explicit parameter assembly unit will generate the following request message: {"requestNo":"20240101_abc123456789cd987654321","idNo":"331441199001013011","systemId":"SYS_AA1","saltEncryption"="FKJCRJ"}.

[0153] Furthermore, the parameter pre-validation unit executes the custom logical expression of the `break` attribute in the data interface configuration request header based on the Aviator expression evaluation engine to determine whether the data capture conditions are met. When a business process can obtain `custInfo` from the global parameter information, this business process will not need to call the configured customer information query interface to obtain customer information; or, if the customer's age is determined to be less than 18, indicating a minor who does not meet the business age requirement, there is also no need to call the query interface to obtain customer information; only when the global parameter information of a process does not include `custInfo` and the customer is an adult is the query interface required to obtain customer information. When the data request meets the validation conditions explicitly specified in step one, the parameter pre-validation unit will perform format validation on each request parameter. When all parameters are successfully validated, the attached... Figure 4 The data retrieval request record shown has its VALIDATE_LABEL field set to '1' - verification successful. These successfully verified records will serve as reference sample data for the implicit parameter autocorrection unit during correction.

[0154] Step 3: If, during a business process, the customer information query interface detects an invalid parameter value during the parameter pre-validation unit of the dynamic parameter assembly module for data request parameters (e.g., the systemId request parameter mapping value does not start with the prefix "SYS_"), the implicit parameter auto-correction unit will automatically correct the value from the attached parameter. Figure 8The system retrieves the correct data format for the parameter name from the data request parameter format table shown and attempts to correct its parameter value to the correct value. Through this automatic parameter correction mechanism, the system can significantly reduce request anomalies caused by inconsistent parameter formats, thereby improving the success rate of data retrieval.

[0155] Step 4: When the system retrieves data from the customer information query interface of the business process, it will obtain the request message and request protocol type from the data request parameter dynamic assembly module, and by default, obtain a request thread from the high-level thread pool for data retrieval processing. This customer information query interface will perform data retrieval processing based on the DUBBO request protocol. After successful data retrieval (i.e., when its response code is "200"), the system will automatically convert and map the response parameters according to the interface configuration information and response message, and update the attached... Figure 4 The data call request log table shows the values ​​of the "Request Processing Status" and "Request Processing Time" fields for the corresponding records.

[0156] Step 5: When multiple business threads concurrently call the customer information query interface to retrieve customer information, the system simultaneously uses a sliding window algorithm to calculate the average request processing time and success rate of each interface in real time within the latest time window. Assuming the system initially calculates an average request processing time (AVGPT) of 150 milliseconds and an average success rate (AVGSR) of 100%, the system will match this interface to a high-level thread pool to obtain request threads for data scraping. After 10 minutes of high-concurrency data scraping, the calculated system AVGPT and AVGSR are 250ms and 94%, respectively. The system will then immediately match this interface to a lower-level, medium-level thread pool and begin controlling the request rate. Assuming the current system time is 11:00 and the sliding window algorithm's real-time task period is 1 minute (i.e., the statistical time window is 1 minute), the timeline for adjusting the data scraping request strategy for this interface is shown in the table below:

[0157]

[0158] As shown in the system timeline above, the system's scheduled tasks periodically collect metrics such as average processing time and average success rate of the data interface. Based on the range of these metrics, the system automatically selects appropriate data capture request strategies for processing, indirectly adjusting the data capture rate by switching between different thread pool levels. When the processing time of the called interface increases, it indicates a decrease in processing capacity and an increase in processing pressure. Therefore, the system is adjusted to a lower-level thread pool to reduce the waiting time for the request thread, thereby lowering the request rate and reducing the load on the called system to ensure system stability and availability. When the system load returns to a healthy state (average processing time and average success rate reach normal levels), the system is adjusted back to a higher-level thread pool to increase the interface request rate and maximize the system's throughput for processing cases.

[0159] Step Six: After the data capture and monitoring adjustment module in Step Four completes the data response parameter capture, replacement, and mapping, the data usage and custom extension module will call the callback method in real time to notify the caller (corresponding business process data capture node) to use the response data. Simultaneously, assuming the system needs to synchronously transmit the captured customer information to the marketing system, the system can asynchronously listen to the Kafka message queue that stores the interface response information, adapt the response information to the data structure specified by the marketing system, and then synchronously push the customer information to the marketing system. Alternatively, if the system also needs customer information data analysis, it can simultaneously pull data from this message queue, break it down, and store it in a database for data analysis.

[0160] This embodiment provides an adaptive data crawling method. Based on the interface request initiated by the target user, a preset configuration file, and a preset custom mapping rule table, request parameters are configured to obtain data configuration information. Pre-crawling validation is performed on the data configuration information to obtain the validation result. Based on the validation result and a preset sliding window algorithm, a target-level thread pool is matched from a preset-level thread pool. When a request thread is obtained from the target-level thread pool, a data crawling request is initiated according to the request protocol type corresponding to the request message. The request protocol types include HTTP, DUBBO, and KAFKA protocols. The success of the data crawling request is determined to obtain the response result. Based on the response result, the priority of the request thread is adjusted, and data crawling is completed according to the priority-adjusted request thread. Compared to traditional data crawling methods that cannot adapt to system updates in a timely manner and require manual intervention for system updates and maintenance, leading to data processing delays and affecting the accuracy and success rate of data crawling, this application can automatically map request parameters, notify the custom mapping rule table to assemble complex parameters and perform parameter pre-validation, and filter thread priorities to complete data crawling, thus improving the accuracy and success rate of data crawling.

[0161] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the adaptive data crawling method of this application. Any simple modifications based on this technical concept are within the protection scope of this application.

[0162] This application also provides an adaptive data capture device, please refer to... Figure 12 The adaptive data capture device includes:

[0163] The data interface configuration management module 10 is used to configure request parameters based on the interface request initiated by the target user, the preset configuration file, and the preset custom mapping rule table, and obtain data configuration information.

[0164] The data request parameter dynamic assembly module 20 is used to perform pre-data capture verification on the data configuration information and obtain the pre-verification result.

[0165] The data capture and monitoring adjustment module 30 is used to match the target level thread pool from the preset level thread pool based on the pre-verification result and the preset sliding window algorithm.

[0166] The data capture and monitoring adjustment module 30 is also used to enable threads in the target level thread pool to complete data capture.

[0167] Furthermore, the data interface configuration management module 10 is also used to map request parameters based on the interface request initiated by the target user and a preset configuration file to obtain display parameters. The preset configuration file includes at least one parameter of data name, data configuration file path, node response code mapping relationship, response parameter mapping relationship, request header node attribute information, and request parameter mapping relationship. The module configures the display parameters based on a preset custom mapping rule table and generates a request message; and determines data configuration information based on the request message.

[0168] Furthermore, the data request parameter dynamic assembly module 20 is also used to execute the custom logical expression of the break attribute and cando attribute of the request header in the data configuration information based on the Aviator expression evaluation engine to determine whether the data request meets the data capture conditions and obtain the condition judgment result; if the condition judgment result is that the data request meets the data capture conditions, then perform pre-data capture verification on each request parameter in the data configuration information and obtain the pre-verification result.

[0169] Furthermore, the data capture and monitoring adjustment module 30 is also used to, if the pre-verification result is successful, to calculate in real time the average request processing time and average processing success rate of each interface in the latest time window based on a preset sliding window algorithm; and to match the target level thread pool from the preset level thread pool according to the average request processing time and the average processing success rate.

[0170] Furthermore, the data capture and monitoring adjustment module 30 is also used to: if the pre-verification result is successful, mark the successfully verified request parameters based on a preset timed task and record the successfully verified request parameters in a preset request flow table as format sample data; if the pre-verification result is unsuccessful, correct the parameter format of the unsuccessful request parameters based on a preset supervised machine learning algorithm and the format sample data, and determine whether the corrected request parameters conform to the parameter format corresponding to the format sample data; if the corrected request parameters conform to the parameter format, determine that the pre-verification result is successful, and execute the step of real-time statistics of the average request processing time and average processing success rate of each interface in the latest time window based on a preset sliding window algorithm; if the corrected request parameters do not conform to the parameter format, determine that the pre-verification result is unsuccessful, and end the pre-verification process.

[0171] Furthermore, the data capture and monitoring adjustment module 30 is also configured to: if the average request processing time and the average processing success rate are within a first preset range, then match a high-level thread pool from the preset-level thread pool as the target-level thread pool; if the average request processing time and the average processing success rate are within a second preset range, then match a medium-level thread pool from the preset-level thread pool as the target-level thread pool; if the average request processing time and the average processing success rate are within a third preset range, then match a low-level thread pool from the preset-level thread pool as the target-level thread pool.

[0172] Furthermore, the data capture and monitoring adjustment module 30 is also used to initiate a data capture request according to the request protocol type corresponding to the request message when a request thread is obtained in the target level thread pool. The request protocol type includes HTTP protocol, DUBBO protocol and KAFKA protocol. It is also used to determine whether the data capture request is successfully responded to and obtain the response result. Based on the response result, the priority of the request thread is adjusted, and the data capture is completed according to the request thread after the priority adjustment.

[0173] The adaptive data crawling device provided in this application, employing the adaptive data crawling method in the above embodiments, can solve the problem that traditional data crawling methods cannot adapt to system updates in a timely manner, requiring manual intervention for system updates and maintenance, leading to data processing delays and affecting the accuracy and success rate of data crawling. Compared with the prior art, the beneficial effects of the adaptive data crawling device provided in this application are the same as those of the adaptive data crawling method provided in the above embodiments, and other technical features in the adaptive data crawling device are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0174] This application provides an adaptive data capture device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, which are executed by the at least one processor to enable the at least one processor to perform the adaptive data capture method in Embodiment 1 above.

[0175] The following is for reference. Figure 13 The diagram illustrates a structural schematic of an adaptive data capture device suitable for implementing embodiments of this application. The adaptive data capture device in these embodiments may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Description), PMPs (Portable Media Players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 13 The adaptive data capture device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments of this application.

[0176] like Figure 13As shown, the adaptive data acquisition device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.) that can perform various appropriate actions and processes according to a program stored in a read-only memory 1002 or a program loaded from a storage device 1003 into a random access memory 1004. The random access memory 1004 also stores various programs and data required for the operation of the adaptive data acquisition device. The processing unit 1001, the read-only memory 1002, and the random access memory 1004 are interconnected via a bus 1005. An input / output interface 1006 is also connected to the bus. Typically, the following systems can be connected to the input / output interface 1006: input devices 1007 including, for example, a touchscreen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 1003 including, for example, magnetic tape, hard disk, etc.; and communication devices 1009. Communication device 1009 allows the adaptive data capture device to communicate wirelessly or wiredly with other devices to exchange data. While the figure shows an adaptive data capture device with various systems, it should be understood that implementation or possession of all the systems shown is not required. More or fewer systems may be implemented alternatively.

[0177] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from read-only memory 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0178] The adaptive data crawling device provided in this application, employing the adaptive data crawling method described in the above embodiments, solves the technical problem that traditional data crawling methods cannot adapt to system updates in a timely manner, requiring manual intervention for system updates and maintenance, leading to data processing delays and affecting the accuracy and success rate of data crawling. Compared with the prior art, the beneficial effects of the adaptive data crawling device provided in this application are the same as those of the adaptive data crawling method provided in the above embodiments, and other technical features of this adaptive data crawling device are the same as those disclosed in the previous embodiment method, and will not be repeated here.

[0179] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0180] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0181] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, the computer-readable program instructions being used to execute the adaptive data capture method in the above embodiments.

[0182] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0183] The aforementioned computer-readable storage medium may be included in the adaptive data capture device; or it may exist independently and not assembled into the adaptive data capture device.

[0184] The aforementioned computer-readable storage medium carries one or more programs. When these programs are executed by the adaptive data crawling device, the adaptive data crawling device: configures request parameters based on the interface request initiated by the target user, a preset configuration file, and a preset custom mapping rule table to obtain data configuration information; performs pre-crawling verification on the data configuration information to obtain pre-verification results; matches a target-level thread pool from a preset-level thread pool based on the pre-verification results and a preset sliding window algorithm; and enables threads in the target-level thread pool to complete data crawling.

[0185] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0186] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0187] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0188] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the aforementioned adaptive data crawling method. This solves the technical problem that traditional data crawling methods cannot adapt to system updates in a timely manner, requiring manual intervention for system updates and maintenance, leading to data processing delays and affecting the accuracy and success rate of data crawling. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the adaptive data crawling method provided in the above embodiments, and will not be repeated here.

[0189] The above description is only a part of the embodiments of this application and does not limit the scope of this application. All equivalent structural transformations made under the technical concept of this application and using the content of this application specification and drawings, or direct / indirect applications in other related technical fields, are included within the protection scope of this application.

Claims

1. A method of adaptive data scraping, the method comprising: The adaptive data grabbing method comprises: Based on the interface request initiated by the target user, the preset configuration file and the preset custom mapping rule table, the request parameters are configured to obtain data configuration information; The data configuration information is pre-checking, and a pre-checking result is obtained; Based on the pre-checking result, a preset sliding window algorithm is used to match a target level thread pool from a preset level thread pool; Enable the thread in the target level thread pool to complete data grabbing; The step of pre-checking the data configuration information to obtain the pre-checking result comprises: Based on the Aviator expression evaluation engine, the break attribute and the cando attribute of the request header in the data configuration information are used to execute the custom logic expression to determine whether the data request meets the data grabbing condition, and a condition judgment result is obtained; If the condition judgment result is that the data request meets the data grabbing condition, the pre-checking of each request parameter in the data configuration information is performed to obtain the pre-checking result; The step of matching the target level thread pool from the preset level thread pool based on the pre-checking result and the preset sliding window algorithm comprises: If the pre-checking result is that the checking is successful, the average request processing time and the average processing success rate of each interface in the current latest time window are calculated in real time based on the preset sliding window algorithm; The target level thread pool is matched from the preset level thread pool according to the average request processing time and the average processing success rate; The step of matching the target level thread pool from the preset level thread pool based on the pre-checking result and the preset sliding window algorithm comprises: If the pre-checking result is that the checking is successful, the request parameters that pass the checking are marked based on the preset timing task, and the request parameters that pass the checking are recorded in the preset request flow table as format sample data; If the pre-checking result is that the checking fails, the request parameters that fail the checking are parameter format corrected based on the preset supervised machine learning algorithm and the format sample data, and it is determined whether the corrected request parameters meet the parameter format corresponding to the format sample data; If the corrected request parameters meet the parameter format, it is determined that the pre-checking result is that the checking is successful, and the step of calculating the average request processing time and the average processing success rate of each interface in the current latest time window in real time based on the preset sliding window algorithm is performed; If the corrected request parameters do not meet the parameter format, it is determined that the pre-checking result is that the checking fails, and the pre-checking process is ended.

2. The adaptive data scraping method of claim 1, wherein, The step of configuring the request parameters based on the interface request initiated by the target user, the preset configuration file and the preset custom mapping rule table to obtain the data configuration information comprises: Map the request parameters based on the interface request initiated by the target user and a preset configuration file, obtain display parameters, and the preset configuration file includes at least one of data name, data configuration file path, node response code mapping relationship, response parameter mapping relationship, request header node attribute information, and request parameter mapping relationship; Configure the display parameters based on a preset custom mapping rule table, and generate a request message; Determine data configuration information based on the request message.

3. The adaptive data scraping method of claim 1, wherein, The preset level thread pool includes a high-level thread pool, a medium-level thread pool, and a low-level thread pool. The step of matching a target level thread pool from the preset level thread pool according to the average request processing time and the average processing success rate includes: If the average request processing time and the average processing success rate are in a first preset interval, match a high-level thread pool from the preset level thread pool as the target level thread pool; If the average request processing time and the average processing success rate are in a second preset interval, match a medium-level thread pool from the preset level thread pool as the target level thread pool; If the average request processing time and the average processing success rate are in a third preset interval, match a low-level thread pool from the preset level thread pool as the target level thread pool.

4. The adaptive data scraping method of claim 2, wherein, The step of enabling threads in the target level thread pool to complete data scraping includes: When a request thread in the target level thread pool is obtained, initiate a scraping request according to the request protocol type corresponding to the request message, and the request protocol type includes HTTP protocol, DUBBO protocol, and KAFKA protocol; Determine whether the scraping request is successfully responded, and obtain a response result; Adjust the priority of the request thread based on the response result, and complete data scraping according to the request thread after priority adjustment.

5. An adaptive data scraping device, characterized by, The device includes a memory, a processor, and a computer program stored on the memory and executable on the processor, and the computer program is configured to implement the steps of the adaptive data scraping method of any one of claims 1 to 4.

6. A storage medium, characterized by The storage medium is a computer readable storage medium, and the storage medium stores a computer program, and the computer program is executed by the processor to implement the steps of the adaptive data scraping method of any one of claims 1 to 4.

7. An adaptive data scraping apparatus, comprising: The adaptive data scraping device includes: A data interface configuration management module configured to configure request parameters based on an interface request initiated by a target user, a preset configuration file, and a preset custom mapping rule table, and obtain data configuration information; A data request parameter dynamic assembly module configured to perform pre-scraping verification on the data configuration information, and obtain a pre-verification result; A data scraping and monitoring adjustment module configured to match a target level thread pool from a preset level thread pool based on the pre-verification result and a preset sliding window algorithm; The data scraping and monitoring adjustment module is further configured to enable threads in the target level thread pool to complete data scraping. The step of performing pre-checking on the data configuration information to obtain a pre-checking result comprises: Based on the Aviator expression evaluation engine, the custom logic expression of the break attribute and the cando attribute of the request header in the data configuration information is executed to determine whether the data request meets the data collection condition, and a condition judgment result is obtained; If the condition judgment result is that the data request meets the data collection condition, pre-checking is performed on each request parameter in the data configuration information to obtain a pre-checking result; The step of matching a target level thread pool from a preset level thread pool based on the pre-checking result and a preset sliding window algorithm comprises: If the pre-checking result is a successful check, the average request processing time and the average processing success rate of each interface in the current latest time window are calculated in real time based on the preset sliding window algorithm; The target level thread pool is matched from the preset level thread pool according to the average request processing time and the average processing success rate; Before the step of, if the pre-checking result is a successful check, the average request processing time and the average processing success rate of each interface in the current latest time window are calculated in real time based on the preset sliding window algorithm, the step further comprises: If the pre-checking result is a successful check, the successful request parameters are marked based on a preset timing task, and the successful request parameters are recorded in a preset request flow table as format sample data; If the pre-checking result is a failed check, the parameter format of the failed request parameters is corrected based on a preset supervised machine learning algorithm and the format sample data, and it is determined whether the corrected request parameters meet the parameter format corresponding to the format sample data; If the corrected request parameters meet the parameter format, it is determined that the pre-checking result is a successful check, and the step of calculating the average request processing time and the average processing success rate of each interface in the current latest time window in real time based on the preset sliding window algorithm is executed; If the corrected request parameters do not meet the parameter format, it is determined that the pre-checking result is a failed check, and the pre-checking process is ended.

Images