Automatic data acquisition and standardization processing method and system

Abstract

The invention discloses an automatic data acquisition and standardization processing method and system, and relates to the technical field of big data processing, and the method comprises the steps: configuring acquisition task information, rapidly completing data acquisition preparation based on a preset template, and storing data and configuration information to a database. And simulating user operation to load a dynamic page and generate a page snapshot, thereby ensuring comprehensive and accurate data acquisition. In the processing stage, unstructured data is converted into structured data by using OCR, and standardization processing is performed through a multi-parameter nonlinear standardization formula to ensure consistency. And the database mapping tool automatically stores the processed data into the database, so that errors are reduced. And then generating a data processing flow, and outputting an acquisition frequency optimization value Fopt through pre-acquisition testing to ensure that the task is executed at the optimal frequency. And calculating an evaluation index Eval through an evaluation algorithm, comparing the evaluation index Eval with a performance threshold value E, and optimizing and outputting an optimization index Optim until the acquisition performance returns to normal.

Description

Technical Field

[0001] The present invention relates to the technical field of big data processing, and in particular to a method and system for automated data collection and standardized processing. Background Art

[0002] In the field of big data processing, data collection is the basic link of information processing. It obtains raw data from different sources through various collection tools and technologies. In the specific application of intelligent information management, especially for unstructured and heterogeneous data, the complexity and real-time requirements of data collection are increasing. Therefore, automated data collection tools must not only have efficient data acquisition capabilities, but also be able to perform data standardization processing based on the characteristics of different data sources to ensure data consistency and availability.

[0003] At present, most existing data collection methods rely on direct connection to the system interface or manual configuration, which requires a large amount of development and has poor flexibility, making it difficult to cope with frequently changing business system requirements. At the same time, traditional methods are often not efficient enough in dealing with dynamically loaded web page content and unstructured data. This limitation leads to inefficiency, incomplete data, and security issues in the data collection process. Especially when processing heterogeneous data sources, data integration and standardization require a lot of manual intervention, which cannot meet the business needs of rapid response. Summary of the invention

[0004] In view of the deficiencies in the prior art, the present invention provides a method and system for automated data collection and standardized processing, which solves the problems mentioned in the background technology.

[0005] To achieve the above objectives, the present invention is implemented by the following technical scheme: comprising the following steps:

[0006] S1. Configure collection task information: configure the data collection template to set the collection system information and task information, collect data, and store the collected data and collection configuration data in the system database;

[0007] S2. Dynamic page loading and parsing: Loading dynamic content web pages by simulating user operations, parsing the interactive interface of the page, and obtaining the data field list and page snapshot;

[0008] S3, parameter identification: perform a box selection operation on the page snapshot, use the OCR tool to convert the unstructured data of the boxed part into structured data, and obtain the mapping field description list;

[0009] S4, field standardization and mapping: construct a multi-parameter nonlinear standardization formula, standardize the data field list and the mapping field description list according to the configured standardization rules, output the standardized data Norm, and automatically import the data field list and the mapping field description list through the database mapping tool and store them in the system database;

[0010] S5, data processing flow generation and testing: according to S1 to S4, a data processing flow is generated, and a pre-collection test is performed. During the pre-collection test, a multi-dimensional collection frequency optimization formula is used to calculate and output an optimized collection frequency value Fopt;

[0011] S6. Task scheduling and automated execution: According to the optimized acquisition frequency optimization value Fopt, the acquisition and processing tasks are automatically executed to generate the entire data processing flow. After the data processing flow is completed, the comprehensive analysis algorithm formula is used to calculate and output the comprehensive evaluation index Eval. The preset acquisition performance threshold E is compared with the obtained comprehensive index Eval for preliminary comparative evaluation. Based on the preliminary evaluation results, the acquisition optimization formula is used to calculate the output optimization index Optim for secondary optimization.

[0012] Preferably, said S1 includes S11, S12 and S13;

[0013] S11. Configure the collection task information: configure the data collection task information, including the task name, collection frequency and collection mode, through the graphical user interface provided by the configuration personnel to the enterprise system. At the same time, the configuration personnel also set the unique identification, automatic execution frequency and collection mode of each task through the graphical interface;

[0014] The task name is a unique name set by the configuration personnel for each task for task management and tracking;

[0015] The collection frequency uses a scheduling task framework to automatically trigger task execution to periodically perform automated data collection;

[0016] The collection mode is set to collect only incremental data through the CDC technology, timestamp field and task name identification of the system database;

[0017] S12, target system configuration: the configuration personnel inputs the name of the data source system in the configuration interface, and saves the name of the data source system in the system database. The configuration personnel provides login information of the target data source system during configuration, and the login information includes a user name and a password. The user name and the password are encrypted and stored in the system database using the AES encryption method, and the data source system is automatically logged in using an HTTP request. The configuration personnel defines the data page path of the target data source system, and automatically accesses the data page path of the target data source system through the API interface provided by the data source system to capture the target data, and collects the configuration data through the running log record of the collection task;

[0018] The collection configuration data includes the total collection time Tt, the number of collection times Np, the resource utilization rate Ru, the enterprise system capacity S, the error rate Er, and the collection data volume Ds;

[0019] S13, database configuration and storage: connect to the system database through the JDBC interface and the ODBC interface, and the configuration personnel set the system database location structure for storing the target data and the collection configuration data. The system database location structure includes the address, table structure and storage format, and stores the target data and the collection configuration data in the database according to the set system database location structure, and then performs data encryption processing.

[0020] Preferably, S2 includes S21 and S22;

[0021] S21, dynamic page loading: after configuring the collection task information, use WebDriver to simulate real user operations. When WebDriver loads the page, execute the JavaScript code in the browser to display the asynchronously loaded data. After loading, use WebDriver to extract the user name and password from the system database, automatically fill in the login form and submit it;

[0022] S22. Network interaction interface analysis: When the page is loading, Selenium is used in combination with the browser's developer tools to monitor network traffic and capture the interaction requests between the page and the server. The parsed page obtains a list of network interaction interfaces and selects a data interface. The system parses the data fields in the data interface and generates a list of data fields. After the page is loaded, Selenium provides a built-in screenshot function to automatically generate a screenshot system for the page and obtain a page snapshot.

[0023] Preferably, S3, parameter identification: by using optical character recognition technology OCR to identify data in non-standard text and images in page snapshots, and using OCR tools to convert unstructured data into structured data to extract text information in the page snapshots, and obtain a mapping field description list.

[0024] Preferably, said S4 includes S41 and S42;

[0025] S41, standardization configuration: according to the standardization rules configured by the user in the graphical interface, the standardization rules include unit unification, nonlinear scaling and noise filtering, a multi-parameter nonlinear standardization formula is constructed, and the standardized processing data Norm is calculated and output;

[0026] The standardized processing data Norm is calculated and obtained by the following multi-parameter nonlinear standardization formula;

[0027] ;

[0028] In the formula, x represents the original data, that is, the value in the data field list and the mapping field description list that needs to be standardized. represents the average value of the original data x, represents the standard deviation of the original data x, log represents the logarithmic function, Represents the first smoothing factor, which is used to control the scaling effect on the original data x. represents the second smoothing factor, which is a protection term used to prevent errors in logarithmic operations;

[0029] S42. Data mapping: Based on the acquired data field list, field description list and data standard configurations, the data mapping is completed one by one according to the data field list and the mapping field description list through the database mapping tool, and automatically imported into the system database table.

[0030] Preferably, said S5 includes S51 and S52;

[0031] S51, process generation: according to the configured configuration collection task information, dynamic page loading and parsing, parameter identification and field standardization and mapping, use the workflow engine to automatically manage and execute the configuration collection task information to field standardization and mapping, and set the scheduled execution through the scheduling engine;

[0032] S52, pre-collection test and optimization: automatically perform pre-collection test after the process is generated, use multi-dimensional collection frequency optimization formula in the pre-collection test process, input the collection configuration data, calculate and output the collection frequency optimization value Fopt;

[0033] The acquisition frequency optimization value Fopt is calculated and obtained by the following multi-dimensional acquisition frequency optimization formula;

[0034] ;

[0035] In the formula, argmin represents the independent variable value that makes the objective function reach the lower limit, Fc represents the acquisition frequency, represents the collection efficiency, Indicates resource utilization. Indicates data quality and error rate. a1, a2 and a3 represent the preset weight values ​​of collection efficiency, resource utilization rate and data quality and error rate respectively. The specific values ​​are set by the user.

[0036] Preferably, said S6 includes S61, S62 and S63;

[0037] S61. After pre-collection testing and optimization, automatically execute the collection and processing tasks, complete the entire data processing flow, and construct a comprehensive analysis algorithm formula to calculate and output the comprehensive evaluation index Eval;

[0038] The comprehensive evaluation index Eval is calculated by the following comprehensive analysis algorithm formula;

[0039] ;

[0040] In the formula, It represents the total amount of data collected within the collection time, Tc represents the collection time, Qdata represents the data quality, Sreq represents the resource demand, b1, b2 and b3 represent the preset weight values ​​of the effective data volume, resource utilization rate and data quality respectively, and the specific values ​​are set by the user.

[0041] Preferably, S62, extracting the average value of the data collection performance index and presetting it as the collection performance threshold E, then comparing and evaluating the collection performance threshold E with the obtained comprehensive evaluation index Eval, analyzing the data collection performance, and generating an optimization execution strategy, the specific evaluation content is as follows;

[0042] When the comprehensive evaluation index Eval> the collection performance threshold E, it means that the collection performance is normal and there is no need to execute the optimization strategy;

[0043] When the comprehensive evaluation index Eval ≤ the collection performance threshold E, it means that the collection performance is abnormal, and the optimization strategy is executed at this time.

[0044] Preferably, in S63, when the acquisition performance is initially evaluated to be abnormal, an acquisition optimization formula is constructed, the current comprehensive evaluation index Eval is input into the acquisition optimization formula, the optimization calculation outputs the optimization index Optim, a secondary optimization is performed, and the optimization iterative execution process steps are optimized until the output acquisition performance is normal and the iteration is stopped;

[0045] The optimization index Optim is calculated and output by the following acquisition optimization formula;

[0046] ;

[0047] In the formula, Niterations represents the number of iterations, Qi represents the effect of the i-th optimization, and Ri represents the resource consumption of the i-th optimization. Represents the first optimization weight, reflecting the importance of the evaluation results in the optimization process in the optimization decision. Represents the second optimization weight, reflecting the importance of resource consumption in the optimization decision.

[0048] An automated data collection and standardized processing system, comprising a configuration module, a loading module, a parameter identification module, a processing flow generation module and an evaluation and optimization module;

[0049] The configuration module is used to configure the data collection template to set the collection system information and task information, perform data collection, and store the collected data and collection configuration data in the system database;

[0050] The loading module is used to simulate user operations to load a web page with dynamic content, parse the interactive interface of the page, and obtain a data field list and a page snapshot;

[0051] The parameter recognition module is used to perform a box selection operation on the page snapshot, convert the unstructured data of the box selection part into structured data using an OCR tool, and obtain a mapping field description list;

[0052] The generation processing flow module is used to construct a multi-parameter nonlinear standardization formula, standardize the data field list and the mapping field description list according to the configuration standardization rules, output the standardized processing data Norm, and automatically import the data field list and the mapping field description list through the database mapping tool, and store them in the system database, and generate a data processing flow, and perform a pre-collection test. During the pre-collection test, a multi-dimensional collection frequency optimization formula is used to calculate and output the collection frequency optimization value Fopt;

[0053] The evaluation and optimization module is used to automatically execute the collection and processing tasks according to the optimized collection frequency optimization value Fopt, generate the entire data processing flow, and after the data processing flow is completed, use the comprehensive analysis algorithm formula to calculate and output the comprehensive evaluation index Eval, and perform a preliminary comparative evaluation on the preset collection performance threshold E and the obtained comprehensive index Eval. Based on the preliminary evaluation results, use the collection optimization formula to calculate the output optimization index Optim for secondary optimization.

[0054] The present invention provides a method and system for automated data collection and standardized processing, which has the following beneficial effects:

[0055] (1) This method realizes the automated data collection process by configuring the collection task information and loading and parsing dynamic pages. WebDriver is used to simulate user operations, load web pages with dynamic content, and combine network interactive interface parsing technology to obtain data field lists and page snapshots, ensuring the accuracy and real-time nature of data collection. At the same time, the system automatically stores the collected data and configuration information in the system database, avoiding omissions and errors in the traditional manual collection process and improving collection efficiency.

[0056] (2) In the parameter recognition stage, this method converts the unstructured page snapshot data into structured data through the OCR tool to obtain the mapping field description list. Subsequently, the multi-parameter nonlinear standardization formula S4 is constructed to standardize the data fields according to the user-configured standardization rules to generate standardized data Norm. This process not only automatically completes the data conversion and standardization, but also ensures the consistency and compatibility of the data, reducing the workload of manual intervention and manual data cleaning.

[0057] (3) This method generates a data processing flow and performs a pre-collection test, and uses a multi-dimensional collection frequency optimization formula to calculate the collection frequency optimization value Fopt to ensure a balance between data collection frequency and system resource usage. During task scheduling and automated execution, the system calculates the comprehensive evaluation index Eval through a comprehensive evaluation algorithm formula, compares it with the preset collection performance threshold E, identifies collection performance problems in a timely manner, and performs secondary optimization through the collection optimization formula to calculate the output optimization index Optim. This intelligent optimization process effectively reduces resource consumption and improves the overall performance of data collection and processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 A schematic diagram of a method flow for automated data collection and standardization processing according to the present invention;

[0059] Figure 2 The present invention is a schematic diagram of the system steps of automated data collection and standardized processing. DETAILED DESCRIPTION

[0060] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0061] Example 1

[0062] See also Figure 1The present invention provides a method for automated data collection and standardized processing. To achieve the above objectives, the present invention is implemented through the following technical solutions: comprising the following steps:

[0063] S1. Configure collection task information: configure the data collection template to set the collection system information and task information, collect data, and store the collected data and collection configuration data in the system database;

[0064] S2. Dynamic page loading and parsing: Loading dynamic content web pages by simulating user operations, parsing the interactive interface of the page, and obtaining the data field list and page snapshot;

[0065] S3, parameter identification: perform a box selection operation on the page snapshot, use the OCR tool to convert the unstructured data of the boxed part into structured data, and obtain the mapping field description list;

[0066] S4, field standardization and mapping: construct a multi-parameter nonlinear standardization formula, standardize the data field list and the mapping field description list according to the configured standardization rules, output the standardized data Norm, and automatically import the data field list and the mapping field description list through the database mapping tool and store them in the system database;

[0067] S5, data processing flow generation and testing: according to S1 to S4, a data processing flow is generated, and a pre-collection test is performed. During the pre-collection test, a multi-dimensional collection frequency optimization formula is used to calculate and output an optimized collection frequency value Fopt;

[0068] S6. Task scheduling and automated execution: According to the optimized acquisition frequency optimization value Fopt, the acquisition and processing tasks are automatically executed to generate the entire data processing flow. After the data processing flow is completed, the comprehensive analysis algorithm formula is used to calculate and output the comprehensive evaluation index Eval. The preset acquisition performance threshold E is compared with the obtained comprehensive index Eval for preliminary comparative evaluation. Based on the preliminary evaluation results, the acquisition optimization formula is used to calculate the output optimization index Optim for secondary optimization.

[0069] In this embodiment, the method configures the collection task information, and the system can quickly complete the preparation work of data collection according to the preset template and task configuration, and store the collected raw data and configuration information in the database, laying the foundation for subsequent processing. Then, through dynamic page loading and parsing, the system can simulate user operations, load dynamic web page content, parse the data fields in the page and generate page snapshots to ensure that the collected data is comprehensive and accurate. In the data processing stage, the system uses the OCR tool to convert the unstructured data in the page snapshot into structured data through parameter recognition, generates a mapping field description list, and solves the problem of processing unstructured data. Subsequently, field standardization and mapping are used to standardize the data through a multi-parameter nonlinear standardization formula, and the standardized data Norm is output. At the same time, the database mapping tool can automatically import the processed data into the system database to ensure the consistency and manageability of the data and reduce the risk of errors in manual processing. Finally, through data processing process generation and testing and task scheduling and automatic execution, the system generates a complete processing flow according to the previous configuration, and outputs the collection frequency optimization value Fopt in the pre-collection test to ensure that the task is executed at the optimal collection frequency. After the entire data processing process is completed, the system calculates the evaluation index Eval through a comprehensive analysis algorithm, compares it with the preset acquisition performance threshold E, determines whether it is necessary to further optimize the output optimization index Optim, and then optimizes the iterative execution process steps until the output acquisition performance stops iterating normally. Compared with traditional technologies, this system effectively improves the automation of data collection and processing, significantly optimizes resource utilization and data processing performance, reduces human intervention, and improves overall efficiency and accuracy.

[0070] Example 2

[0071] This embodiment is explained in Example 1, please refer to Figure 1 , specifically: S1 includes S11, S12 and S13;

[0072] S11. Configure the collection task information: configure the data collection task information, including the task name, collection frequency and collection mode, through the graphical user interface provided by the configuration personnel to the enterprise system. At the same time, the configuration personnel also set the unique identification, automatic execution frequency and collection mode of each task through the graphical interface;

[0073] Task name: The configuration personnel sets a unique name for each task for task management and tracking;

[0074] The frequency of collection is achieved by using the scheduling task framework to automatically trigger task execution to perform automated data collection at regular intervals;

[0075] The collection mode is set to collect only incremental data through the CDC technology of the system database, the timestamp field and the task name identifier.

[0076] In the enterprise system interface, the configuration personnel set a task name for each data collection task by configuring the data collection template, which is convenient for searching in the log, viewing in the task management and identifying during the debugging process. The displacement key UniqueKey in the system database is used to verify the uniqueness of the task name. When the task name conflicts, a prompt pops up to ask the configuration personnel to change the task name. At the same time, the configuration personnel set the collection frequency in the task configuration interface of the enterprise system interface. The collection frequency includes hourly, daily and weekly, and use the scheduling task framework to trigger the task execution at a fixed time. The scheduling task framework automatically executes the collection task based on the predefined time rules, and then sets it to collect only incremental data through the CDC technology, timestamp field and task name identifier of the system database.

[0077] S12. Target system configuration: The configuration personnel inputs the name of the data source system in the configuration interface and saves the name of the data source system in the system database. When generating logs or debugging, the name can be used to quickly find related collection tasks. The configuration personnel provide the login information of the target data source system during configuration. The login information includes a user name and a password. The user name and password are encrypted using the AES encryption method and stored in the system database. The HTTP request is used to automatically log in to the data source system. The configuration personnel defines the data page path of the target data source system and automatically accesses the data page path of the target data source system through the API interface provided by the data source system to capture the target data. The collection configuration data is recorded in the running log of the collection task.

[0078] The collection configuration data includes the total collection time Tt, the number of collection times Np, the resource utilization rate Ru, the enterprise system capacity S, the error rate Er, and the amount of collected data Ds;

[0079] S13. Database configuration and storage: Connect to the system database through the JDBC interface and ODBC interface. The configuration personnel set the system database location structure for storing target data and collection configuration data. The system database location structure includes the address, table structure and storage format. The target data and collection configuration data are stored in the database according to the set system database location structure, and then the data is encrypted.

[0080] In this embodiment, the configuration of the method to collect task information simplifies the configuration process through a graphical user interface, automatically sets the task name, collection frequency and collection mode, realizes automated and regular incremental data collection, reduces the complexity of manual management, and improves the accuracy and efficiency of data collection. The target system configuration ensures the security of data transmission by encrypting login information and automating access to the target data source system, and automatically captures target data through the API interface, reducing the risk of errors caused by manual operations. Finally, the database configuration and storage connects to the system database through the JDBC interface and the ODBC interface, ensuring that the collected data is stored safely and accurately according to the set database structure, and encrypts the data to further improve the security of the data.

[0081] Example 3

[0082] This embodiment is explained in Example 2. Please refer to Figure 1 , specifically: S2 includes S21 and S22;

[0083] S21. Dynamic page loading: After configuring the collection task information, use WebDriver to simulate real user operations. When WebDriver loads the page, execute the JavaScript code in the browser to display the asynchronously loaded data. The asynchronously loaded data includes table or chart data obtained from the server through Ajax requests. The system ensures that all dynamic data has been loaded by waiting for the page to load. After loading, the user name and password in the system database are extracted through WebDriver, and the login form is automatically filled in and submitted.

[0084] S22. Network interaction interface analysis: When the page is loading, Selenium is used in combination with the browser's developer tools to monitor network traffic and capture the interaction requests between the page and the server. The parsed page obtains a list of network interaction interfaces and selects a data interface. The system parses the data fields in the data interface and generates a list of data fields. After the page is loaded, Selenium provides a built-in screenshot function to automatically generate a screenshot system for the page and obtain a page snapshot.

[0085] S3. Parameter identification: Use optical character recognition technology (OCR) to identify data in non-standard text and images in page snapshots, and use OCR tools to convert unstructured data into structured data to extract text information in page snapshots and obtain a mapping field description list.

[0086] In the present embodiment, the method uses WebDriver to simulate the operation of real users through dynamic page loading, and automatically executes JavaScript code to asynchronously load page data, ensuring that dynamic content such as tables and chart data can be fully presented and collected. In addition, the system can automatically complete the login operation by extracting the username and password in the database, thereby automating the entire data acquisition process, and the network interaction interface parsing uses Selenium to monitor the interaction request between the web page and the server, parse out the valid data field and generate a field list, and generate a page snapshot through the screenshot function to ensure that the data is traceable and reproducible. On this basis, parameter recognition recognizes non-standard text and image data in the page snapshot through OCR technology, and automatically converts unstructured data into structured data. The system can intelligently extract page text information and generate a mapping field description list, simplifying the process of processing unstructured data.

[0087] Example 4

[0088] This embodiment is explained in Example 3, please refer to Figure 1 , specifically: S4 includes S41 and S42;

[0089] S41, standardization configuration: according to the user configuration of standardization rules in the graphical interface, the standardization rules include unit unification, nonlinear scaling and noise filtering, construct a multi-parameter nonlinear standardization formula, and calculate and output the standardized processing data Norm;

[0090] The standardized processing data Norm is calculated and obtained by the following multi-parameter nonlinear standardization formula;

[0091] ;

[0092] In the formula, x represents the original data, that is, the value in the data field list and the mapping field description list that needs to be standardized. Represents the average value of the original data x. The mean is used to center the data. By subtracting the mean, the data is adjusted to a range around zero so that the standardized data is not affected by the absolute value. It represents the standard deviation of the original data x. The standard deviation is used to adjust the scale of the data. By dividing it by the standard deviation, the distribution of the data is standardized so that data with different characteristics can be uniformly compared during processing. Log represents the logarithmic function. Represents the first smoothing factor, which is used to control the scaling effect on the original data x. Represents the second smoothing factor, which is used to prevent errors in logarithmic operations. where x takes zero or negative value;

[0093] S42. Data mapping: Based on the acquired data field list, field description list and data standard configurations, the data mapping is completed one by one according to the data field list and the mapping field description list through the database mapping tool, and automatically imported into the system database table.

[0094] In this embodiment, the method can set standardization rules through a graphical interface through a standardized configuration, including unit conversion, nonlinear scaling and noise filtering. The system constructs a multi-parameter nonlinear standardization formula based on these rules to generate the standardized data Norm. The formula adjusts the data by the mean and standard deviation so that it is distributed around zero, ensuring that data with different characteristics are processed under a unified scale. This standardization not only improves the consistency of the data, but also greatly reduces the data deviation in subsequent analysis, ensuring that data from different sources can be uniformly processed and compared. Data mapping automatically imports the standardized data into the system database through a database mapping tool. The system maps one by one according to the configured data field list and the mapping field description list to ensure that the data is stored in a predefined format, thereby improving the automation level of data management.

[0095] Example 5

[0096] This embodiment is explained in Example 4. Please refer to Figure 1 , specifically: S5 includes S51 and S52;

[0097] S51, process generation: according to the configured configuration collection task information, dynamic page loading and parsing, parameter identification and field standardization and mapping, use the workflow engine to automatically manage and execute the configuration collection task information to field standardization and mapping, and set the scheduled execution through the scheduling engine;

[0098] S52, pre-collection test and optimization: automatically perform pre-collection test after the process is generated, use multi-dimensional collection frequency optimization formula in the pre-collection test process, input the collection configuration data, calculate and output the collection frequency optimization value Fopt;

[0099] The acquisition frequency optimization value Fopt is calculated and obtained by the following multi-dimensional acquisition frequency optimization formula;

[0100] ;

[0101] In the formula, argmin represents the independent variable value that makes the objective function reach the lower limit, Fc represents the acquisition frequency, represents the collection efficiency, Indicates resource utilization. Indicates data quality and error rate. a1, a2 and a3 represent the preset weight values ​​of collection efficiency, resource utilization rate and data quality and error rate respectively. The specific values ​​are set by the user.

[0102] In this embodiment, the method is generated through a process, and the system can use the workflow engine to automatically manage and execute the entire data collection and standardization process according to the configured collection task information, dynamic page loading, parameter identification and field standardization steps. This process is set to execute regularly through the scheduling engine, which ensures the continuity and stability of data collection and processing, without manual intervention, and greatly improves work efficiency. Pre-collection test and optimization evaluate the efficiency and resource usage of the collection process through pre-collection testing. The system uses a multi-dimensional collection frequency optimization formula, combined with the collection configuration data, to calculate and output the optimal collection frequency optimization value Fopt. This optimization value ensures that data collection achieves the highest efficiency and quality with the least resource consumption by considering factors such as collection frequency, collection efficiency, resource utilization and data quality.

[0103] Example 6

[0104] This embodiment is explained in Example 1, please refer to Figure 1 , specifically: S6 includes S61, S62 and S63;

[0105] S61. After pre-collection testing and optimization, automatically execute the collection and processing tasks, complete the entire data processing flow, and construct a comprehensive analysis algorithm formula to calculate and output the comprehensive evaluation index Eval;

[0106] The comprehensive evaluation index Eval is calculated by the following comprehensive analysis algorithm formula;

[0107] ;

[0108] In the formula, It represents the total amount of data collected within the collection time, Tc represents the collection time, Qdata represents the data quality, Sreq represents the resource demand, b1, b2 and b3 represent the preset weight values ​​of the effective data volume, resource utilization rate and data quality respectively, and the specific values ​​are set by the user.

[0109] S62, extracting the average value of the data collection performance index and presetting it as the collection performance threshold E, then comparing and evaluating the collection performance threshold E with the obtained comprehensive evaluation index Eval, analyzing the data collection performance, and generating an optimization execution strategy. The specific evaluation content is as follows;

[0110] When the comprehensive evaluation index Eval> the collection performance threshold E, it means that the collection performance is normal and there is no need to execute the optimization strategy;

[0111] When the comprehensive evaluation index Eval ≤ the collection performance threshold E, it means that the collection performance is abnormal, and the optimization strategy is executed at this time.

[0112] S63, when the acquisition performance is abnormal through preliminary evaluation, construct an acquisition optimization formula, input the current comprehensive evaluation index Eval into the acquisition optimization formula, optimize the calculation output optimization index Optim, perform secondary optimization, and then optimize the iterative execution process steps until the output acquisition performance is normal and the iteration is stopped;

[0113] The optimization index Optim is calculated and output through the following acquisition optimization formula;

[0114] ;

[0115] In the formula, Niterations represents the number of iterations, Qi represents the effect of the i-th optimization, and Ri represents the resource consumption of the i-th optimization. Represents the first optimization weight, reflecting the importance of the evaluation results in the optimization process in the optimization decision. Represents the second optimization weight, reflecting the importance of resource consumption in the optimization decision.

[0116] In this embodiment, the method calculates the comprehensive evaluation index Eval through a comprehensive analysis algorithm formula, and quantitatively evaluates multiple dimensions involved in the data collection process, such as the total data volume, collection time, data quality, and resource requirements, to ensure comprehensive monitoring of the overall performance of the collection task. By comparing the comprehensive evaluation index with the preset collection performance threshold E, it is determined whether the performance of the collection task meets the standard. When the evaluation index is lower than the threshold, the system will automatically generate an optimization strategy to ensure that the collection task is corrected in time when the performance is abnormal. Further, in the case of abnormal performance, the collection optimization formula is used for secondary optimization. The system calculates the optimization index Optim through multiple iterative optimizations, and gradually improves the collection performance until the performance returns to normal levels. The optimization process takes into account the trade-off between collection effect and resource consumption, ensuring that resource waste is minimized while improving collection quality.

[0117] Example 7

[0118] See also Figure 2 , an automated data collection and standardization processing system, including a configuration module, a loading module, a parameter identification module, a processing flow generation module and an evaluation and optimization module;

[0119] The configuration module is used to configure the data collection template to set the collection system information and task information, perform data collection, and store the collected data and collection configuration data in the system database;

[0120] The loading module is used to simulate user operations to load dynamic content web pages, parse the interactive interface of the page, and obtain the data field list and page snapshot;

[0121] The parameter recognition module is used to perform a box selection operation on the page snapshot, use the OCR tool to convert the unstructured data of the box selection part into structured data, and obtain the mapping field description list;

[0122] The processing flow generation module is used to construct a multi-parameter nonlinear standardization formula, standardize the data field list and the mapping field description list according to the configuration standardization rules, output the standardized processing data Norm, and automatically import the data field list and the mapping field description list through the database mapping tool, and store them in the system database, and generate a data processing flow, and perform a pre-collection test. During the pre-collection test, a multi-dimensional collection frequency optimization formula is used to calculate and output the collection frequency optimization value Fopt;

[0123] The evaluation and optimization module is used to automatically perform collection and processing tasks according to the optimized collection frequency optimization value Fopt, generate the entire data processing flow, and after the data processing flow is completed, use the comprehensive analysis algorithm formula to calculate and output the comprehensive evaluation index Eval, and perform a preliminary comparative evaluation between the preset collection performance threshold E and the obtained comprehensive index Eval. Based on the preliminary evaluation results, use the collection optimization formula to calculate the output optimization index Optim for secondary optimization.

[0124] While the embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that many changes, modifications, substitutions and variations can be made to the embodiments without departing from the principles and spirit of the invention.

Claims

1. A method for automated data collection and standardized processing, characterized in that: The following steps are involved: S1. Configure collection task information: configure the data collection template to set the collection system information and task information, collect data, and store the collected data and collection configuration data in the system database; S2. Dynamic page loading and parsing: Loading dynamic content web pages by simulating user operations, parsing the interactive interface of the page, and obtaining the data field list and page snapshot; S3, parameter identification: perform a box selection operation on the page snapshot, use the OCR tool to convert the unstructured data of the boxed part into structured data, and obtain the mapping field description list; S4, field standardization and mapping: construct a multi-parameter nonlinear standardization formula, standardize the data field list and the mapping field description list according to the configured standardization rules, output the standardized data Norm, and automatically import the data field list and the mapping field description list through the database mapping tool and store them in the system database; S5, data processing flow generation and testing: according to S1 to S4, a data processing flow is generated, and a pre-collection test is performed. During the pre-collection test, a multi-dimensional collection frequency optimization formula is used to calculate and output an optimized collection frequency value Fopt; S6. Task scheduling and automated execution: According to the optimized acquisition frequency optimization value Fopt, the acquisition and processing tasks are automatically executed to generate the entire data processing flow. After the data processing flow is completed, the comprehensive analysis algorithm formula is used to calculate and output the comprehensive evaluation index Eval. The preset acquisition performance threshold E is compared with the obtained comprehensive index Eval for preliminary comparative evaluation. Based on the preliminary evaluation results, the acquisition optimization formula is used to calculate the output optimization index Optim for secondary optimization.

2. The method of automated data collection and standardization processing according to claim 1, characterized in that: Said S1 includes S11, S12 and S13; S11. Configure the collection task information: configure the data collection task information, including the task name, collection frequency and collection mode, through the graphical user interface provided by the configuration personnel to the enterprise system. At the same time, the configuration personnel then set the unique identification, automatic execution frequency and collection mode of each task through the graphical interface; The task name is a unique name set by the configuration personnel for each task for task management and tracking; The collection frequency uses a scheduling task framework to automatically trigger task execution to periodically perform automated data collection; The collection mode is set to collect only incremental data through the CDC technology, timestamp field and task name identification of the system database; S12, target system configuration: the configuration personnel inputs the name of the data source system in the configuration interface, and saves the name of the data source system in the system database. The configuration personnel provides login information of the target data source system during configuration, and the login information includes a user name and a password. The user name and the password are encrypted and stored in the system database using the AES encryption method, and the data source system is automatically logged in using an HTTP request. The configuration personnel defines the data page path of the target data source system, and automatically accesses the data page path of the target data source system through the API interface provided by the data source system to capture the target data, and collects the configuration data through the running log record of the collection task; The collection configuration data includes the total collection time Tt, the number of collection times Np, the resource utilization rate Ru, the enterprise system capacity S, the error rate Er, and the collection data volume Ds; S13, database configuration and storage: connect to the system database through the JDBC interface and the ODBC interface, and the configuration personnel set the system database location structure for storing the target data and the collection configuration data. The system database location structure includes the address, table structure and storage format, and stores the target data and the collection configuration data in the database according to the set system database location structure, and then performs data encryption processing.

3. The method of automated data collection and standardization processing according to claim 1, characterized in that: The S2 includes S21 and S22; S21, dynamic page loading: after configuring the collection task information, use WebDriver to simulate real user operations. When WebDriver loads the page, execute the JavaScript code in the browser to display the asynchronously loaded data. After loading, use WebDriver to extract the user name and password from the system database, automatically fill in the login form and submit it; S22. Network interaction interface analysis: When the page is loading, Selenium is used in combination with the browser's developer tools to monitor network traffic and capture the interaction requests between the page and the server. The parsed page obtains a list of network interaction interfaces and selects a data interface. The system parses the data fields in the data interface and generates a list of data fields. After the page is loaded, Selenium provides a built-in screenshot function to automatically generate a screenshot system for the page and obtain a page snapshot.

4. The method of automated data collection and standardization processing according to claim 1, characterized in that: S3. Parameter identification: Use optical character recognition technology (OCR) to identify data in non-standard text and images in page snapshots, and use OCR tools to convert unstructured data into structured data to extract text information in page snapshots and obtain a mapping field description list.

5. The method of automated data collection and standardization processing according to claim 1, characterized in that: The S4 includes S41 and S42; S41, standardization configuration: according to the standardization rules configured by the user in the graphical interface, the standardization rules include unit unification, nonlinear scaling and noise filtering, a multi-parameter nonlinear standardization formula is constructed, and the standardized processing data Norm is calculated and output; The standardized processing data Norm is calculated and obtained by the following multi-parameter nonlinear standardization formula; ； In the formula, x represents the original data, that is, the value in the data field list and the mapping field description list that needs to be standardized. represents the average value of the original data x, represents the standard deviation of the original data x, log represents the logarithmic function, Represents the first smoothing factor, which is used to control the scaling effect on the original data x. represents the second smoothing factor, which is a protection term used to prevent errors in logarithmic operations; S42. Data mapping: Based on the acquired data field list, field description list and data standard configurations, the data mapping is completed one by one through the database mapping tool according to the data field list and the mapping field description list, and automatically imported into the system database table.

6. The method of automated data collection and standardization processing according to claim 2, characterized in that: The S5 includes S51 and S52; S51, process generation: according to the configured configuration collection task information, dynamic page loading and parsing, parameter identification and field standardization and mapping, use the workflow engine to automatically manage and execute the configuration collection task information to field standardization and mapping, and set the scheduled execution through the scheduling engine; S52, pre-collection test and optimization: automatically perform pre-collection test after the process is generated, use multi-dimensional collection frequency optimization formula in the pre-collection test process, input the collection configuration data, calculate and output the collection frequency optimization value Fopt; The acquisition frequency optimization value Fopt is calculated and obtained by the following multi-dimensional acquisition frequency optimization formula; ； In the formula, argmin represents the independent variable value that makes the objective function reach the lower limit, Fc represents the acquisition frequency, represents the collection efficiency, Indicates resource utilization. Indicates data quality and error rate. a1, a2 and a3 represent the preset weight values ​​of collection efficiency, resource utilization rate and data quality and error rate respectively. The specific values ​​are set by the user.

7. The method of automated data collection and standardization processing according to claim 2, characterized in that: The S6 includes S61, S62 and S63; S61. After pre-collection testing and optimization, automatically execute the collection and processing tasks, complete the entire data processing flow, and construct a comprehensive analysis algorithm formula to calculate and output the comprehensive evaluation index Eval; The comprehensive evaluation index Eval is calculated by the following comprehensive analysis algorithm formula; ； In the formula, It represents the total amount of data collected within the collection time, Tc represents the collection time, Qdata represents the data quality, Sreq represents the resource demand, b1, b2 and b3 represent the preset weight values ​​of the effective data volume, resource utilization rate and data quality respectively, and the specific values ​​are set by the user.

8. The method of automated data collection and standardization processing according to claim 7, characterized in that: S62, extracting the average value of the data collection performance index and presetting it as the collection performance threshold E, then comparing and evaluating the collection performance threshold E with the obtained comprehensive evaluation index Eval, analyzing the data collection performance, and generating an optimization execution strategy. The specific evaluation content is as follows; When the comprehensive evaluation index Eval> the collection performance threshold E, it means that the collection performance is normal and there is no need to execute the optimization strategy; When the comprehensive evaluation index Eval ≤ the collection performance threshold E, it means that the collection performance is abnormal, and the optimization strategy is executed at this time.

9. The method of automated data collection and standardization processing according to claim 8, characterized in that: S63, when the acquisition performance is abnormal through preliminary evaluation, construct an acquisition optimization formula, input the current comprehensive evaluation index Eval into the acquisition optimization formula, optimize the calculation output optimization index Optim, perform secondary optimization, and then optimize the iterative execution process steps until the output acquisition performance is normal and the iteration is stopped; The optimization index Optim is calculated and output by the following acquisition optimization formula; ； In the formula, Niterations represents the number of iterations, Qi represents the effect of the i-th optimization, and Ri represents the resource consumption of the i-th optimization. Represents the first optimization weight, reflecting the importance of the evaluation results in the optimization process in the optimization decision. Represents the second optimization weight, reflecting the importance of resource consumption in the optimization decision.

10. A system for automated data collection and standardized processing, applied to an automated data collection and standardized processing method according to any one of claims 1 to 9, characterized in that: It includes configuration module, loading module, parameter identification module, process generation module and evaluation and optimization module; The configuration module is used to configure the data collection template to set the collection system information and task information, perform data collection, and store the collected data and collection configuration data in the system database; The loading module is used to simulate user operations to load a web page with dynamic content, parse the interactive interface of the page, and obtain a data field list and a page snapshot; The parameter recognition module is used to perform a box selection operation on the page snapshot, convert the unstructured data of the box selection part into structured data using an OCR tool, and obtain a mapping field description list; The generation processing flow module is used to construct a multi-parameter nonlinear standardization formula, standardize the data field list and the mapping field description list according to the configuration standardization rules, output the standardized processing data Norm, and automatically import the data field list and the mapping field description list through the database mapping tool, and store them in the system database, and generate a data processing flow, and perform a pre-collection test. During the pre-collection test, a multi-dimensional collection frequency optimization formula is used to calculate and output the collection frequency optimization value Fopt; The evaluation and optimization module is used to automatically execute the collection and processing tasks according to the optimized collection frequency optimization value Fopt, generate the entire data processing flow, and after the data processing flow is completed, use the comprehensive analysis algorithm formula to calculate and output the comprehensive evaluation index Eval, and perform a preliminary comparative evaluation on the preset collection performance threshold E and the obtained comprehensive index Eval. Based on the preliminary evaluation results, use the collection optimization formula to calculate the output optimization index Optim for secondary optimization.

Images