A method and system for automated data collection and standardized processing

Abstract

The application discloses a kind of automated data acquisition and standardization processing method and system, it is related to big data processing technical field, the method is prepared by configuring acquisition task information, data acquisition is quickly completed based on preset template, and data and configuration information are stored to database. Simulate user operation to load dynamic page and generate page snapshot, ensure that data acquisition is comprehensive and accurate. In the processing stage, the unstructured data is converted to structured data using OCR, and standardized processing is performed through a multi-parameter nonlinear standardization formula to ensure consistency. The database mapping tool automatically stores the processed data into the database, reducing errors. Then generate data processing flow, output acquisition frequency optimization value Fopt through pre-acquisition test, ensure that the task is executed at the optimal frequency. Calculate evaluation index Eval through evaluation algorithm, and compare with performance threshold E, output optimization index Optim until the acquisition performance returns to normal.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of big data processing, in particular to a method and system for automatic data collection and standardized processing. BACKGROUND

[0002] In the field of big data processing, data collection is the basic link of information processing, which acquires raw data from different sources through various collection tools and technologies. In the specific application of intelligent information management, especially for unstructured and heterogeneous data, the complexity and real-time requirements of data collection are increasing. Therefore, automatic data collection tools not only need to have efficient data acquisition capability, but also need to be able to process data standardization according to the characteristics of different data sources to ensure the consistency and availability of data.

[0003] At present, the existing data collection methods mostly rely on system interface direct connection or manual configuration, which has large development quantity and poor flexibility, and is difficult to meet the frequent changing business system requirements. At the same time, facing the dynamic loading of web page content and unstructured data, the traditional method is often not efficient enough. This limitation leads to low efficiency, incomplete data and security problems in the data collection process, especially in the processing of heterogeneous data sources, the integration and standardization of data need a lot of manual intervention, which cannot meet the rapid response business requirements. SUMMARY

[0004] In view of the deficiencies of the prior art, the present application provides a method and system for automatic data collection and standardized processing, which solves the problems mentioned in the background art.

[0005] To achieve the above purpose, the present application is realized by the following technical scheme: comprising the following steps:

[0006] S1, configuring collection task information: configuring data collection template to set collection system information and task information, collecting data, and storing the collected data and collection configuration data into the system database;

[0007] S2, dynamic page loading and parsing: loading the dynamic content page by simulating user operation, parsing the interactive interface of the page, obtaining the data field list and the page snapshot;

[0008] S3, parameter identification: frame selection operation is performed on the page snapshot, and the unstructured data in the frame selected part is converted into structured data using an OCR tool to obtain a mapping field description list;

[0009] S4, field standardization and mapping: constructing a multi-parameter nonlinear standardization formula, standardizing the data field list and the mapping field description list according to the configured standardization rules, outputting the standardized data Norm, and automatically importing the data field list and the mapping field description list through the database mapping tool and storing them in the system database;

[0010] S5, data processing flow generation and test: according to S1 to S4, generating a data processing flow and performing a pre-collection test, using a multi-dimensional collection frequency optimization formula to calculate and output an optimized collection frequency value Fopt during the pre-collection test;

[0011] S6, task scheduling and automatic execution: according to the optimized collection frequency value Fopt, automatically executing collection and processing tasks to generate the entire data processing flow, using a comprehensive analysis algorithm formula to calculate and output a comprehensive evaluation index Eval after the data processing flow is executed, comparing the preset collection performance threshold E with the obtained comprehensive index Eval for preliminary evaluation, and based on the preliminary evaluation result, using a collection optimization formula to calculate and output an optimization index Optim for secondary optimization.

[0012] Preferably, S1 includes S11, S12 and S13;

[0013] S11, configure collection task information: configure data collection task information including task name, collection frequency and collection mode through a graphical user interface provided by an enterprise system by a configuration personnel, and the configuration personnel sets a unique identifier, automatic execution frequency and collection mode for each task through the image interface;

[0014] The task name is set by the configuration personnel for each task to have a unique name for task management and tracking;

[0015] The collection frequency automatically triggers task execution for periodic execution of automatic data collection by using a scheduling task framework;

[0016] The collection mode sets to collect only incremental data through the CDC technology, the timestamp field and the task name identifier of the system database;

[0017] S12, target system configuration: the name of the data source system is input by the configuration personnel in the configuration interface, and the name of the data source system is saved in the system database. The login information of the target data source system is provided by the configuration personnel during configuration, including the username and password, which are encrypted and stored in the system database using the AES encryption method, and the HTTP request is used to automatically log in to the data source system. The configuration personnel define the data page path of the target data source system, automatically access the data page path of the target data source system through the API interface provided by the data source system, and grab the target data. The collection configuration data is recorded by the running log of the collection task;

[0018] The collection configuration data includes total collection time Tt, collection times Np, resource utilization rate Ru, enterprise system capacity S, error rate Er, and collection data volume Ds.

[0019] S13, database configuration and storage: connect the system database through JDBC interface and ODBC interface, and set the system database location structure of the target data and collection configuration data storage by the configuration personnel, including address, table structure and storage format, and store the target data and collection configuration data in the database according to the set system database location structure, and then perform data encryption processing.

[0020] Preferably, the S2 includes S21 and S22;

[0021] S21, dynamic page loading: after the configuration of the collection task information is completed, WebDriver is used to simulate real user operation, and JavaScript code in the browser is executed when WebDriver loads the page to display asynchronously loaded data. After loading, the username and password in the system database are extracted by WebDriver to automatically fill in the login form and submit;

[0022] S22, network interaction interface analysis: Selenium is used to listen to network traffic and capture the interaction request between the page and the server when the page is loaded, and the network interaction interface list is obtained by analyzing the page. The data interface is selected, the data field in the data interface is analyzed by the system, and the data field list is generated. After the page is loaded, the built-in screenshot function provided by Selenium is used to automatically generate the screenshot system of the page, and the page snapshot is obtained.

[0023] Preferably, S3, parameter identification: the data in the non-standard text and image in the page snapshot is recognized by using optical character recognition technology OCR, and the non-structured data is converted into structured data by using OCR tool to extract the text information in the page snapshot, and the mapping field description list is obtained.

[0024] Preferably, the S4 comprises S41 and S42;

[0025] S41, standardization configuration: according to the user configuring the standardization rule in the graphical interface, the standardization rule comprises unit unification, nonlinear scaling and noise filtering, a multi-parameter nonlinear standardization formula is constructed, and output standardized processing data Norm is calculated;

[0026] The standardized processing data Norm is calculated by the following multi-parameter nonlinear standardization formula;

[0027] ;

[0028] In the formula, x represents original data, that is, the numerical value in the data field list and the mapping field description list which needs to be standardized, represents the average value of the original data x, represents the standard deviation of the original data x, log represents a logarithmic function, represents a first smoothing factor, which is used to control the scaling effect on the original data x, represents a second smoothing factor, which is used as a protection item to prevent the logarithmic operation from being wrong;

[0029] S42, data mapping: according to the obtained data field list, field description list and data standard three configurations, data mapping is completed one by one according to the data field list and the mapping field description list through a database mapping tool, and is automatically imported into a system database table.

[0030] Preferably, the S5 comprises S51 and S52;

[0031] S51, flow generation: according to the configured configuration collection task information, dynamic page loading and analysis, parameter identification and field standardization and mapping, a workflow engine is used to automatically manage and execute the configuration collection task information to the field standardization and mapping, and a scheduling engine is used to set a timing execution;

[0032] S52, pre-collection test and optimization: after the flow generation, a pre-collection test is automatically executed, a multi-dimensional collection frequency optimization formula is used in the pre-collection test process, and collection configuration data is input, and a collection frequency optimization value Fopt is calculated and output;

[0033] The collection frequency optimization value Fopt is calculated by the following multi-dimensional collection frequency optimization formula;

[0034] ;

[0035] In the formula, argmin represents the independent variable value that makes the target function reach the lower limit, Fc represents the collection frequency, represents the collection efficiency, represents resource usage rate, represents data quality and error rate, a1, a2 and a3 respectively represent preset weight values of collection efficiency, resource usage rate and data quality and error rate, and the specific values thereof are set by a user.

[0036] Preferably, the S6 comprises S61, S62 and S63.

[0037] S61, after pre-collection testing and optimization, automatically performs collection and processing tasks, completes the entire data processing flow, and constructs a comprehensive analysis algorithm formula to calculate an output comprehensive evaluation index Eval;

[0038] The comprehensive evaluation index Eval is calculated by the following comprehensive analysis algorithm formula:

[0039] ;

[0040] In the formula, represents the total amount of data collected within the collection time, Tc represents the collection time, Qdata represents the data quality, Sreq represents the resource demand amount, b1, b2 and b3 respectively represent preset weight values of the effective data amount, resource usage rate and data quality, and the specific values thereof are set by a user.

[0041] Preferably, S62, the average value of the data collection performance index is preset as a collection performance threshold E, and the collection performance threshold E is compared with the obtained comprehensive evaluation index Eval to evaluate the data collection performance and generate an optimization execution strategy, and the specific evaluation content is as follows:

[0042] When the comprehensive evaluation index Eval is greater than the collection performance threshold E, it indicates that the collection performance is normal, and no optimization strategy needs to be executed.

[0043] When the comprehensive evaluation index Eval is less than or equal to the collection performance threshold E, it indicates that the collection performance is abnormal, and the optimization strategy is executed at this time.

[0044] Preferably, S63, when the collection performance is preliminarily evaluated as abnormal, a collection optimization formula is constructed, the current comprehensive evaluation index Eval is input into the collection optimization formula, an optimization index Optim is calculated and output by optimization, secondary optimization is performed, and the iteration execution flow steps are repeatedly executed until the collection performance is normal and the iteration is stopped.

[0045] The optimization index Optim is calculated and output by the following collection optimization formula:

[0046] ;

[0047] In the formula, Niterations represents the number of iterations, Qi represents the effect of the i-th optimization, Ri represents the resource consumption of the i-th optimization, represents the first optimization weight, reflecting the importance of the evaluation result in the optimization process in the optimization decision, represents the second optimization weight, reflecting the importance of the resource consumption in the optimization decision.

[0048] An automated data acquisition and standardized processing system comprises a configuration module, a loading module, a parameter identification module, a generation processing flow module and an evaluation optimization module.

[0049] The configuration module is configured to configure a data acquisition template to set acquisition system information and task information, perform data acquisition, and store the set acquired data and acquisition configuration data into a system database.

[0050] The loading module is configured to simulate user operation to load a dynamic content webpage, parse the interactive interface of the webpage, obtain a data field list and a webpage snapshot.

[0051] The parameter identification module is configured to frame the webpage snapshot, convert the unstructured data of the framed part into structured data using an OCR tool, and obtain a mapping field description list.

[0052] The generation processing flow module is configured to construct a multi-parameter nonlinear standardization formula, standardize the data field list and the mapping field description list according to a configuration standardization rule, output standardized processing data Norm, automatically import the data field list and the mapping field description list through a database mapping tool, store them in the system database, generate a data processing flow, and perform a pre-acquisition test.

[0053] The evaluation optimization module is configured to automatically perform acquisition and processing tasks according to the optimized acquisition frequency optimization value Fopt, generate an entire data processing flow, calculate an output comprehensive evaluation index Eval using a comprehensive analysis algorithm formula after the data processing flow is executed, preliminarily compare and evaluate the preset acquisition performance threshold E and the obtained comprehensive index Eval, and based on the preliminary evaluation result, calculate an output optimization index Optim using an acquisition optimization formula, and perform secondary optimization.

[0054] The present application provides an automated data acquisition and standardized processing method and system. The present application has the following advantages:

[0055] (1) The method realizes an automatic data collection process by configuring collection task information and dynamically loading and parsing pages. The WebDriver is used to simulate user operations, load pages with dynamic content, and combine with network interaction interface parsing technology to obtain data field lists and page snapshots, ensuring the accuracy and real-time performance of data collection. At the same time, the system automatically stores the collected data and configuration information in the system database, avoiding omissions and errors in the traditional manual collection process and improving collection efficiency.

[0056] (2) In the parameter identification stage, the method converts unstructured page snapshot data into structured data through OCR tools to obtain a mapping field description list. Then, a multi-parameter nonlinear standardization formula S4 is constructed, and the data fields are standardized according to the user's configured standardization rules to generate standardized data Norm. This process not only automatically completes data conversion and standardization, but also ensures data consistency and compatibility, reducing the workload of manual intervention and data cleaning.

[0057] (3) The method generates a data processing flow and performs pre-collection testing, calculates the collection frequency optimization value Fopt using a multi-dimensional collection frequency optimization formula, and ensures the balance between data collection frequency and system resource usage. In the task scheduling and automatic execution process, the system calculates the comprehensive evaluation index Eval through a comprehensive evaluation algorithm formula, compares it with the preset collection performance threshold E, identifies collection performance problems in a timely manner, and performs secondary optimization through a collection optimization formula to calculate the output optimization index Optim. This intelligent optimization process effectively reduces resource consumption and improves the overall performance of data collection and processing. BRIEF DESCRIPTION OF DRAWINGS

[0058] Figure 1 is a flowchart of the method of the present application for automatic data collection and standardization processing;

[0059] Figure 2 is a system step diagram of the present application for automatic data collection and standardization processing. DETAILED DESCRIPTION

[0060] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor are within the scope of protection of the present application.

[0061] Example 1

[0062] Please refer to Figure 1The application provides an automatic data collection and standardized processing method, and achieves the above object through the following technical scheme.

[0063] S1, configure collection task information: configure data collection template to set collection system information and task information, perform data collection, and store the set collected data and collection configuration data into a system database;

[0064] S2, dynamic page loading and parsing: load a webpage with dynamic content by simulating user operation, parse the interactive interface of the webpage, obtain a data field list and a page snapshot;

[0065] S3, parameter identification: frame the page snapshot, convert the unstructured data of the framed part into structured data using an OCR tool, and obtain a mapping field description list;

[0066] S4, field standardization and mapping: construct a multi-parameter nonlinear standardization formula, standardize the data field list and the mapping field description list according to a configuration standardization rule, output standardized processing data Norm, and automatically import the data field list and the mapping field description list into the system database through a database mapping tool;

[0067] S5, data processing flow generation and test: generate a data processing flow according to S1 to S4, and perform pre-collection test, use a multi-dimensional collection frequency optimization formula to calculate and output an optimized collection frequency value Fopt during the pre-collection test;

[0068] S6, task scheduling and automatic execution: automatically execute collection and processing tasks according to the optimized collection frequency value Fopt, generate the entire data processing flow, use a comprehensive analysis algorithm formula to calculate and output a comprehensive evaluation index Eval after the data processing flow is executed, preliminarily compare and evaluate the preset collection performance threshold E and the obtained comprehensive index Eval, and based on the preliminary evaluation result, use a collection optimization formula to calculate and output an optimization index Optim for secondary optimization.

[0069] In this embodiment, the method configures the collection task information, and the system can complete the preparation work of data collection quickly according to the pre-set template and task configuration, and store the collected raw data and configuration information into the database, thereby laying a foundation for subsequent processing. Then, through dynamic page loading and analysis, the system can simulate user operation, load dynamic web page content, analyze data fields in the page and generate a page snapshot, so as to ensure that the collected data is comprehensive and accurate. In the data processing stage, the system uses the OCR tool to convert the unstructured data in the page snapshot into structured data through parameter identification, generates a mapping field description list, and solves the problem of processing unstructured data. Subsequently, the field standardization and mapping are used, the data is standardized by using a multi-parameter nonlinear standardization formula, and the standardized data Norm is output. At the same time, the database mapping tool can automatically import the processed data into the system database, so as to ensure the consistency and manageability of the data and reduce the error risk in manual processing. Finally, through the data processing flow generation and test and task scheduling and automatic execution, the system generates a complete processing flow according to the previous configuration, and outputs the collection frequency optimization value Fopt in the pre-collection test, so as to ensure that the task is executed at the optimal collection frequency. After the completion of the entire data processing flow, the system calculates the evaluation index Eval through the comprehensive analysis algorithm, compares it with the pre-set collection performance threshold E, judges whether further optimization is needed, outputs the optimization index Optim, and optimizes and iterates the execution process steps until the collection performance is normal and the iteration is stopped. Compared with the traditional technology, the system effectively improves the automation degree of data collection and processing, significantly optimizes the resource utilization rate and data processing performance, reduces human intervention, and improves the overall efficiency and accuracy.

[0070] Embodiment 2

[0071] This embodiment is an explanation and description in Embodiment 1, please refer to Figure 1 , specifically: S1 includes S11, S12 and S13;

[0072] S11, configure collection task information: through the graphical user interface provided by the enterprise system, configure the data collection task information by the configuration personnel, including task name, collection frequency and collection mode, and the configuration personnel sets the unique identification, automatic execution frequency and collection mode of each task through the image interface;

[0073] The task name is set by the configuration personnel for each task, and is used for task management and tracking;

[0074] The collection frequency is automatically triggered by using the scheduling task framework to execute the task to execute the automatic collection data periodically;

[0075] The collection mode is identified by the CDC technology, the timestamp field and the task name of the system database, and is set to collect only incremental data

[0076] In the enterprise system interface, the configuration personnel sets the task name for each data collection task by configuring the data collection template, which is convenient for finding in the log, viewing in the task management and identifying in the debugging process. The uniqueness of the task name is verified by the displacement key UniqueKey in the system database, and when the task name conflicts, a prompt is popped up to let the configuration personnel change the task name. At the same time, the configuration personnel sets the collection frequency in the task configuration interface of the enterprise system interface, including every hour, every day and every week, and uses the scheduling task framework to trigger the task execution at a fixed time. The scheduling task framework automatically executes the collection task based on the predefined time rules, and then uses the CDC technology, the timestamp field and the task name of the system database to set to collect only incremental data.

[0077] S12, target system configuration: the configuration personnel inputs the name of the data source system in the configuration interface, saves the name of the data source system in the system database, and can quickly find the related collection task when generating logs or debugging. The configuration personnel provides the login information of the target data source system when configuring, including the username and password, and at the same time, the username and password are encrypted and stored in the system database using the AES encryption method, and the HTTP request is used to automatically log in to the data source system. The configuration personnel defines the data page path of the target data source system, automatically accesses the data page path of the target data source system through the API interface provided by the data source system to grab the target data, and records the collection configuration data through the running log of the collection task.

[0078] The collection configuration data includes the total collection time Tt, the collection times Np, the resource utilization rate Ru, the enterprise system capacity S, the error rate Er and the collection data volume Ds.

[0079] S13, database configuration and storage: connect the system database through the JDBC interface and the ODBC interface, and the configuration personnel sets the system database location structure of the target data and the collection configuration data storage, including the address, the table structure and the storage format, and stores the target data and the collection configuration data in the database according to the set system database location structure, and then performs data encryption processing.

[0080] In this embodiment, the configuration of the method simplifies the configuration process through the graphical user interface, automatically sets the task name, collection frequency and collection mode, realizes automatic and periodic incremental data collection, reduces the complexity of manual management, and improves the accuracy and efficiency of data collection. The target system configuration is encrypted by logging in and automatically accessing the target data source system, ensuring the security of data transmission, and automatically grabbing the target data through the API interface, reducing the error risk caused by manual operation. Finally, the database configuration and storage connect the system database through the JDBC interface and the ODBC interface, ensure that the collected data are stored safely and accurately according to the set database structure, and the data are encrypted, further improving the security of the data.

[0081] Embodiment 3

[0082] This embodiment is an explanation and description in embodiment 2, please refer to Figure 1 , specifically: S2 includes S21 and S22;

[0083] S21, dynamic page loading: after the configuration of the collection task information is completed, WebDriver is used to simulate real user operation, and at the same time, JavaScript code in the browser is executed when WebDriver loads the page, and the asynchronously loaded data is displayed. The asynchronously loaded data includes table or chart data obtained from the server through Ajax request, the system ensures that all dynamic data has been loaded by waiting for the page to be loaded, and after the loading is completed, the username and password in the system database are extracted through WebDriver, the login form is automatically filled and submitted;

[0084] S22, network interaction interface analysis: Selenium is used to listen to network traffic and capture the interaction request between the page and the server through the browser's developer tools when the page is loaded, and the network interaction interface list is obtained by analyzing the page, and the data interface is selected. The system analyzes the data fields in the data interface and generates a data field list, and after the page is loaded, the built-in screenshot function provided by Selenium is used to automatically generate a screenshot of the page system, and a page snapshot is obtained.

[0085] S3, parameter identification: the data in the non-standard text and images in the page snapshot are identified by using the optical character recognition technology OCR, and the non-structured data is converted into structured data by using the OCR tool. The text information in the page snapshot is extracted, and a mapping field description list is obtained.

[0086] In this embodiment, the method simulates the operation of a real user through dynamic page loading using WebDriver, and automatically executes JavaScript code to asynchronously load page data, ensuring that dynamic content such as tables and chart data can be fully rendered and collected. In addition, the system can automatically complete the login operation by extracting the username and password in the database, thereby automating the entire data acquisition process. The network interaction interface parsing uses Selenium to listen to the interaction requests between the web page and the server, parses the valid data fields and generates a field list, and generates a page snapshot through the screenshot function to ensure data traceability and reproducibility. On this basis, parameter identification uses OCR technology to identify non-standard text and image data in the page snapshot, automatically converting unstructured data into structured data. The system can intelligently extract page text information and generate a mapping field description list, simplifying the process of handling unstructured data.

[0087] Embodiment 4

[0088] This embodiment is an explanation and description in Embodiment 3, please refer to Figure 1 , specifically: S4 includes S41 and S42;

[0089] S41, standardization configuration: according to the user's configuration of standardization rules in the graphical interface, the standardization rules include unit unification, nonlinear scaling and noise filtering, a multi-parameter nonlinear standardization formula is constructed, and the output standardization processing data Norm is calculated;

[0090] The standardization processing data Norm is calculated by the following multi-parameter nonlinear standardization formula;

[0091] ;

[0092] In the formula, x represents the original data, i.e. the numerical value in the data field list and the mapping field description list that needs to be standardized, represents the mean value of the original data x, the mean value is used for centering the data, by subtracting the mean value, the data is adjusted to a range around zero, so that the standardized data is not affected by the absolute value size, represents the standard deviation of the original data x, the standard deviation is used for scale adjustment of the data, by dividing by the standard deviation, the distribution of the data is standardized, so that the data of different characteristics can be compared uniformly in processing, log represents the logarithmic function, represents the first smoothing factor, which is used to control the scaling effect of the original data x, represents the second smoothing factor, which is used as a protection item to prevent the log operation from being wrong, specifically x takes zero or negative value;

[0093] S42, data mapping: according to the obtained data field list, field description list and data standard configuration, data mapping is completed one by one through the database mapping tool according to the data field list and the mapping field description list, and is automatically imported into the system database table.

[0094] In this embodiment, the method allows users to set standardization rules through a graphical interface, including unit conversion, nonlinear scaling and noise filtering. The system constructs a multi-parameter nonlinear standardization formula according to these rules to generate standardized data Norm. The formula adjusts the data around zero by mean and standard deviation, ensuring that data of different features are processed on a unified scale. This standardization not only improves the consistency of data, but also greatly reduces the data bias in subsequent analysis, ensuring that data from different sources can be uniformly processed and compared. Data mapping automatically imports the standardized data into the system database through the database mapping tool. The system maps one by one according to the configured data field list and mapping field description list, ensuring that the data is stored in the pre-defined format, and improving the automation level of data management.

[0095] Embodiment 5

[0096] This embodiment is an explanation and description in embodiment 4, please refer to Figure 1 , specifically: S5 includes S51 and S52;

[0097] S51, flow generation: according to the configured configuration collection task information, dynamic page loading and parsing, parameter identification and field standardization and mapping, use workflow engine to automatically manage and execute the configuration collection task information to field standardization and mapping, and set timing execution through scheduling engine;

[0098] S52, pre-collection test and optimization: automatically execute pre-collection test after flow generation, use multi-dimensional collection frequency optimization formula during pre-collection test, and input collection configuration data to calculate output collection frequency optimization value Fopt;

[0099] The collection frequency optimization value Fopt is calculated and obtained through the following multi-dimensional collection frequency optimization formula;

[0100] ;

[0101] In the formula, argmin represents the independent variable value that makes the objective function reach the lower limit, Fc represents the collection frequency, represents the collection efficiency, represents the resource utilization rate, Data quality and error rate, a1, a2 and a3 represent preset weight values of collection efficiency, resource usage and data quality and error rate respectively, and the specific values are set by the user.

[0102] In this embodiment, the method is generated by a process. The system can automatically manage and execute the entire data collection and standardization process by using a workflow engine according to the configured collection task information, dynamic page loading, parameter identification and field standardization steps. This process is set to execute at a specified time through a scheduling engine, ensuring the continuity and stability of data collection and processing without human intervention, greatly improving work efficiency. Pre-collection testing and optimization Through pre-collection testing, the efficiency and resource usage of the collection process are evaluated. The system uses a multi-dimensional collection frequency optimization formula to calculate and output the optimal collection frequency optimization value Fopt by combining the collection configuration data. This optimization value considers factors such as collection frequency, collection efficiency, resource usage and data quality to ensure the highest efficiency and quality of data collection with the least resource consumption.

[0103] Embodiment 6

[0104] This embodiment is an explanation and description in Embodiment 1. Please refer to Figure 1 , specifically: S6 includes S61, S62 and S63;

[0105] S61, after pre-collection testing and optimization, automatically execute collection and processing tasks, complete the entire data processing process, and build a comprehensive analysis algorithm formula to calculate and output the comprehensive evaluation index Eval;

[0106] The comprehensive evaluation index Eval is calculated by the following comprehensive analysis algorithm formula:

[0107] ;

[0108] In the formula, represents the total amount of data collected within the collection time, Tc represents the collection time, Qdata represents the data quality, Sreq represents the resource demand, b1, b2 and b3 represent the preset weight values of effective data volume, resource usage and data quality respectively, and the specific values are set by the user.

[0109] S62, the average value of the data collection performance index is extracted and preset as the collection performance threshold E. Then, the collection performance threshold E is compared and evaluated with the obtained comprehensive evaluation index Eval to analyze the data collection performance and generate an optimization execution strategy. The specific evaluation content is as follows:

[0110] When the comprehensive evaluation index Eval is greater than the collection performance threshold E, it indicates that the collection performance is normal and no optimization strategy needs to be executed;

[0111] When the comprehensive evaluation index Eval is less than or equal to the collection performance threshold E, it indicates that the collection performance is abnormal, and at this time, the optimization strategy is executed.

[0112] S63, when the preliminary evaluation indicates that the collection performance is abnormal, a collection optimization formula is constructed, the current comprehensive evaluation index Eval is input into the collection optimization formula, the optimization calculation outputs an optimization index Optim, secondary optimization is performed, and the optimization iteration execution flow is performed until the output collection performance is normal and the iteration is stopped.

[0113] The optimization index Optim is calculated and output by the following collection optimization formula:

[0114]

[0115] In the formula, Niterations represents the number of iterations, Qi represents the effect of the i-th optimization, Ri represents the resource consumption of the i-th optimization, represents the first optimization weight, reflecting the importance of the evaluation result in the optimization process in the optimization decision, represents the second optimization weight, reflecting the importance of resource consumption in the optimization decision.

[0116] In this embodiment, the method calculates the comprehensive evaluation index Eval by a comprehensive analysis algorithm formula, quantitatively evaluates multiple dimensions such as the total data volume, collection time, data quality and resource demand involved in the data collection process, and ensures comprehensive monitoring of the overall performance of the collection task. Then, by comparing the comprehensive evaluation index with the preset collection performance threshold E, it is judged whether the performance of the collection task meets the standard. When the evaluation index is lower than the threshold, the system automatically generates an optimization strategy to ensure that the collection task is corrected in time when the performance is abnormal. Further, in the case of abnormal performance, secondary optimization is performed using the collection optimization formula, and the system calculates the optimization index Optim through multiple iteration optimization, gradually improves the collection performance, and until the performance returns to the normal level. The optimization process considers the trade-off between collection effect and resource consumption, ensuring that the collection quality is improved while minimizing resource waste.

[0117] Embodiment 7

[0118] Please refer to Figure 2 An automatic data collection and standardized processing system, comprising a configuration module, a loading module, a parameter identification module, a generation processing flow module and an evaluation optimization module.

[0119] The configuration module is used to configure the data collection template to set the collection system information and task information, perform data collection, and store the set collected data and collection configuration data into the system database.

[0120] ​The loading module is used for simulating user operation of loading a webpage of dynamic content, parsing an interactive interface of the webpage, obtaining a data field list and a webpage snapshot;

[0121] The parameter identification module is used for frame selection operation on the webpage snapshot, converting unstructured data of the frame selected part into structured data by using an OCR tool, and obtaining a mapping field description list;

[0122] The generation processing flow module is used for constructing a multi-parameter nonlinear standardization formula, performing standardization processing on the data field list and the mapping field description list according to a configuration standardization rule, outputting standardization processing data Norm, automatically importing the data field list and the mapping field description list by using a database mapping tool, storing the data field list and the mapping field description list in a system database, generating a data processing flow, and performing pre-collection testing, wherein a multi-dimensional collection frequency optimization formula is used in the pre-collection testing to calculate and output a collection frequency optimization value Fopt;

[0123] The evaluation optimization module is used for automatically performing collection and processing tasks according to the optimized collection frequency optimization value Fopt, generating an entire data processing flow, calculating and outputting a comprehensive evaluation index Eval by using a comprehensive analysis algorithm formula after the data processing flow is executed, preliminarily comparing and evaluating the preset collection performance threshold E with the obtained comprehensive index Eval, and calculating and outputting an optimization index Optim by using a collection optimization formula based on the preliminary evaluation result to perform secondary optimization.

[0124] Although the embodiments of the present application have been shown and described, it can be understood by those skilled in the art that various changes, modifications, replacements and variations can be made to the embodiments without departing from the principles and spirits of the present application.

Claims

1. A method for automated data collection and standardized processing, characterized by: The method comprises the following steps: S1, configuring acquisition task information: configuring data acquisition template to set acquisition system information and task information, performing data acquisition, and storing the set acquired data and acquisition configuration data into a system database; S2, dynamic page loading and parsing: loading a dynamic content webpage by simulating user operation, parsing the interactive interface of the page, obtaining a data field list and a page snapshot; S3, parameter identification: performing a frame selection operation on the page snapshot, converting the unstructured data of the frame selected part into structured data using an OCR tool, and obtaining a mapping field description list; S4, field standardization and mapping: constructing a multi-parameter nonlinear standardization formula, standardizing the data field list and the mapping field description list according to a configured standardization rule, outputting standardized processing data Norm, and automatically importing the data field list and the mapping field description list into the system database through a database mapping tool and storing them in the system database; S5, data processing flow generation and testing: generating a data processing flow according to S1 to S4, and performing pre-acquisition testing, wherein a multi-dimensional acquisition frequency optimization formula is used to calculate and output an acquisition frequency optimization value Fopt during the pre-acquisition testing; S6, task scheduling and automatic execution: automatically executing acquisition and processing tasks according to the optimized acquisition frequency optimization value Fopt, generating an entire data processing flow, using a comprehensive analysis algorithm formula to calculate and output a comprehensive evaluation index Eval after the data processing flow is executed, preliminarily comparing and evaluating the preset acquisition performance threshold E and the obtained comprehensive index Eval, and based on the preliminary evaluation result, using an acquisition optimization formula to calculate and output an optimization index Optim for secondary optimization.

2. The method of claim 1, wherein: The S1 comprises S11, S12 and S13; S11, configuring acquisition task information: a configuration personnel performs configuration data acquisition task information through a graphical user interface provided by an enterprise system, including task name, acquisition frequency and acquisition mode, and the configuration personnel further sets a unique identifier, automatic execution frequency and acquisition mode of each task through the image interface; The task name is set by the configuration personnel for each task to have a unique name for task management and tracking; The acquisition frequency is automatically triggered to execute periodic automatic acquisition data by using a scheduling task framework; The acquisition mode is set to only acquire incremental data by using CDC technology, a timestamp field and a task name identifier of the system database; S12, target system configuration: through the configuration personnel in the configuration interface input data source system name, save the data source system name in the system database, the configuration personnel in the configuration provides the login information of the target data source system, the login information includes the username and password, at the same time, the username and password are encrypted and stored in the system database using AES encryption method, and the HTTP request is used to automatically log in the data source system, the configuration personnel define the data page path of the target data source system, through the API interface provided by the data source system, the data page path of the target data source system is automatically accessed to grab the target data, and the collection configuration data is recorded through the running log of the collection task; The collection configuration data includes total collection time Tt, collection times Np, resource utilization rate Ru, enterprise system capacity S, error rate Er and collection data volume Ds; S13, database configuration and storage: connect the system database through JDBC interface and ODBC interface, configure personnel set the system database location structure of target data and collection configuration data storage, the system database location structure includes address, table structure and storage format, and store the target data and collection configuration data into the database according to the set system database location structure, and then perform data encryption processing.

3. The method of claim 1, wherein: The S2 includes S21 and S22; S21, dynamic page loading: after the configuration of the collection task information is completed, the WebDriver is used to simulate the real user operation, and at the same time, the JavaScript code in the browser is executed when the WebDriver loads the page, the asynchronously loaded data is displayed, and after the loading is completed, the username and password in the system database are extracted through the WebDriver, the login form is automatically filled and submitted; S22, network interaction interface analysis: when the page is loaded, the Selenium is used to listen to the network traffic combined with the browser developer tool, and the interaction request between the page and the server is grabbed, the network interaction interface list is obtained by analyzing the page, and the data interface is selected, the system analyzes the data field in the data interface and generates the data field list, and after the page is loaded, the built-in screenshot function provided by the Selenium is used to automatically generate the screenshot system of the page, and the page snapshot is obtained.

4. The method of claim 1, wherein: S3, parameter identification: the data in the non-standard text and image in the page snapshot is identified by using the optical character recognition technology OCR, the non-structured data is converted into structured data by using the OCR tool, the text information in the page snapshot is extracted, and the mapping field description list is obtained.

5. The method of claim 1, wherein: The S4 includes S41 and S42; S41, standardization configuration: according to the user in the graphical interface configuration standardization rule, the standardization rule includes unit unification, nonlinear scaling and noise filtering, a multi-parameter nonlinear standardization formula is constructed, and the output standardization processing data Norm is calculated; The standardization processing data Norm is calculated by the following multi-parameter nonlinear standardization formula; ； In the formula, x represents original data, i.e. the values in the data field list and the mapping field description list to be normalized, represents the average of the original data x, represents the standard deviation of the original data x, and log represents a logarithmic function, represents a first smoothing factor for controlling the scaling effect on the original data x, represents a second smoothing factor for preventing the log operation from error; S42, data mapping: according to the obtained data field list, field description list and data standard configuration, the data mapping is completed one by one through the database mapping tool according to the data field list and the mapping field description list, and is automatically imported into the system database table.

6. The method of claim 2, wherein: The S5 includes S51 and S52; S51, flow generation: according to the configured configuration collection task information, dynamic page loading and parsing, parameter identification and field standardization and mapping, the workflow engine is used to automatically manage and execute the configuration collection task information to the field standardization and mapping, and the scheduling engine is set to execute at a fixed time; S52, pre-collection test and optimization: after the flow generation, the pre-collection test is automatically executed, the multi-dimensional collection frequency optimization formula is used in the pre-collection test process, and the collection configuration data is input, and the optimized collection frequency value Fopt is calculated and output; The collection frequency optimization value Fopt is calculated and obtained through the following multi-dimensional collection frequency optimization formula; ； wherein argmin represents the argument value that makes the objective function reach the lower limit, Fc represents the collection frequency, represents the collection efficiency, represents the resource usage rate, represents the data quality and error rate, and a1, a2 and a3 respectively represent preset weight values of the collection efficiency, the resource usage rate and the data quality and error rate, the specific numerical values of which are set by the user.

7. The method of claim 2, wherein: The S6 includes S61, S62 and S63; S61, after the pre-collection test and optimization, the collection and processing task is automatically executed, the whole data processing flow is completed, the comprehensive analysis algorithm formula is constructed, and the comprehensive evaluation index Eval is calculated and output; The comprehensive evaluation index Eval is calculated and obtained through the following comprehensive analysis algorithm formula; ； In the formula, represents the total amount of data collected within the collection time, Tc represents the collection time, Qdata represents the data quality, Sreq represents the resource requirement, b1, b2, and b3 represent preset weight values of the effective data amount, the resource usage rate, and the data quality, respectively, and the specific values thereof are set by the user.

8. The method of claim 7, wherein: S62, the average value of the data collection performance index is extracted and preset as the collection performance threshold E, and the collection performance threshold E is compared and evaluated with the obtained comprehensive evaluation index Eval, the data collection performance is analyzed, and the optimization execution strategy is generated, and the specific evaluation content is as follows: When the comprehensive evaluation index Eval is greater than the collection performance threshold E, it indicates that the collection performance is normal, and the optimization strategy does not need to be executed; When the comprehensive evaluation index Eval is less than or equal to the collection performance threshold E, it indicates that the collection performance is abnormal, and the optimization strategy is executed.

9. The method of claim 8, wherein: S63, when the collection performance is preliminarily evaluated as abnormal, the collection optimization formula is constructed, the current comprehensive evaluation index Eval is input into the collection optimization formula, the optimization index Optim is calculated and output, the secondary optimization is performed, and the iteration execution process is performed until the collection performance is normal and the iteration is stopped; The optimization index Optim is calculated and output through the following collection optimization formula; ； where Niterations represents the number of iterations, Qi represents the effect of the i-th optimization, Ri represents the resource consumption of the i-th optimization, represents the first optimization weight, reflecting the importance of the evaluation result in the optimization process in the optimization decision, represents the second optimization weight, reflecting the importance of the resource consumption in the optimization decision.

10. A system for automated data collection and standardization, applied to the method of any one of claims 1-9, characterized in that: It includes a configuration module, a loading module, a parameter identification module, a generation processing flow module and an evaluation optimization module. The configuration module is used for configuring the data collection template to set the collection system information and task information, collecting data, and storing the set collected data and collection configuration data into the system database. The loading module is used for simulating user operation to load the web page of dynamic content, parsing the interactive interface of the page, obtaining the data field list and the page snapshot; The parameter identification module is used for frame selection operation on the page snapshot, using the OCR tool to convert the non-structured data of the frame selected part into structured data, and obtaining the mapping field description list; The generation processing flow module is configured to construct a multi-parameter nonlinear normalization formula, perform normalization processing on the data field list and the mapping field description list according to a configuration normalization rule, output normalized processing data Norm, automatically import the data field list and the mapping field description list through a database mapping tool, store the data field list and the mapping field description list in a system database, generate a data processing flow, and perform a pre-acquisition test, wherein a multi-dimensional acquisition frequency optimization formula is used in the pre-acquisition test to calculate and output an acquisition frequency optimization value Fopt; The evaluation optimization module is configured to automatically perform acquisition and processing tasks according to the optimized acquisition frequency optimization value Fopt, generate an entire data processing flow, use a comprehensive analysis algorithm formula to calculate and output a comprehensive evaluation index Eval after the data processing flow is executed, preliminarily compare and evaluate the preset acquisition performance threshold E and the obtained comprehensive index Eval, and use an acquisition optimization formula to calculate and output an optimization index Optim based on a preliminary evaluation result for secondary optimization.

Images