Automatic data acquisition method and device, equipment and storage medium

By obtaining the parameter configuration information and data acquisition scheme input by the user, crawling and preprocessing the browser data and inputting it into a large model for integrated processing, the complexity of data acquisition in the existing technology is solved, and the intelligence and efficiency of data acquisition are improved.

CN119025736APending Publication Date: 2024-11-26COSMO INSTITUTE OF INDUSTRIAL INTELLIGENCE (QINGDAO) CO LTD +1
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202411120035.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-15
Publication Date
2024-11-26

AI Technical Summary

Technical Problem

The data acquisition process in the prior art is complex, requiring deep programming skills and in-depth understanding of the data source structure, request and response mechanism, which increases coding difficulty and complexity.

Method used

By obtaining the parameter configuration information entered by the user, including ID, URL and configuration selector, combining the data acquisition plan to determine the data acquisition identifier, grab the original data of the browser and preprocess it, and finally input the data into the big model for integration and processing, achieving efficient Flexible data collection.

Benefits of technology

It improves the intelligence and efficiency of data acquisition, reduces the coding complexity, and achieves efficient and flexible data acquisition.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119025736A_ABST
    Figure CN119025736A_ABST
Patent Text Reader

Abstract

The invention belongs to the field of industrial internet, and particularly relates to an automatic data collection method, device and equipment and a storage medium, parameter configuration information input by a user is obtained, the parameter configuration information comprises an ID, a URL and a configuration selector, and a data collection scheme is obtained according to the ID, the URL, the configuration selector and the data collection scheme. Determining a data acquisition identifier corresponding to the parameter configuration information input by the user, capturing original data in a browser according to the data acquisition identifier corresponding to the parameter configuration information input by the user, and preprocessing the original data to obtain preprocessed data, inputting the preprocessed data into the large model for integration processing to obtain collected data; according to the method, efficient and flexible data acquisition is realized, and the intelligent degree of data acquisition is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application belongs to the field of industrial Internet, and specifically relates to an automated data collection method, device, equipment and storage medium. Background Art

[0002] In the context of the big data era, the demand for data collection is growing. This is mainly because the development of the Internet and information technology has led to an explosive growth in the amount of information in various data sources (such as web pages, API interfaces, and databases). The desire of enterprises, research institutions, and individuals for accurate and real-time data has promoted the development of automated data collection technology.

[0003] In the prior art, data collection by coding usually involves writing automated scripts or programs that can send requests to target data sources (such as web pages, API interfaces, databases, etc.) and parse the response content to extract the required data. These processes may include simulating browser behavior, executing HTTP requests, processing data formats (such as HTML, JSON, XML, etc.), and storing the extracted data in local files or databases.

[0004] However, although data collection through coding provides high flexibility and powerful functions, it also faces the problem of code writing complexity. Writing automated scripts or programs to collect data requires not only deep programming skills, but also a deep understanding of the structure of the target data source, the request and response mechanism, and the processing of data formats, which increases the difficulty and complexity of coding. Summary of the invention

[0005] The present application provides an automated data collection method, apparatus, device and storage medium to solve the problems of difficulty and high threshold in implementing the technology used in data collection.

[0006] In a first aspect, the present application provides an automated data collection method, comprising:

[0007] Obtain parameter configuration information input by the user, the parameter configuration information including: ID, URL and configuration selector;

[0008] Determine the data collection identifier corresponding to the parameter configuration information input by the user according to the ID, the URL, the configuration selector and the data collection scheme;

[0009] According to the data collection identifier corresponding to the parameter configuration information input by the user, the original data in the browser is captured, and the original data is preprocessed to obtain preprocessed data;

[0010] The preprocessed data is input into the large model for integration and processing to obtain the collected data.

[0011] Optionally, before obtaining the parameter configuration information input by the user, the method further includes:

[0012] Get ID information, URL information, and configuration selector information;

[0013] Generate a data collection identifier according to the ID information, the URL information and the configuration selector information;

[0014] The ID information, the URL information, the configuration selector information and the data collection identifier are associated to construct a data collection plan.

[0015] Optionally, capturing raw data in the browser according to the data collection identifier corresponding to the parameter configuration information input by the user includes:

[0016] Generate request information according to the data collection identifier corresponding to the parameter configuration information input by the user;

[0017] The request information is sent to the browser, the original data in the browser is captured, and the original data is saved and exported.

[0018] Optionally, preprocessing the original data to obtain preprocessed data includes:

[0019] The raw data is cleaned, converted, normalized or standardized to remove noise, fill missing values, and convert data types to obtain preprocessed data.

[0020] Optionally, the pre-processed data is input into a large model for integration processing to obtain collected data, including:

[0021] Format the preprocessed data according to the input requirements of the large model to obtain formatted data;

[0022] A deep learning algorithm is used to integrate the formatted data according to the rules to obtain the collected data.

[0023] Optionally, the pre-processed data is input into the large model for integration processing to obtain the collected data, which includes:

[0024] Get the collected data display instruction;

[0025] Generate a data visualization icon according to the collected data display instruction and the collected data;

[0026] Generate a data collection report based on the data collection process, data preprocessing results, data integration processing results and data visualization charts.

[0027] In a second aspect, the present application provides an automated data collection device, comprising:

[0028] An acquisition module is used to acquire parameter configuration information input by a user, wherein the parameter configuration information includes: ID, URL and configuration selector;

[0029] A determination module, used to determine the data collection identifier corresponding to the parameter configuration information input by the user according to the ID, the URL, the configuration selector and the data collection scheme;

[0030] A processing module, used to capture the original data in the browser according to the data collection identifier corresponding to the parameter configuration information input by the user, and pre-process the original data to obtain pre-processed data;

[0031] The processing module is also used to input the pre-processed data into the large model for integration processing to obtain collected data.

[0032] Optionally, the automatic data collection device further includes: a generation module;

[0033] The acquisition module is also used to obtain ID information, URL information and configuration selector information;

[0034] The generating module is used to generate a data collection identifier according to the ID information, the URL information and the configuration selector information;

[0035] The determination module is further used to associate the ID information, the URL information, the configuration selector information and the data collection identifier to construct a data collection plan.

[0036] Optionally, the generating module is further used to generate request information according to the data collection identifier corresponding to the parameter configuration information input by the user;

[0037] The processing module is further used to send the request information to the browser, capture the original data in the browser, and save and export the original data.

[0038] Optionally, the processing module is also used to clean, convert, normalize or standardize the original data to remove noise, fill missing values, convert data types, and obtain preprocessed data.

[0039] Optionally, the processing module is further used to format the preprocessed data according to the input requirements of the large model to obtain formatted data;

[0040] The processing module is also used to use a deep learning algorithm to perform rule integration on the formatted data to obtain collected data.

[0041] Optionally, the acquisition module is further used to acquire a collection data display instruction;

[0042] The generating module is further used to generate a data visualization icon according to the collected data display instruction and the collected data;

[0043] The generation module is also used to generate a data collection report based on the data collection process, data preprocessing results, data integration processing results and data visualization charts.

[0044] In a third aspect, the present application provides an automated data collection device, comprising:

[0045] Memory;

[0046] processor;

[0047] Wherein, the memory stores computer-executable instructions;

[0048] The processor executes the computer-executable instructions stored in the memory to implement the automated data collection method as described in the first aspect and various possible implementations of the first aspect.

[0049] In a fourth aspect, the present application provides a computer storage medium, characterized in that the computer storage medium stores computer execution instructions, and when the computer execution instructions are executed by a processor, they are used to implement the automated data collection method as described in the first aspect and various possible implementations of the first aspect.

[0050] The automated data collection method provided by the present application obtains parameter configuration information input by a user, wherein the parameter configuration information includes: an ID, a URL, and a configuration selector, and determines a data collection identifier corresponding to the parameter configuration information input by the user according to the ID, the URL, the configuration selector, and a data collection scheme. According to the data collection identifier corresponding to the parameter configuration information input by the user, the original data in the browser is captured, and the original data is preprocessed to obtain preprocessed data, and the preprocessed data is input into a large model for integration processing to obtain collected data. The method realizes efficient and flexible data collection and improves the intelligence level of data collection. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0052] Figure 1 This is the process of the automated data collection method provided by this application Figure 1 ;

[0053] Figure 2 This is the process of the automated data collection method provided by this application Figure 2 ;

[0054] Figure 3 It is a structural schematic diagram of the automated data acquisition device provided by the present application;

[0055] Figure 4 It is a structural schematic diagram of the automated data acquisition device provided in this application.

[0056] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0057] In order to make the purpose, technical solutions and advantages of this application clearer, the technical solutions in this application will be clearly and completely described below in conjunction with the drawings in this application. Obviously, the described embodiments are part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0058] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in sequences other than those illustrated or described herein, for example.

[0059] In the embodiments of the present application, the words "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0060] With the advent of the big data era, the demand for data collection has risen sharply. Behind this is the rapid progress of the Internet and information technology, which have jointly contributed to the explosive growth of massive information in data sources such as web pages, API interfaces, and databases. The urgent demand for accurate and real-time data from enterprises, scientific research institutions, and individuals has become a powerful driving force for the continuous innovation of automated data collection technology.

[0061] Currently, data collection using programming has become mainstream. This process involves building scripts or programs that can automatically send requests to the target data source and parse the response content. These technical tools can simulate browser operations, execute HTTP requests, flexibly process a variety of data formats (HTML, JSON, XML, etc.), and efficiently store the collected data locally or in a database.

[0062] Although this method gives data collection extremely high flexibility and powerful functions, it is also accompanied by challenges that cannot be ignored, and the programming complexity has increased significantly. To successfully write such automated scripts or programs, developers not only need to have solid programming skills, but also need to have a deep understanding of the internal structure, interaction protocol and data processing logic of the target data source, which also increases the difficulty and threshold of technical implementation.

[0063] In response to the above problems, the automated data collection method provided in the present application receives parameter configuration information including ID, URL and configuration selector input by the user, determines the data collection identifier in combination with the data collection plan, and then efficiently captures the original data in the browser and performs preprocessing to improve the data quality. Finally, the preprocessed data is sent to the large model for intelligent integration processing, thereby achieving a significant improvement in the efficiency, flexibility and intelligence of data collection.

[0064] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0065] Figure 1 This is the process of the automated data collection method provided by this application Figure 1 .like Figure 1 As shown, the automated data collection method provided in this embodiment includes:

[0066] S101: Acquire parameter configuration information input by a user, wherein the parameter configuration information includes: ID, URL, and configuration selector.

[0067] Among them, obtaining parameter configuration information input by the user refers to receiving the parameter configuration information input by the user through a user interface (such as a web form, a command line tool, or a graphical interface), and the parameter configuration data is used to configure or customize the behavior of a program, application, or service.

[0068] The parameter configuration information specifically includes three key parts: ID (used to uniquely identify a configuration or resource), URL (the address of the network resource that needs to be processed or accessed), and configuration selector (used to specify the rules or methods for selecting or locating data or elements in the page pointed to by the URL).

[0069] It can be understood that the user interface includes controls such as input boxes, drop-down selection boxes, check boxes, etc., which are used to receive IDs, URLs, and configuration selectors input by users. Before the user submits the parameter configuration information, necessary input validation is performed to ensure the uniqueness of the ID, the correct format of the URL, and the validity of the configuration selector. The configuration information that has passed the validation is stored in an appropriate location (such as a database, configuration file, or memory).

[0070] S102: Determine a data collection identifier corresponding to the parameter configuration information input by the user according to the ID, the URL, the configuration selector and the data collection scheme.

[0071] The data collection identifier is a unique or specific identifier used to locate and execute data collection tasks that match the parameter configuration information in the system. Determining the data collection identifier ensures the accuracy and pertinence of data collection.

[0072] The reason for determining the data collection identifier is that different data collection tasks involve different data sources, data formats, and collection rules. By combining the parameter configuration information entered by the user and the data collection plan, the boundaries and requirements of each collection task can be accurately defined to ensure the accuracy and consistency of the data.

[0073] S103: According to the data collection identifier corresponding to the parameter configuration information input by the user, the original data in the browser is captured, and the original data is preprocessed to obtain preprocessed data.

[0074] It can be understood that according to the parameter configuration information input by the user, the original data matching the parameter configuration information is accurately identified and captured in the browser. Subsequently, a series of preprocessing operations are performed on these original data to improve data quality, unify the format or remove noise, etc., so as to obtain preprocessed data suitable for subsequent analysis or application.

[0075] First, due to the wide variety of data in the browser and different formats, direct analysis or processing may encounter compatibility issues or data quality issues. Secondly, users only need a part of the data, or need the data to be presented in a specific format. Therefore, by accurately collecting and preprocessing data according to user parameter configuration, the accuracy and validity of the data can be ensured, the errors and costs in the subsequent processing process can be reduced, and the efficiency of overall data processing can be improved.

[0076] Use browser automation tools (such as Selenium, Puppeteer, etc.) or API interfaces to locate and capture the corresponding raw data in the browser according to the parsed parameter configuration information. Perform preprocessing operations such as cleaning (such as removing invalid characters and filling null values), conversion (such as format unification and data type conversion), and filtering (such as filtering data based on specific rules) on the collected raw data. After completing the preprocessing, verify the quality of the data to ensure that the processing results meet expectations.

[0077] S104: Input the preprocessed data into the large model for integration processing to obtain collected data.

[0078] The reason for inputting preprocessed data into the big model for integration is that, although preprocessing can improve the quality and applicability of data, it is often not enough to directly meet the needs of complex analysis or decision-making. The big model, through its powerful computing power and complex algorithm structure, can further explore the potential value in the data, discover hidden patterns and connections between data, and thus generate more comprehensive, accurate and insightful collected data.

[0079] Input the preprocessed data into the model in the format and manner required by the large model. This may require converting the data into a specific data structure (such as tensor, data frame, etc.) and performing necessary batch processing or data enhancement operations. Integrate the input data in the large model, which may include feature extraction and pattern recognition. Finally, based on the output of the model, generate and extract the required acquisition data.

[0080] The automated data collection method provided in this embodiment obtains parameter configuration information input by a user, wherein the parameter configuration information includes: an ID, a URL, and a configuration selector, and determines a data collection identifier corresponding to the parameter configuration information input by the user according to the ID, the URL, the configuration selector, and a data collection scheme. According to the data collection identifier corresponding to the parameter configuration information input by the user, the original data in the browser is captured, and the original data is preprocessed to obtain preprocessed data, and the preprocessed data is input into a large model for integration processing to obtain collected data. The method realizes efficient and flexible data collection and improves the intelligence level of data collection.

[0081] Figure 2 This is the process of the automated data collection method provided by this application Figure 2 .like Figure 2 As shown, in this embodiment Figure 1 Based on the embodiment, the automated data collection method is described in detail. The automated data collection method shown in this embodiment includes:

[0082] S201: Obtain ID information, URL information, and configuration selector information.

[0083] The ID information usually refers to the unique identifier in the HTML element. The URL information is the address of the web page. The configuration selector information is one or more instructions for selecting required data from the HTML document, such as CSS selectors or XPath expressions, which are used to extract required content from complex web page structures.

[0084] ID information allows users to accurately access and operate specific elements on a page. URL information is the gateway to any web page. Whether it is a user visiting directly or a crawler crawling data, you need to know the URL of the target page. Configuring selector information makes data crawling or automation tasks more flexible and powerful, and can handle various complex web page structures and extract information from them.

[0085] S202: Generate a data collection identifier according to the ID information, the URL information and the configuration selector information.

[0086] The ID information, URL information and configuration selector information are combined or processed into a unique identifier (i.e., data collection identifier). This identifier is used to identify a specific data collection task, ensuring the accuracy of the data, the clarity of the source, and the consistency of subsequent processing.

[0087] Design the rules for generating data collection identifiers based on the ID information, URL information, and configuration selector information. This includes determining the format, length, and fields included in the identifier. For example, a hash function can be used to combine key information into a short string as an identifier. Then, write code or configuration scripts to generate data collection identifiers based on the designed rules. This usually involves parsing, formatting, and combining ID information, URL information, and configuration selector information. Associate the generated data collection identifier with the corresponding data collection task and use it to identify data during data collection, processing, and storage.

[0088] S203: Associating the ID information, the URL information, the configuration selector information and the data collection identifier to construct a data collection plan.

[0089] Associating ID information, URL information, configuration selector information and data collection identifier to build a data collection plan is a method to customize data capture strategies by integrating multiple key information elements during the data collection process. It aims to ensure the accuracy, efficiency and traceability of data collection. ID information is used to uniquely identify the data source or data item, URL information specifies the network location of the data, configuration selector information defines the rules or patterns for extracting specific data from the location, and the data collection identifier is the unique identification code for the entire collection task, which is convenient for subsequent management and analysis.

[0090] The reason why we need to associate these four types of information and build a data collection plan is mainly because the explosive growth and diversification of modern data makes it extremely complicated and time-consuming to accurately extract the required data directly from massive information. By designing a data collection plan, we can greatly reduce errors and redundancies in the data capture process and improve data quality.

[0091] This step can use regular expressions, XPath or CSS selectors and other technical means to describe the configuration selector information to ensure that the required data can be accurately extracted from the specified URL location. Next, the ID information, URL information, configuration selector information and data collection identifier are associated according to certain logical relationships to build a complete data collection process. Finally, this solution is implemented by writing scripts or using professional data collection tools to perform data collection tasks.

[0092] S204: Acquire parameter configuration information input by the user, where the parameter configuration information includes: ID, URL, and configuration selector.

[0093] Among them, step S204 is similar to step S101 and will not be repeated here.

[0094] S205: Determine a data collection identifier corresponding to the parameter configuration information input by the user according to the ID, the URL, the configuration selector and the data collection scheme.

[0095] Determining the data collection identifier corresponding to the parameter configuration information entered by the user based on the ID, URL, configuration selector and data collection scheme is a process of assigning or confirming a unique identifier (i.e., data collection identifier) ​​for data capture requirements through matching and verification processes in a data collection environment. This identifier associates the user's personalized parameter settings with the data collection scheme to ensure the accuracy of data collection.

[0096] First, the data collection solution library contains a variety of collection strategies based on different IDs, URLs, and configuration selectors. When the user enters parameter configuration information, the system will find a matching data collection solution by comparing these parameters with the similarities of the data collection solution library in the library. If a completely matching solution is found, the data collection identifier corresponding to the solution is used directly; if a partial match or no matching solution is found, a new collection solution needs to be dynamically generated based on the user input and a new data collection identifier is assigned to it.

[0097] S206: Generate request information according to the data collection identifier corresponding to the parameter configuration information input by the user.

[0098] The system first parses the parameter configuration information entered by the user through the interface, API or other means. Then, the system uses the parameter configuration information to match or convert predefined data collection identifiers (such as API endpoints, database query statements, file paths, etc.) to construct request information that meets the requirements. The request information includes the target address (URL), request method (such as GET, POST), request headers (Headers), request body (Body), etc.

[0099] By allowing users to configure parameters according to actual needs, the system can flexibly adjust the strategy and scope of data collection to meet diverse data analysis, monitoring or business operation needs. At the same time, automatically generating request information reduces the tediousness and errors of manually writing requests, and improves the efficiency and accuracy of data collection.

[0100] The system needs to receive and parse the parameter configuration information entered by the user to ensure the integrity and correctness of the information. This includes verifying the format and type of the parameters and whether they meet the necessary constraints.

[0101] Based on the parsed parameters, the system searches for the corresponding data collection identifier in the internal database or configuration file. This may involve mapping, calculating, or converting the parameters to ensure that they match the actual data collection resources (such as APIs, databases). Once the correct data collection identifier is found or calculated, the system can begin to build the request information. This includes determining the target address of the request, selecting the appropriate request method, setting the necessary request headers and request bodies, etc.

[0102] S207: Send the request information to the browser, capture the original data in the browser, and save and export the original data.

[0103] Send request information (such as HTTP request) to the browser to trigger the loading of the web page. Then, the system will capture and grab the raw data generated by the browser after executing the request information, including HTML documents, CSS styles, JavaScript codes, network request responses, etc. Finally, save the captured raw data to the local file system or database for subsequent analysis, processing or export.

[0104] S208: Clean, convert, normalize or standardize the original data to remove noise, fill missing values, convert data types, and obtain preprocessed data.

[0105] S209: Format the preprocessed data according to the input requirements of the large model to obtain formatted data.

[0106] Since raw data often contains various problems, such as data inconsistency, missing values, noise, outliers, etc., these problems will directly affect the accuracy of data analysis and the performance of machine learning models. Data cleaning can remove these unfavorable factors and ensure the accuracy and integrity of the data.

[0107] Data cleaning: First, identify and process missing values, which can be done by filling (such as mean, median, mode filling or predicted filling), deletion or interpolation; second, detect and correct erroneous data, such as identifying outliers by setting thresholds and deciding whether to retain, correct or delete; finally, remove or transform duplicate data to ensure the uniqueness of the data set.

[0108] Data conversion: Based on the analysis requirements, data is converted from one type to another (such as text to numeric encoding), or data is converted from one format to another (such as the unification of date and time formats). In addition, it may also include feature construction, that is, generating new features based on existing features.

[0109] Normalization or standardization: Normalization is to scale the data so that it falls into a small specific interval (such as between 0 and 1), which is often used when the attribute value range varies greatly; standardization is to adjust the data by subtracting the mean and then dividing by the standard deviation to make it conform to the standard normal distribution, which is more suitable for situations where the data distribution varies greatly. Which method to choose depends on the specific characteristics of the data and the analysis requirements.

[0110] S210: Using a deep learning algorithm to perform rule integration on the formatted data to obtain collected data.

[0111] By leveraging the powerful nonlinear mapping and feature extraction capabilities of deep learning models, we can conduct in-depth analysis of data that has been processed in a specific format, automatically discover hidden laws, patterns, or relationships in the data, and integrate or reconstruct the data based on these laws, ultimately obtaining one or more forms of collected data.

[0112] Deep learning algorithms are used for rule integration of formatted data because they can automatically learn and abstract high-level feature representations from raw data. In addition, deep learning models have good generalization capabilities and can handle large-scale, high-dimensional data and effectively deal with noise and outliers in the data. Therefore, rule integration through deep learning algorithms can more accurately reveal the laws and patterns behind the data, thereby obtaining more valuable collected data.

[0113] Deep learning models can include convolutional neural networks (CNN), recurrent neural networks (RNN), long short-term memory networks (LSTM), and autoencoders.

[0114] S211: Obtaining collected data display instructions.

[0115] Receive specific instructions for how to display the collected data. These instructions include data sorting, filtering conditions, visualization types (such as charts, maps, lists, etc.), display formats (such as date formats, number formats), etc., to ensure that the data can be displayed accurately and intuitively according to the needs of users or systems.

[0116] S212: Generate a data visualization icon according to the collected data display instruction and the collected data.

[0117] The process of processing the collected data according to specific display instructions and converting it into visual forms such as graphics, images or animations.

[0118] Data visualization icons can transform complex and abstract data into information that is easy to understand and perceive. Compared with plain text or digital tables, visualization icons can more intuitively display the distribution, trend, and comparative relationship of data, allowing users to quickly capture key information and improve the efficiency of data analysis and decision-making. In addition, visualization icons can also enhance the attractiveness of data, stimulate users' interest and curiosity, and promote the dissemination and sharing of data.

[0119] S213: Generate a data collection report based on the data collection process, data preprocessing results, data integration processing results, and data visualization charts.

[0120] Data collection reports play a vital role in data quality management and data analysis accuracy. First, data collection reports can ensure the traceability of the data collection process, making it easy to quickly locate the cause and take measures when data quality problems occur. Second, by recording the results of data preprocessing and integration in detail, data collection reports improve the accuracy and reliability of data analysis.

[0121] The automated data collection method provided in this embodiment obtains ID information, URL information and configuration selector information, generates a data collection identifier according to the ID information, URL information and configuration selector information, associates the ID information, URL information, configuration selector information and data collection identifier, builds a data collection plan, obtains parameter configuration information input by a user, the parameter configuration information includes: ID, URL and configuration selector, determines the data collection identifier corresponding to the parameter configuration information input by the user according to the ID, URL, configuration selector and data collection plan, generates request information according to the data collection identifier corresponding to the parameter configuration information input by the user, and sends the request information to the browser. In the browser, the raw data in the browser is captured, the raw data is saved and exported, the raw data is cleaned, converted, normalized or standardized to remove noise, fill missing values, convert data types, and obtain preprocessed data, the preprocessed data is formatted according to the input requirements of the large model to obtain formatted data, the formatted data is rule-integrated using a deep learning algorithm to obtain collected data, a collection data display instruction is obtained, a data visualization icon is generated according to the collection data display instruction and the collection data, and a data collection report is generated according to the data collection process, data preprocessing results, data integration results, and data visualization charts; this method realizes efficient and flexible data collection and improves the intelligence of data collection.

[0122] Figure 3 Schematic diagram of the structure of the automatic data acquisition device provided by this application. Figure 3 As shown, the present application provides an automated data collection device, the automated data collection device 300 comprising:

[0123] The acquisition module 301 is used to acquire parameter configuration information input by the user, wherein the parameter configuration information includes: ID, URL and configuration selector;

[0124] A determination module 302, configured to determine a data collection identifier corresponding to the parameter configuration information input by the user according to the ID, the URL, the configuration selector and the data collection scheme;

[0125] The processing module 303 is used to capture the original data in the browser according to the data collection identifier corresponding to the parameter configuration information input by the user, and pre-process the original data to obtain pre-processed data;

[0126] The processing module 303 is also used to input the pre-processed data into the large model for integration processing to obtain collected data.

[0127] Optionally, the automatic data collection device further includes: a generating module 304;

[0128] The acquisition module 301 is also used to acquire ID information, URL information and configuration selector information;

[0129] The generating module 304 is used to generate a data collection identifier according to the ID information, the URL information and the configuration selector information;

[0130] The determination module 302 is further used to associate the ID information, the URL information, the configuration selector information and the data collection identifier to construct a data collection plan.

[0131] Optionally, the generating module 304 is further configured to generate request information according to the data collection identifier corresponding to the parameter configuration information input by the user;

[0132] The processing module 303 is further used to send the request information to the browser, capture the original data in the browser, and save and export the original data.

[0133] Optionally, the processing module 303 is also used to clean, convert, normalize or standardize the original data to remove noise, fill missing values, convert data types, and obtain pre-processed data.

[0134] Optionally, the processing module 303 is further used to format the preprocessed data according to the input requirements of the large model to obtain formatted data;

[0135] The processing module 303 is further used to use a deep learning algorithm to perform rule integration on the formatted data to obtain collected data.

[0136] Optionally, the acquisition module 301 is further used to acquire a collection data display instruction;

[0137] The generating module 304 is further used to generate a data visualization icon according to the collected data display instruction and the collected data;

[0138] The generating module 304 is further used to generate a data collection report according to the data collection process, data preprocessing results, data integration processing results and data visualization charts.

[0139] Figure 4 Schematic diagram of the structure of the automated data acquisition device provided by this application. Figure 4 As shown, the present application provides an automatic data collection device, and the automatic data collection 400 includes: a receiver 401, a transmitter 402, a processor 403 and a memory 404.

[0140] Receiver 401, for receiving instructions and data;

[0141] A transmitter 402, used for sending instructions and data;

[0142] Memory 404, for storing computer-executable instructions;

[0143] The processor 403 is used to execute the computer-executable instructions stored in the memory 404 to implement the various steps performed by the automatic data collection method in the above embodiment. For details, please refer to the relevant description in the above automatic data collection method embodiment.

[0144] Optionally, the memory 404 may be independent or integrated with the processor 403 .

[0145] When the memory 404 is independently provided, the electronic device further includes a bus for connecting the memory 404 and the processor 403 .

[0146] The present application also provides a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the automated data collection method performed by the automated data collection device described above is implemented.

[0147] It will be appreciated by those skilled in the art that all or some of the steps, systems, and functional modules / units in the methods disclosed above may be implemented as software, firmware, hardware, and appropriate combinations thereof. In hardware implementations, the division between the functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed by several physical components in cooperation. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or implemented as hardware, or implemented as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer storage medium (or non-transitory medium) and a communication medium (or temporary medium). As known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by a computer. In addition, it is well known to those of ordinary skill in the art that communication media typically contain computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

[0148] So far, the technical solution of the present application has been described in conjunction with the preferred embodiments shown in the accompanying drawings. However, it is easy for those skilled in the art to understand that the protection scope of the present application is obviously not limited to these specific embodiments, and the above embodiments are only used to illustrate the technical solution of the present application rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein by equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. An automated data collection method, characterized in that: The method comprises: Obtain parameter configuration information input by the user, the parameter configuration information including: ID, URL and configuration selector; Determine the data collection identifier corresponding to the parameter configuration information input by the user according to the ID, the URL, the configuration selector and the data collection scheme; According to the data collection identifier corresponding to the parameter configuration information input by the user, the original data in the browser is captured, and the original data is preprocessed to obtain preprocessed data; The preprocessed data is input into the large model for integration and processing to obtain the collected data.

2. The method according to claim 1, characterized in that Before obtaining the parameter configuration information input by the user, the method further includes: Get ID information, URL information, and configuration selector information; Generate a data collection identifier according to the ID information, the URL information and the configuration selector information; The ID information, the URL information, the configuration selector information and the data collection identifier are associated to construct a data collection plan.

3. The method according to claim 1, characterized in that The method of capturing raw data in the browser according to the data collection identifier corresponding to the parameter configuration information input by the user includes: Generate request information according to the data collection identifier corresponding to the parameter configuration information input by the user; The request information is sent to the browser, the original data in the browser is captured, and the original data is saved and exported.

4. The method according to claim 1, characterized in that The preprocessing of the original data to obtain preprocessed data includes: The raw data is cleaned, converted, normalized or standardized to remove noise, fill missing values, and convert data types to obtain preprocessed data.

5. The method according to claim 1, characterized in that The pre-processed data is input into the large model for integration processing to obtain the collected data, including: Format the preprocessed data according to the input requirements of the large model to obtain formatted data; A deep learning algorithm is used to integrate the formatted data according to the rules to obtain the collected data.

6. The method according to claim 1, characterized in that The pre-processed data is input into the large model for integration processing to obtain the collected data, including: Get the collected data display instruction; Generate a data visualization icon according to the collected data display instruction and the collected data; Generate a data collection report based on the data collection process, data preprocessing results, data integration processing results and data visualization charts.

7. An automated data collection device, characterized in that: include: An acquisition module is used to acquire parameter configuration information input by a user, wherein the parameter configuration information includes: ID, URL and configuration selector; A determination module, used to determine the data collection identifier corresponding to the parameter configuration information input by the user according to the ID, the URL, the configuration selector and the data collection scheme; A processing module, used to capture the original data in the browser according to the data collection identifier corresponding to the parameter configuration information input by the user, and pre-process the original data to obtain pre-processed data; The processing module is also used to input the pre-processed data into the large model for integration processing to obtain collected data.

8. The device according to claim 7, characterized in that Also includes: Generate modules; The generating module is used to generate request information according to the data collection identifier corresponding to the parameter configuration information input by the user; The processing module is further used to send the request information to the browser, capture the original data in the browser, and save and export the original data.

9. An automated data collection device, characterized in that: include: Memory; processor; Wherein, the memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the automated data collection method according to any one of claims 1 to 6.

10. A computer storage medium, characterized in that: The computer storage medium stores computer executable instructions, which, when executed by a processor, are used to implement the automated data collection method according to any one of claims 1 to 6.

Citation Information

Patent Citations

  • Webpage data collection method and apparatus, computer device and storage medium

    CN108304498A

  • Data capture method and system, medium and electronic equipment

    CN110765334A