Data acquisition and automatic storage method, system and equipment
By configuring API interface information and data mapping rules, the automated collection and storage of interface data has been achieved, solving the problems of poor adaptability and cumbersome operation in existing technologies, and improving the system's adaptability and maintainability.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- YUNSHANG (GUIZHOU) DATA DEVELOPMENT CO LTD
- Filing Date
- 2026-01-22
- Publication Date
- 2026-05-01
AI Technical Summary
Existing technologies suffer from poor adaptability, poor configuration flexibility, limited scalability, and cumbersome operation during interface data collection and storage, making it difficult to quickly respond to business needs.
By configuring the basic call information and data processing information of the API interface to collect data, data extraction information is generated, mapping rules between source fields in the database and target table fields in the target database are established to realize automated data collection and storage. It supports configurable data extraction and transformation logic, uses a visual interface for configuration, and supports multi-node parallel collection and anomaly handling.
It improved the system's adaptability and maintainability, reduced manual operation costs, and enabled rapid response to business needs and an efficient data collection process.
Smart Images
Figure CN121958404A_ABST
Abstract
Description
Data acquisition and automated data entry methods, systems and equipment Technical Field
[0001] This invention relates to the field of data acquisition technology, specifically to a data acquisition and automatic data entry method, system, and device. Background Technology
[0002] With the rapid development of information technology, data interaction between various systems is becoming increasingly frequent. Data acquisition, as a pre-processing step, directly impacts the effectiveness of subsequent data applications due to its efficiency and reliability. Currently, interface data acquisition mainly relies on traditional customized development or general-purpose acquisition tools, but these methods have significant limitations in practical applications: First, insufficient compatibility. Different system interfaces often use different protocols (such as HTTP and WebService) and data formats (JSON and XML). Traditional solutions require developing separate acquisition logic for each interface, resulting in high adaptation costs and repetitive work. Second, poor configuration flexibility. Core configurations such as interface parameters, request methods, and parsing rules are often hard-coded in the code. Modifying the configuration requires recompilation and deployment, making it difficult to cope with frequently changing business needs. Third, limited scalability. Adding a new interface protocol or data format requires deep modification of the core code, resulting in high module coupling and an inability to quickly respond to expansion needs. Furthermore, the process of acquiring data and then storing it in a database is cumbersome and complex. Summary of the Invention
[0003] In view of this, the purpose of the present invention is to provide a data acquisition and automatic data entry method, system and device to overcome the problems of poor adaptability and cumbersome data entry when data is acquired and entered into the database through an interface.
[0004] To achieve the above objectives, the present invention adopts the following technical solution: Firstly, this application provides a data acquisition and automatic database entry method, characterized by comprising: configuring basic call information and data processing information for API interface data acquisition, wherein the data processing information includes data extraction information generated based on data hierarchy and node information; generating database source fields based on the data extraction information, and establishing mapping rules between the database source fields and target table fields in the target database; calling the API interface based on the basic call information to obtain the result data returned by the API interface; extracting target data from the result data based on the data extraction information; processing the target data, and writing the processed data into the corresponding target table fields based on the database source fields corresponding to the target data and the mapping rules.
[0005] Furthermore, in some embodiments of this application, the basic call information includes: entry network environment, interface address, request method, authentication method, request parameters, return example, pagination configuration, and return structure.
[0006] Furthermore, in some embodiments of this application, for JSON formatted data, the data hierarchy and node information are determined based on the jsonpath path.
[0007] Furthermore, in some embodiments of this application, the step of generating database source fields based on the data extraction information and establishing mapping rules between the database source fields and target table fields in the target database includes: renaming the jsonpath path information of the JSON format data to be extracted to obtain database source fields; using the database source fields as target table fields in the target database to create a new target table in the target database; or mapping and binding the database source fields with existing target table fields in the target database.
[0008] Furthermore, in some embodiments of this application, the step of calling the API interface based on the basic call information and obtaining the result data returned by the API interface includes: triggering the API data collection task based on manual and timed triggering methods; distributing the data collection task to multiple nodes so that data collection can be performed in parallel by multiple nodes; wherein, when multiple nodes perform data collection in parallel, it is based on predefined task dependencies and periodic rules.
[0009] Furthermore, in some embodiments of this application, extracting target data from the result data based on the data extraction information includes: parsing the result data to obtain parsing results; and extracting data from the corresponding jsonpath path based on the data hierarchy and node information.
[0010] Furthermore, in some embodiments of this application, the data processing information further includes data conversion rule information, and the processing of the target data includes: converting the extracted data into a target format based on the data conversion rule information.
[0011] Furthermore, in some embodiments of this application, the method further includes: real-time collection of operational data and determination of abnormal information; wherein the operational data includes request time, success rate, number of abnormal events, CPU utilization, and memory utilization; the abnormal information includes network abnormalities, service abnormalities, and parsing abnormalities; determining an abnormality handling strategy based on the abnormality information, and displaying the operational status; wherein the abnormality handling strategy includes retrying for network abnormalities using an exponential backoff algorithm.
[0012] Secondly, this application provides a data acquisition and automatic database entry system, comprising: a configuration module for configuring basic call information and data processing information for API interface data acquisition, wherein the data processing information includes data extraction information generated based on data hierarchy and node information; a running module for generating database source fields based on the data extraction information and establishing mapping rules between the database source fields and target table fields in the target database; calling the API interface based on the basic call information to obtain the result data returned by the API interface; extracting target data from the result data based on the data extraction information; and processing the target data and writing the processed data into the corresponding target table fields based on the database source fields corresponding to the target data and the mapping rules.
[0013] Thirdly, this application provides a data acquisition and automatic data entry device, including a processor and a memory, wherein the processor is connected to the memory; wherein the processor is used to call and execute a program stored in the memory; the memory is used to store the program, and the program is used to execute at least the above-described data acquisition and automatic data entry method.
[0014] The beneficial effects of this application include: by introducing configurable data extraction information, it is possible to intuitively locate and extract any specific data node in the complex nested structure returned by the API, thus solving the shortcomings of traditional tools in selective data collection. Furthermore, the complete transformation logic from the data source to the target table can be defined solely through relevant configuration. When the collection conditions change, only the corresponding configuration needs to be changed to complete the adjustment, without modifying the program code or redeploying, thereby improving the system's adaptability and maintainability. Attached Figure Description
[0015] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0016] Figure 1 is a flowchart illustrating the data acquisition and automatic warehousing method provided in an embodiment of the present invention; Figure 2 is a structural diagram illustrating the data acquisition and automatic warehousing system provided in an embodiment of the present invention; Figure 3 is a structural diagram illustrating the data acquisition and automatic warehousing device provided in an embodiment of the present invention. Detailed Implementation
[0017] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be described in detail below. Obviously, the described embodiments are merely some embodiments of this invention, and not all embodiments. Based on the embodiments of this invention, all other implementation methods obtained by those skilled in the art without creative effort are within the scope of protection of this invention.
[0018] Figure 1 is a flowchart illustrating the data acquisition and automatic data entry method provided in an embodiment of the present invention. Referring to Figure 1, this embodiment may include the following steps: S101, configuring the basic call information and data processing information for API interface data acquisition.
[0019] The basic call information is used by the system (the data collection and automatic data entry system for executing the data collection and automatic data entry method provided in this application) to determine how to call the API for data collection; the data processing information includes data extraction information generated based on data hierarchy and node information, which is used to extract the result data returned by the API, thereby obtaining the target data that the user wants to obtain.
[0020] S102. Generate database source fields based on data extraction information, and establish mapping rules between database source fields and target table fields in the target database.
[0021] Specifically, in this application, users can pre-generate database source fields based on data extraction information, and then map the database source fields to the fields of the target table in the database storing the target data, i.e., the target table fields. Thus, after the target data is subsequently obtained, the target data is automatically written into the target table fields based on the database source fields and mapping rules corresponding to the target data, thereby achieving automatic database entry.
[0022] S103. Call the API interface based on the basic call information and obtain the result data returned by the API interface.
[0023] S104. Extract target data from the result data based on the data.
[0024] S105. Process the target data and, based on the database source fields and mapping rules corresponding to the target data, write the processed data into the corresponding destination table fields.
[0025] The data acquisition and automatic data entry method provided in this application, by introducing configurable data extraction information, can intuitively locate and extract any specific data node in the complex nested structure returned by the API, thus solving the shortcomings of traditional tools in selective data acquisition. Furthermore, the complete transformation logic from the data source to the target table can be defined solely through relevant configuration. When the acquisition conditions change, only the corresponding configuration needs to be modified to complete the adjustment, without modifying the program code or redeploying, thereby improving the system's adaptability and maintainability.
[0026] Furthermore, in this embodiment of the application, the aforementioned basic call information specifically includes: entry network environment, interface address, request method, authentication method, request parameters, return example, pagination configuration, and return structure, etc.
[0027] In the data collection and automatic data entry method provided in this application, users can configure data parameters in the system's visual interface, including the basic configuration of the aforementioned basic call information (of course, users can configure only part of the above configuration, and the system will perform corresponding data queries based on the user's configured partial data to automatically fill in other data, or display reminders to the user to configure the remaining items).
[0028] Specifically, in the above configuration, the ingress network environment can be configured with a specific network type; the interface address can be an HTTP request URL, and the corresponding interface type can include HTTPS and HTTP; the request method can include GET and POST request methods; the request parameters can include header request parameters, query request parameters, body request parameters, and pagination request parameters, etc., where body request parameters can include Form-data, x-www-form-urlencoded, Raw-json, and Raw-xml; the authentication method can include API Key and Basic Auth, and of course, interfaces that can be accessed without authorization information can also be set.
[0029] The system can also be configured to handle data in formats such as JSON and XML by setting the return example and return structure.
[0030] The returned example can be the result data returned when a specific API interface responds successfully, i.e., a successful response example. When the type of the result data returned by the interface is uncertain, it can be determined by sending a relevant request, and the returned result data will be automatically populated after the request is successful. In practical applications, a specified value or a user-defined value can be set as an interface success response identifier (which can be used in conjunction with a status description to explain the actual meaning of the identifier) to determine whether the request was successful.
[0031] Furthermore, in this application, for JSON format data, the data hierarchy and node information are determined based on the jsonpath path. That is, for the JSON format result data returned by the API interface, the jsonpath path to be collected can be predefined so that the subsequent system can extract the result data returned by the API based on the jsonpath path.
[0032] Meanwhile, in order to achieve automatic data entry into the database, the jsonpath path information of the JSON data to be extracted is renamed to obtain the database source field; then the database source field is used as the target table field in the target database to create a new target table in the target database; or the database source field is mapped and bound to the target table field that already exists in the target database.
[0033] In practical applications, data input can be configured through the system's preset database output components, including setting basic connection information and configuring field mapping rules for data tables.
[0034] The basic connection information is used to specify the database type (such as MySQL, PostgreSQL), select a registered data source instance (i.e., the target database), and configure the name and optional schema of the target database. The data table configuration is used to select or create data tables and specify the database source fields to be written (i.e., the database source fields obtained by renaming the jsonpath path information in the above steps) and the destination table fields (fields of the data table in the target database). The field mapping rules are used to accurately map and bind the specified database source fields and destination table fields to achieve automatic data entry into the database.
[0035] In practical applications, when the target table does not exist in the target database, users can create it themselves. This involves renaming the database source field obtained by configuring the jsonpath through the interface input component and using it as the field of the new table, i.e., the field of the new target table. Then, the table name, table comments, etc. are customized to complete the creation of the target table. After the creation is completed (and when the target table exists in the target database), by establishing the mapping relationship between the above data source field and the target table field, data from different paths in the result data returned by the interface can be saved to the corresponding target table field.
[0036] Furthermore, in some embodiments of this application, calling an API interface based on basic call information and obtaining the result data returned by the API interface includes: triggering an API data collection task based on manual and timed triggering methods; and distributing the data collection task to multiple nodes so that data collection can be performed in parallel by multiple nodes; wherein, when multiple nodes perform data collection in parallel, it is based on predefined task dependencies and periodic rules.
[0037] Specifically, during the actual execution of the data collection task, a separate job module can be responsible for execution. This module, based on multi-task orchestration and workflow execution (the relevant scheduling strategy configuration can be part of the basic call information, either configured simultaneously with or after the basic call information is configured), provides two triggering mechanisms: manual triggering (for real-time data collection) and automatic triggering based on a configured scheduling cycle (e.g., the execution frequency can be defined using Cron expressions, such as setting it to execute periodically by year, month, week, day, hour, or minute). By defining task dependencies and setting cycle rules, the data collection and automatic data entry method provided in this application can achieve automated sequential execution of batch tasks, meeting the needs of timed and event-driven data collection.
[0038] For example, specific interface acquisition plugins for execution can be developed based on open-source ETL tools. A Framework+plugin architecture can be adopted to abstract data source reading and database writing into Reader / Writer plugins.
[0039] In practical applications, the system can use its internal interface input component to obtain the configuration parameters of the data collection task, i.e., the basic call information mentioned above. Then, it can query all the parameter information of the registered data sources from the data source, construct the start request interface, and trigger the API interface to obtain the result data. After the result data is obtained, the interface input component can determine the JSON path to be collected based on the configured data extraction information, so as to extract the value of the result data (as will be mentioned later, a parsing step can be performed before extraction), and then write the data into the channel pipeline. At this time, the database output component can obtain the extracted target data from the channel pipeline, and then write the target data into the target table field of the target database according to the mapping relationship.
[0040] Scheduling strategy configuration: Its core function is to automate, standardize, and efficiently manage all interface data collection tasks. A unified scheduling engine coordinates task execution rhythm, resource allocation, and exception handling to ensure a stable collection process, data integrity, and compliance with business requirements. The specific configuration is described below: The invocation method supports manual and scheduled scheduling; scheduled tasks run periodically. The scheduling engine dynamically controls task execution based on the current server's CPU and memory usage, ensuring normal task execution even with a large number of data collection tasks.
[0041] It should be noted that during the configuration phase of the above data collection task, the system program can parse the successful response sample data of the API, extract the values under each path according to the configured jsonpath path, and write to the database according to the mapping relationship between the database source fields (corresponding to the jsonpath path) and the destination table fields. When the data collection task is completed, a preview result of the writing can be generated (such as displaying the written data) for the user to confirm the completion status of the task.
[0042] Furthermore, in some embodiments of this application, after obtaining the result data returned by the API, the method further includes parsing the result data to obtain a parsing result. The aforementioned extraction of target data based on data hierarchy and node information is performed on the parsing result, such as extracting data from the corresponding jsonpath path in the parsing result. The data processing information also includes data conversion rule information, used to convert the extracted target data into a target format based on the data conversion rules in the data conversion rule information.
[0043] Specifically, when collecting data from APIs, the returned data is typically in JSON or XML format. Taking JSON as an example (and XML similarly), in cases with multiple data sets, the returned data structure is usually complex, often multi-level. However, API collection often only requires a portion of the data. If manual API calls are used for data collection, the data must be filtered after each collection, which is time-consuming, labor-intensive, and prone to errors. To simplify the configuration and collection process for API data synchronization tasks, this application specifies the data levels and nodes to be collected during the task configuration phase, enabling one-time configuration, automatic parsing and extraction, and no subsequent intervention required. Specifically, users can quickly select the desired data level and nodes through a visual interface; each node name begins with a specific symbol such as "$" and is concatenated with a period ".". For example, selecting the "status" node results in a node path of $.status. This method allows for precise location of the desired data node. Furthermore, it distinguishes between nodes with the same name at different levels. Finally, the extracted target data is converted into a target format suitable for direct database entry using `transform` based on data transformation rules.
[0044] Thus, by selecting data nodes at different levels, the interface data to be collected and its corresponding node paths are filtered out. These node paths are then designed into data tables, including using the node path as the field name and the node name as the field comments. When collecting API interface data, the parsed, extracted, and format-converted data is automatically saved to the target table field in the database for easy subsequent statistics and analysis.
[0045] Furthermore, existing technologies lack robust anomaly handling strategies during data acquisition, making them prone to interruptions when faced with network fluctuations, interface timeouts, and other issues. They also lack effective retry and degradation strategies, resulting in poor system stability. At the same time, the lack of systematic monitoring of key indicators in the acquisition process (such as success rate and time consumption) makes it difficult to quickly locate the root cause of problems, leading to weak monitoring capabilities of the relevant systems.
[0046] Based on this, the data acquisition and automatic database entry method provided by the present invention further includes: real-time acquisition of operational data and determination of abnormal information; wherein, the operational data includes request time, success rate, number of abnormal events, CPU utilization, and memory utilization; the abnormal information includes network abnormalities, business abnormalities, and parsing abnormalities; determining an abnormality handling strategy based on the abnormality information, and displaying the operational status; wherein, the abnormality handling strategy includes retrying for network abnormalities using an exponential backoff algorithm.
[0047] The following is a detailed description of the principle and actual usage process of the data acquisition and automatic database entry method of this application, using a complete embodiment: Step 1: Configure the interface data source. Users can configure the interface data source through the system's visual interface. Specific information includes the entry network environment, interface address, request method, authentication method, request parameters, return example, pagination configuration, and some or all of the return structure (some fields can be set as required; after the user configures the required fields, the system automatically retrieves information for other items through querying) mentioned above. After the user completes the configuration, the system automatically verifies the legality of the interface information (such as URL format and required field checks) to ensure the accuracy of the information. Finally, the data can be stored in the database in a structured data format for subsequent process calls.
[0048] Step 2: Create an interface data collection task. Similarly, users can operate through the system's visual interface, including configuring the task name and task scheduling method (manual triggering and scheduled triggering, with scheduled triggering supporting cron configuration) in the basic call information, as well as configuring data processing information. After configuration, the scheduling service will be called to distribute the task to different nodes for execution, processing multiple interface data collection tasks in parallel, and displaying the overall data collection efficiency.
[0049] Step 3: Execute the interface data collection task through the interface input component and database output component. This includes configuring the interface input component and database output component, integrating the configuration information from the previous steps into them; using Camel to parse, extract, and convert the data format of the interface response results, and also supporting user-defined parameters for parsing the interface response results. Further, determine the data source fields, the target table fields in the target database, and the mapping relationships, and finally store the data in the target table fields of the target database.
[0050] Step 4: Exception Handling and Fault Tolerance. During the execution of the above data collection tasks, different types of exceptions are acquired in layers, such as network exceptions (e.g., connection timeout), business exceptions (e.g., authentication failure), and parsing exceptions (e.g., format errors). This provides a basis for subsequent processing and determines exception handling strategies. For example, for network exceptions, retries are executed using an exponential backoff algorithm (e.g., retry interval: 1s→2s→4s, with a maximum number of retries configured). Simultaneously, detailed exception information (including interface ID, exception type, and stack trace) is recorded. For severe exceptions, DingTalk / SMS alerts are triggered for timely response.
[0051] Step 5: Based on the database output component configured in Step 3, select the corresponding storage adapter (such as MySQL or PostgreSQL) and persist it to the corresponding database, i.e., the target database, through Mybatis. Synchronously record the task return results and notify users via in-site messages, SMS, DingTalk, etc., to form a complete closed loop.
[0052] Step 6: Task Monitoring. During the above process, metrics such as request latency, success rate, number of exceptions, CPU usage, and memory usage are collected in real time and stored in the database, then displayed through the system's visual interface.
[0053] The data acquisition and automatic database entry method provided in this application first configures the information for the API data acquisition task. During the configuration phase, the data hierarchy and nodes to be acquired are clearly defined to establish data extraction rules (such as the configured JSON path for the target data to be extracted), transformation rules (directly converting the target data into standard data suitable for database entry), and database entry rules (i.e., the mapping rules between the database source fields generated based on the data extraction information and the target table fields in the target database, as mentioned above). During task execution, the system automatically parses the result data returned by the interface and extracts and transforms the target data according to the preset rules. Simultaneously, the transformed data, combined with the database entry rules, is written into the target table fields in the target database. Furthermore, it supports configuring API call scheduling strategies, enabling efficient and stable execution of data acquisition tasks. It can efficiently handle API requests with multiple parameter combinations, significantly improving data acquisition efficiency, reducing manual operation costs, and achieving automated management of the entire API data acquisition process.
[0054] Based on the same inventive concept, this application also provides a data acquisition and automatic database entry system for implementing the above-described method embodiments. Figure 2 is a schematic diagram of the structure of the data acquisition and automatic database entry system provided in this embodiment of the invention. As shown in Figure 2, the system includes a configuration module 11, used to configure basic call information and data processing information for API interface data acquisition. The data processing information includes data extraction information generated based on data hierarchy and node information. A running module 12 is used to generate database source fields based on the data extraction information and establish mapping rules between the database source fields and the target table fields in the target database; call the API interface based on the basic call information to obtain the result data returned by the API interface; extract target data from the result data based on the data extraction information; process the target data, and write the processed data into the corresponding target table fields based on the database source fields and mapping rules corresponding to the target data.
[0055] Furthermore, in some embodiments of this application, the configuration module 11 can be a data source management module, responsible for the full lifecycle management of interface configurations and supporting local storage. It provides configuration verification and dynamic loading functions to ensure that configuration information is accurate and takes effect in real time; it also provides a visual interface and API for convenient user configuration management. The running module 12 can include a protocol adaptation module, a request processing module, a data parsing module, an exception handling module, a data storage module, and a monitoring log module.
[0056] The protocol adaptation module is used to achieve adaptive processing of multiple protocols and can consist of two parts: a protocol identifier and a request executor. The protocol identifier automatically determines the protocol type through configuration or URL; the request executor encapsulates the request logic for various protocols, implements underlying communication based on tools such as HttpClient and OkHttp, and supports synchronous / asynchronous modes.
[0057] The request processing module is used to handle the validation, encryption, and request header construction of request parameters. It adopts the interceptor pattern design and supports user-defined interceptors (such as adding signatures and verifying parameter tampering), which enhances the flexibility of request processing.
[0058] The data processing module is used to structure the data returned by the interface, and adopts a plug-in architecture. It has built-in JSON and XML processing plugins, providing data parsing, extraction, processing (i.e., format conversion) and field mapping functions, and outputting standardized data.
[0059] The exception handling module ensures system stability under abnormal scenarios and can be configured with sub-modules such as timeout control, retries, and degradation. Timeout control prevents requests from being blocked for a long time; the retry mechanism resolves transient exceptions; and the degradation strategy prevents persistent interface exceptions from affecting the overall system.
[0060] The data storage module stores standardized data into the target medium based on mapping relationships, supporting relational databases. Flexible switching between different storage media is achieved through the adapter pattern, and the underlying operation can be simplified using frameworks such as MyBatis and Spring Data.
[0061] The monitoring log module records information throughout the entire data collection process, outputs logs through the SLF4J+Logback framework, and supports multi-dimensional filtering and querying. It also collects key indicators and provides real-time monitoring and alarm functions to help users understand the system's operating status.
[0062] The data acquisition and automatic database entry system provided in this application features multi-protocol compatibility, dynamic configuration, easy expansion, and high operational stability. Specific advantages include: based on modular decomposition principles, the data acquisition process is broken down into seven independent modules: interface data source management, protocol adaptation, request processing, data parsing, exception handling, data storage, and monitoring logs, reducing coupling by clearly defining module boundaries; based on a strategy pattern, the request logic for different protocols is encapsulated, and different components are configured to match corresponding processing logic, avoiding redundant development and achieving multi-protocol adaptability; based on a dynamic configuration mechanism, interface parameters are managed via a database, and configuration updates do not require service restarts, improving operational flexibility; based on a plug-in parsing architecture, the system has built-in plugins for processing commonly used data formats and provides extended interfaces to support user-defined processing logic, adapting to diverse data formats; through mechanisms such as timeout control, exponential backoff retries, and circuit breaker degradation, end-to-end fault tolerance is achieved, ensuring system stability in abnormal scenarios; and the entire acquisition process logs and key indicators are recorded for a comprehensive monitoring system, supporting real-time monitoring and problem tracing, improving operational efficiency.
[0063] Based on the same inventive concept, this application also provides a data acquisition and automatic data entry device for implementing the above-described method embodiments. Figure 3 is a schematic diagram of the structure of the data acquisition and automatic data entry device provided in this embodiment. As shown in Figure 3, the data acquisition and automatic data entry device of this embodiment includes a processor 21 and a memory 22, with the processor 21 connected to the memory 22. The processor 21 is used to call and execute the program stored in the memory 22; the memory 22 is used to store the program, which is at least used to execute the data acquisition and automatic data entry method in the above embodiments.
[0064] The specific implementation scheme of the data acquisition and automatic warehousing device provided in this application embodiment can refer to the implementation scheme of the data acquisition and automatic warehousing method in any of the above embodiments, and will not be repeated here.
[0065] It is understood that the same or similar parts in the above embodiments can be referred to each other, and the contents not described in detail in some embodiments can be referred to the same or similar contents in other embodiments.
[0066] It should be noted that in the description of this invention, the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance. Furthermore, in the description of this invention, unless otherwise stated, "a plurality of" means at least two.
[0067] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process, and the scope of the preferred embodiments of the invention includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as will be understood by those skilled in the art to which embodiments of the invention pertain.
[0068] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.
[0069] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.
[0070] Furthermore, the functional units in the various embodiments of the present invention can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.
[0071] The storage media mentioned above can be read-only memory, disk, or optical disk, etc.
[0072] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.
[0073] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.
Claims
1. A method for data acquisition and automatic data entry, characterized in that, include: Configure the basic call information and data processing information for the API interface to collect data. The data processing information includes data extraction information generated based on data hierarchy and node information. Based on the data extraction information, database source fields are generated, and mapping rules are established between the database source fields and the target table fields in the target database; based on the basic call information, the API interface is called to obtain the result data returned by the API interface; based on the data extraction information, target data is extracted from the result data; The target data is processed, and based on the database source fields corresponding to the target data and the mapping rules, the processed data is written to the corresponding destination table fields.
2. The data acquisition and automatic data entry method according to claim 1, characterized in that, The basic call information includes: entry network environment, interface address, request method, authentication method, request parameters, return example, pagination configuration, and return structure.
3. The data acquisition and automatic data entry method according to claim 1, characterized in that, For JSON formatted data, the data hierarchy and node information are determined based on the jsonpath path.
4. The data acquisition and automatic data entry method according to claim 3, characterized in that, The step of generating database source fields based on the data extraction information and establishing mapping rules between the database source fields and the target table fields in the target database includes: renaming the jsonpath path information of the JSON format data to be extracted to obtain the database source fields; using the database source fields as the target table fields in the target database to create a new target table in the target database; or mapping and binding the database source fields with existing target table fields in the target database.
5. The data acquisition and automatic data entry method according to claim 1, characterized in that, The step of calling the API interface based on the basic call information and obtaining the result data returned by the API interface includes: triggering the API data collection task based on manual and timed triggering methods; distributing the data collection task to multiple nodes so that data collection can be performed in parallel by multiple nodes; wherein, when multiple nodes perform data collection in parallel, it runs based on predefined task dependencies and periodic rules.
6. The data acquisition and automatic data entry method according to claim 4, characterized in that, Based on the data extraction information, target data is extracted from the result data, including: parsing the result data to obtain the parsing result; and extracting data from the corresponding jsonpath path based on the data hierarchy and node information.
7. The data acquisition and automatic data entry method according to claim 6, characterized in that, The data processing information also includes data conversion rule information, and the processing of the target data includes: converting the extracted data into a target format based on the data conversion rule information.
8. The data acquisition and automatic data entry method according to claim 1, characterized in that, Also includes: Real-time collection of operational data and identification of anomaly information; wherein, the operational data includes request time, success rate, number of anomalies, CPU utilization, and memory utilization; the anomaly information includes network anomalies, business anomalies, and parsing anomalies; based on the anomaly information, an anomaly handling strategy is determined, and the operational status is displayed; wherein, the anomaly handling strategy includes retrying for network anomalies using an exponential backoff algorithm.
9. A data acquisition and automatic warehousing system, characterized in that, include: The configuration module is used to configure the basic call information and data processing information for API interface data collection. The data processing information includes data extraction information generated based on data hierarchy and node information. The running module is used to generate database source fields based on the extracted data, and to establish mapping rules between the database source fields and the target table fields in the target database; Based on the basic call information, the API interface is invoked to obtain the result data returned by the API interface; based on the data extraction information, the target data is extracted from the result data; The target data is processed, and based on the database source fields corresponding to the target data and the mapping rules, the processed data is written to the corresponding destination table fields.
10. A data acquisition and automatic warehousing device, characterized in that, The system includes a processor and a memory, wherein the processor is connected to the memory; the processor is used to call and execute a program stored in the memory; the memory is used to store the program, which is used at least to execute the data acquisition and automatic data entry method according to any one of claims 1-8.