Data synchronization method and device, electronic equipment and storage medium

Abstract

The invention discloses a data synchronization method and device, electronic equipment and a storage medium, and relates to the technical field of data processing. The data synchronization method comprises the steps of obtaining a to-be-processed flow, and obtaining preset application programming interface configuration information according to identification information of the to-be-processed flow; wherein the preset application programming interface configuration information is configured based on a visual page; determining a network communication configuration template associated with the preset application programming interface configuration information, and determining a network communication operation according to the preset application programming interface configuration information and the network communication configuration template; obtaining return data corresponding to the to-be-processed flow based on the network communication operation; and determining the target data according to the returned data and the preset application programming interface configuration information, and storing the target data to the target database, so that the data of the API is automatically obtained and written into the target database, redevelopment of each data link is avoided, the development complexity is greatly reduced, and the use experience of a user is improved.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a data synchronization method, device, electronic device and storage medium. Background Art

[0002] With the rise of microservice architecture and the continuous improvement of the big data ecosystem, the demand for data exchange between multiple clusters, multiple systems, and multiple data warehouses continues to emerge. A typical new scenario is: accessing the Application Programming Interface (API), such as Restful API, Web Service, distributed service framework Dubbo, general remote procedure call (Google Remote Procedure Call, gRPC), etc., the interface obtains data, parses and converts the data, and writes it to the target database.

[0003] To effectively support this demand, data extraction, transformation, and loading (ETL) tools or customized data collection scripts are often combined, leveraging API calls and database writes to synchronize and store data across systems. However, with the growing demand for API-driven data sources, these tools have shown significant shortcomings in API interface support. Even if traditional ETL tools support APIs, their support typically remains limited to basic REST or SOAP calls, struggling to address complex authentication (such as OAuth2.0), dynamic parameter passing, paginated data retrieval, and interface version compatibility. This results in users needing to develop additional code to adapt to API interactions, or even failing to adapt at all. Furthermore, traditional ETL tools lack flexibility in parsing and transforming unstructured or semi-structured data (such as nested JSON) returned by APIs, making it difficult to efficiently handle complex data structures and often requiring manual scripting for additional transformations. Furthermore, the collection of different data links often requires customized scripts, making it difficult to uniformly manage execution status, error monitoring, and performance optimization across them. This fragmented approach not only increases the maintenance complexity of production environments but also poses significant challenges to the stability and reliability of data links.

[0004] Currently, data loading tools on the market generally lack API integration capabilities and are unable to handle the complex and ever-changing API interface protocols and data formats. In practice, custom development is often required to meet these requirements. Each data link implementation requires design and development from scratch, including interface calls, data conversion, and error handling logic. This results in long implementation cycles and high costs. Furthermore, coding styles and implementation methods may vary between developers, reducing system readability and scalability. Therefore, how to conveniently obtain API data and write it to the target database has become a pressing issue. Summary of the Invention

[0005] The present invention provides a data synchronization method, device, electronic device and storage medium to solve the problem in the prior art that it is impossible to conveniently obtain API data and write it into a target database.

[0006] According to one aspect of the present invention, a data synchronization method is provided, wherein the method comprises:

[0007] Obtaining a process to be processed, and obtaining preset application programming interface configuration information according to identification information of the process to be processed; wherein the preset application programming interface configuration information is based on a visual page configuration;

[0008] Determining a network communication configuration template associated with the preset application programming interface configuration information, and determining a network communication operation according to the preset application programming interface configuration information and the network communication configuration template;

[0009] Acquire return data corresponding to the process to be processed based on the network communication operation;

[0010] Target data is determined according to the returned data and the preset application programming interface configuration information, and the target data is stored in a target database.

[0011] According to another aspect of the present invention, there is provided a data synchronization device, characterized by comprising:

[0012] An information extraction module is used to obtain a process to be processed, and obtain preset application programming interface configuration information according to identification information of the process to be processed; wherein the preset application programming interface configuration information is based on a visual page configuration;

[0013] an operation determination module, configured to determine a network communication configuration template associated with the preset application programming interface configuration information, and determine a network communication operation according to the preset application programming interface configuration information and the network communication configuration template;

[0014] A data determination module, configured to obtain return data corresponding to the process to be processed based on the network communication operation;

[0015] The data storage module is used to determine target data according to the returned data and the preset application programming interface configuration information, and store the target data in a target database.

[0016] According to another aspect of the present invention, an electronic device is provided, comprising:

[0017] at least one processor; and

[0018] a memory communicatively connected to the at least one processor; wherein,

[0019] The memory stores a computer program that can be executed by the at least one processor. The computer program is executed by the at least one processor so that the at least one processor can perform a data synchronization method described in any embodiment of the present invention.

[0020] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement a data synchronization method according to any embodiment of the present invention when executed.

[0021] The technical solution of the embodiment of the present invention obtains the process to be processed, obtains the preset application programming interface configuration information according to the identification information of the process to be processed, determines the network communication configuration template associated with the preset application programming interface configuration information, determines the network communication operation according to the preset application programming interface configuration information and the network communication configuration template, obtains the return data corresponding to the process to be processed based on the network communication operation, determines the target data according to the return data and the preset application programming interface configuration information, stores the target data in the target database, and automatically obtains the API data and writes it into the target database, avoiding the need to redevelop the implementation of each data link; at the same time, the preset application programming interface configuration information is based on the visual page configuration, and the visual configuration page and predefined templates are used to greatly reduce the development complexity and improve the user experience.

[0022] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present invention, nor is it intended to limit the scope of the present invention. Other features of the present invention will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0024] Figure 1 This is a flow chart of a data synchronization method provided according to the first embodiment of the present invention;

[0025] Figure 2 This is a flow chart of a data synchronization method provided according to the second embodiment of the present invention;

[0026] Figure 3 This is a structural diagram of a data synchronization system provided according to a third embodiment of the present invention;

[0027] Figure 4 This is a flowchart of a data synchronization method provided according to Embodiment 3 of the present invention;

[0028] Figure 5 This is a flow chart of a method for configuring preset application programming interface configuration information provided in accordance with a third embodiment of the present invention;

[0029] Figure 6 This is a flowchart of an information monitoring method provided according to Embodiment 3 of the present invention;

[0030] Figure 7 This is a flow chart of a data insertion method provided according to the third embodiment of the present invention;

[0031] Figure 8 This is a structural diagram of a data synchronization device provided according to a fourth embodiment of the present invention;

[0032] Figure 9 It is a structural diagram of an electronic device for implementing a data synchronization method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0033] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0034] It should be noted that the terms "first", "second", etc. in the description and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0035] Example 1

[0036] Figure 1 This is a flow chart of a data synchronization method provided according to the first embodiment of the present invention. This embodiment is applicable to the case of automatically obtaining API return data and writing it into the target database. The method can be executed by a data synchronization device, which can be implemented in the form of hardware and / or software and can be configured in an electronic device. Figure 1 As shown, the method includes:

[0037] S110: Obtain a process to be processed, and obtain preset application programming interface configuration information according to identification information of the process to be processed.

[0038] Among them, the pending process refers to a series of ordered tasks that have not yet been executed or processed. Generally speaking, the pending process can be generated by business system user operation triggering, system timer task triggering, time piece triggering or interaction with external system triggering. In this embodiment, the pending process can be a data extraction, transformation, and loading (ETL) process. Identification information refers to information that can be used to distinguish the pending process. For example, the identification information can include but is not limited to the name, number and identity (Identification, id) of the pending process. The preset application programming interface configuration information refers to pre-configured API interface related information. For example, the preset application programming interface configuration information can include application programming interface configuration information access configuration, application programming interface configuration information parameter configuration, data processing configuration and target database configuration.

[0039] In one embodiment, the API configuration information access configuration may include, but is not limited to, the API interface Uniform Resource Locator (URL) / address, request method / interface method, authentication method, request header, etc.; the API configuration information parameter configuration may include, but is not limited to, request parameters, data format, paging mechanism, timeout setting, retry mechanism, error handling rules, load balancing strategy, etc.; the data processing configuration may include, but is not limited to, mapping and custom script rules; the target database configuration may include, but is not limited to, database type, database connection information, connection pool configuration, data table configuration, batch insert settings, data type mapping, transaction management, error handling strategy, encryption and security configuration, performance optimization configuration, and data synchronization mode, etc. In actual operation, the preset API configuration information can be configured based on a visual page, without the need to write code, significantly lowering the threshold for deployment and configuration.

[0040] In an embodiment, a process to be processed can be detected and extracted, identification information of the process to be processed can be determined, and the associated preset application programming interface configuration information can be found based on the identification information of the process to be processed. In actual operation, a pre-set task trigger center can monitor the process to be processed in real time and prepare to process the process after querying the preset application programming interface configuration information.

[0041] S120: Determine a network communication configuration template associated with the preset application programming interface configuration information, and determine a network communication operation according to the preset application programming interface configuration information and the network communication configuration template.

[0042] Among them, the network communication configuration template refers to a pre-configured template for configuring network communication operations. Different network communication configuration templates may exist for different preset application programming interface configuration information. In an embodiment, different network communication configuration templates may be created based on different protocols or frameworks. For example, different network communication configuration templates may be created for Restful API, WebService, Dubbo, and gRPC. Network communication operations refer to operations of network communication interactions. In actual operations, network communication operations may include Hypertext Transfer Protocol (http) requests and establishing communications.

[0043] In an embodiment, a network communication configuration template associated with preset application programming interface configuration information can be queried, and a network communication operation can be generated using the preset application programming interface configuration information and the network communication configuration template. In actual operation, specific preset application programming interface configuration information can be replaced with the corresponding position in the template to generate the network communication operation; alternatively, a network communication configuration template can be written in a programming language, and general request logic can be encapsulated through functions or classes to dynamically generate network communication operations based on the preset application programming interface configuration information.

[0044] S130. Obtain return data corresponding to the process to be processed based on the network communication operation.

[0045] The return data refers to the data fed back by the target application programming interface after receiving the network communication operation. In the actual operation process, the target application programming interface can be the application programming interface corresponding to the network communication operation, or the application programming interface stored in the preset application programming interface configuration information.

[0046] In an embodiment, data can be obtained from a target application programming interface corresponding to a network communication operation based on a network communication operation request, and the received data can be used as return data. In actual operation, the target address corresponding to the network communication operation, i.e., the target application programming interface, can be determined, and data can be queried based on the interface address in the network communication operation, and the response data content can be parsed to obtain the return data.

[0047] S140 , determining target data according to the returned data and preset application programming interface configuration information, and storing the target data in a target database.

[0048] In an embodiment, a data processing configuration in the preset application programming interface configuration information may be determined, and the returned data may be parsed using the data processing configuration to obtain target data. A target database and corresponding storage method may then be determined according to the target database configuration in the preset application programming interface configuration information, and the target data may be inserted into the target database. In actual operation, the returned data may be parsed and converted using the data processing configuration to conform to the structure and requirements of the target database, and the converted data may then be written to the target database.

[0049] In an embodiment of the present invention, by obtaining a process to be processed, obtaining preset application programming interface configuration information according to identification information of the process to be processed, determining a network communication configuration template associated with the preset application programming interface configuration information, determining a network communication operation according to the preset application programming interface configuration information and the network communication configuration template, obtaining return data corresponding to the process to be processed based on the network communication operation, determining target data according to the return data and the preset application programming interface configuration information, and storing the target data in a target database, the API data is automatically obtained and written into the target database, thereby avoiding the need to redevelop the implementation of each data link; at the same time, the preset application programming interface configuration information is based on a visual page configuration, and through the visual configuration page and predefined templates, the development complexity is greatly reduced, and the user experience is improved.

[0050] In one embodiment, the configuration process of the preset application programming interface configuration information includes:

[0051] Acquire access configuration information and parameter configuration information of the application programming interface input by the user through the visualization page, as well as target database configuration information;

[0052] The access configuration information is used as the application programming interface configuration information access configuration, the parameter configuration information is used as the application programming interface configuration information parameter configuration, and the target database configuration information is used as the target database configuration;

[0053] The data processing configuration associated with the configuration item parameters is extracted, and the application programming interface configuration information access configuration, the application programming interface configuration information parameter configuration, the data processing configuration and the target database configuration are used as the preset application programming interface configuration information.

[0054] Among them, access configuration information refers to configuration information for accessing the application programming interface, which is used to specify the request method for interaction and ensure secure access. Exemplarily, access configuration information may include, but is not limited to, the API interface URL / address, request method / interface method, authentication method, request header, etc. Parameter configuration information is used to specify information such as the parameter format for obtaining returned data. Exemplarily, parameter configuration information may include, but is not limited to, request parameters, data format, paging mechanism, timeout settings, retry mechanism, error handling rules, load balancing strategy, etc. Target database configuration information refers to configuration information for the target database. Exemplarily, target database configuration information may include, but is not limited to, database type, database connection information, connection pool configuration, data table configuration, batch insert settings, data type mapping, transaction management, error handling strategy, encryption and security configuration, performance optimization configuration, and data synchronization mode. Data processing configuration refers to pre-configured data processing rules. Exemplarily, data processing configuration may include, but is not limited to, mapping and custom script rules. In one embodiment, when a user writes a new data processing configuration, it can be stored through a visual page for subsequent application.

[0055] In an embodiment, the access configuration information, parameter configuration information and target database configuration information of the application programming interface input by the user through the visualization page are obtained, and the access configuration information is used as the application programming interface configuration information access configuration, the parameter configuration information is used as the application programming interface configuration information parameter configuration, and the target database configuration information is used as the target database configuration. The data processing configuration associated with the configuration item parameters is determined, and the application programming interface configuration information access configuration, the application programming interface configuration information parameter configuration, the data processing configuration and the target database configuration are used as the preset application programming interface configuration information. In the actual operation process, the visualization page can be a graphical interface, and options for each configuration parameter can be generated. The user can directly configure the corresponding preset application programming interface configuration information by clicking and selecting, which greatly reduces the complexity of development and allows non-technical personnel to quickly get started without writing code, significantly lowering the threshold for deployment and configuration.

[0056] In one embodiment, after determining target data according to the returned data and the preset application programming interface configuration information and storing the target data in the target database, the method further includes:

[0057] Determine the number of target data items successfully stored in the target database, and identify data synchronization anomalies based on the number of items.

[0058] When the abnormal situation is abnormal, the alarm operation is performed according to the preset alarm strategy.

[0059] The number of entries refers to the number of target data successfully stored in the target database, and the abnormality refers to the status of data synchronization. The preset alarm strategy can be understood as a pre-set alarm strategy. Generally speaking, the preset alarm strategy can trigger an alarm or terminate the operation when the abnormality is abnormal.

[0060] In an embodiment, the number of target data items successfully stored in the target database can be determined. If the number of successfully stored items is the same as the number of target data items, the data synchronization anomaly can be determined to be normal. If the number of successfully stored items is different from the number of target data items, the data synchronization anomaly can be determined to be abnormal. If the anomaly is abnormal, an alarm operation is executed according to a preset alarm policy.

[0061] In one embodiment, during the execution of the data synchronization task, corresponding log information can be left. For example, in order to facilitate the monitoring of the completion of the task, different alarm strategies can be set for different log information. For example, a task execution failure alarm can be set for the task start and end time, failure reason, error stack information, input and output data; a data quality problem alarm can be set for data validation rules, data exception records, and the number of error records; an API call failure alarm can be set for the API request URL, request parameters, response status code, error response content, and execution time; a data write database failure alarm can be set for database write operations, failed SQL statements, database error information, and transaction status; a system resource alarm can be set for resource usage, timestamps, and system load conditions; a task scheduling exception alarm can be set for the status of the scheduled task, the reason for scheduling failure, and the task execution log; an API return data exception alarm can be set for the API response content, the comparison between expected and actual data, and the abnormal data field; a scheduled task and scheduling failure alarm can be set for the execution status and failure reason of the scheduled task, so as to facilitate the execution monitoring of the pending process.

[0062] Example 2

[0063] Figure 2 This is a flow chart of a data synchronization method provided according to the second embodiment of the present invention. This embodiment is based on the above embodiment to further optimize and expand, and can be combined with various optional technical solutions in the above embodiment. Figure 2 As shown, the method includes:

[0064] S210: Based on the preset task triggering center, monitor and receive the process to be processed, and determine the identification information of the process to be processed.

[0065] The preset task triggering center may be a pre-set component responsible for discovering pending tasks and triggering corresponding processes.

[0066] In an embodiment, a preset task may be used to trigger the center to monitor and receive the process to be processed. After receiving the process to be processed, identification information of the process to be processed may be extracted.

[0067] S220: Search for preset application programming interface configuration information that matches the identification information.

[0068] The preset application programming interface configuration information includes at least: application programming interface configuration information access configuration, application programming interface configuration information parameter configuration, data processing configuration, and target database configuration. In an embodiment, the preset application programming interface configuration information matching the process to be processed can be queried and confirmed according to the identification information.

[0069] S230: Extracting a network communication configuration template associated with the preset application programming interface configuration information.

[0070] In an embodiment, an application programming interface configuration information access configuration and an application programming interface configuration information parameter configuration of preset application programming interface configuration information may be determined, and a network communication configuration template associated with the application programming interface configuration information access configuration and the application programming interface configuration information parameter configuration may be determined.

[0071] S240: Fill the application programming interface configuration information parameter configuration and the application programming interface configuration information access configuration in the preset application programming interface configuration information into the network communication configuration template to obtain a network communication operation.

[0072] The network communication operation at least includes a hypertext transfer protocol request and establishing communication.

[0073] In an embodiment, the preset application programming interface configuration information can be replaced with the corresponding position of the template to generate the network communication operation; alternatively, a network communication configuration template can be written in a programming language, and the general request logic can be encapsulated through functions or classes to dynamically generate the network communication operation based on the preset application programming interface configuration information. In actual operation, different network communication operations can be generated for different network communication configuration templates. Parameters such as the request header, request method, and parameter format can be configured using the network communication configuration template, and then personalized filling and adjustment can be performed based on the specific preset application programming interface configuration information.

[0074] S250: Establish a communication connection of a target application programming interface corresponding to the network communication operation based on the network communication operation.

[0075] In an embodiment, a communication connection with a target application programming interface may be established through a preset call handler.

[0076] S260: Access the target application programming interface through network communication operations to obtain task data, and use the task data as return data.

[0077] In an embodiment, an API request may be initiated to a target application programming interface with which a communication connection is established through a network communication operation to obtain task data, and the task data may be received as return data after successful acquisition.

[0078] S270: Determine the data processing configuration in the preset application programming interface configuration information, and parse the returned data according to the data processing configuration to obtain target data.

[0079] In an embodiment, the data processing configuration in the preset application programming interface configuration information can be extracted, and the returned data can be parsed using the data processing configuration to obtain the target data. In actual operation, when the data processing configuration is a mapping method, the field information of the returned data can be mapped to the location of the target database; when the data processing configuration is a custom script rule, the custom script rule can be executed to convert the returned data into a format suitable for database storage. In one embodiment, the returned data can be stored in a cache, and the data processing configuration can be executed by a pre-configured plug-in.

[0080] S280: Determine the target database based on the target database configuration in the preset application programming interface configuration information, and transmit and insert the target data into the target database.

[0081] In an embodiment, a target database corresponding to the target database configuration may be determined, and the target data may be transmitted to the target database and stored in a corresponding location in the target database.

[0082] An embodiment of the present invention monitors and receives a process to be processed based on a preset task triggering center, determines identification information of the process to be processed, searches for preset application programming interface configuration information matching the identification information, extracts a network communication configuration template associated with the preset application programming interface configuration information, fills the application programming interface configuration information parameter configuration and the application programming interface configuration information access configuration in the preset application programming interface configuration information into the network communication configuration template, obtains a network communication operation, establishes a communication connection of a target application programming interface corresponding to the network communication operation based on the network communication operation, accesses the target application programming interface through the network communication operation to obtain task data, uses the task data as return data, determines a data processing configuration in the preset application programming interface configuration information, parses the return data according to the data processing configuration to obtain target data, determines a target database based on the target database configuration in the preset application programming interface configuration information, transmits and inserts the target data into the target database, realizes convenient data synchronization, and greatly reduces development complexity and improves the flexibility of data parsing through predefined templates, so as to facilitate rapid adaptation to new API interfaces and business requirements.

[0083] Example 3

[0084] Figure 3 This is a structural diagram of a data synchronization system provided according to the third embodiment of the present invention. This embodiment is based on the above embodiment and takes the external system API as the target application programming interface as an example to further illustrate a data synchronization method. Figure 3 As shown, this system includes two core roles: Controller role and Executor role, which work together to ensure efficient execution and flexible configuration of data links.

[0085] The Controller role is responsible for: receiving user-configured data loading processes and storing them in the database; viewing operation and maintenance information and providing an entry point for operation and maintenance operations; configuring alarm rules and processing information; and managing data processing plug-ins and data source processing plug-ins.

[0086] The Executor role is responsible for: executing processes based on the configuration stored by the controller role; recording and processing runtime log information and analyzing whether alarms are needed; accepting operation and maintenance instructions; dynamically loading and unloading plug-ins, and using plug-ins.

[0087] The Controller role includes configuration management, plug-in management, and operation and maintenance functions. Configuration management includes data loading process configuration, alarm policy configuration, and operation and maintenance policy configuration; plug-in management includes data conversion plug-in management and database plug-in management; and operation and maintenance functions include operation and maintenance data display and operation and maintenance operations.

[0088] The Executor role includes a cache data center, a runtime core, and external interaction interfaces. The cache data center includes the data cache for the data loading process and the cache data center. The runtime core includes data loading process control, an alarm indicator calculator, a task triggering center, a data processing executor, a database interaction executor, and an API call processor. The external interaction interface includes the operation and maintenance portal and the alarm notification integration center.

[0089] The Controller role can notify the Executor role's cache data center of cache updates, updating the data in the configuration storage. The Executor role can read data and store it in the cache. The API call processor can query external system APIs, and the database interaction executor can insert data into the target database. The Alarm Metric Calculator can store batch metrics in the cache and send asynchronous alarms to the Alarm Notification Integration Center, sending alarms to the alarm receiver.

[0090] In one embodiment, Figure 4 This is a flow chart of a data synchronization method provided according to the third embodiment of the present invention. Figure 4 As shown, the method includes:

[0091] The task trigger center discovers the pending process and notifies the data loading controller to start the process. The data loading process controller searches the cache for configuration and prepares to process the process. The API call processor is notified to prepare to process the API. The API call processor generates the corresponding HTTP request based on the configuration or prepares to establish communication. The call processor successfully requests the external system API to obtain data and obtains the return data. The API call processor returns the return data to the loading process controller. The loading process controller calls the data processing executor to prepare for data parsing. The data processing executor searches for the plugin specified in the configuration to perform the parsing task. The data parsing processor returns the parsed data to the process controller. The process controller calls the database interaction executor to prepare to write the parsed data to the target database. The database interaction executor searches for the plugin specified in the configuration to perform the data insertion task. The number of successfully inserted data entries is returned to the process controller, and the process processing ends, completing the current process and waiting for the next trigger. The loading process controller calls the data processing executor to prepare for data parsing, the data processing executor searches for the plugin specified in the configuration to perform the parsing task, and the data parsing processor returns the parsed data to the process controller. This process may be repeated multiple times.

[0092] In one embodiment, Figure 5 This is a flow chart of a configuration method for preset application programming interface configuration information according to the third embodiment of the present invention. Figure 5As shown, the configuration method of the preset application programming interface configuration information includes API access configuration, API parameter configuration, data processing configuration and database configuration.

[0093] The contents of API access configuration and API parameter configuration include the contents shown in Table 1 according to different protocols.

[0094] Table 1

[0095] Configuration items Restful API Web Services Dubbo gRPC API interface URL / address Required Required Required Required Request method / interface method Required Required Required Required Authentication method Required Required Required Required Request header Required Required Required Required Request Parameters Required Required Required Required Data format Required Required Required Required Paging mechanism Optional Optional Optional Optional Timeout settings Optional Optional Optional Optional Retry mechanism Optional Optional Optional Optional Error handling rules Optional Optional Optional Optional Load balancing strategy Optional - Optional -

[0096] The database configuration includes the content shown in Table 2.

[0097] Table 2

[0098]

[0099] The above is the complete configuration content of the data loading process. After completing the above configuration, the user can open a data loading link without the need for custom development.

[0100] In one embodiment, an alarm policy can be configured in the controller. The executor will leave corresponding log information during the execution of tasks in the data loading process. The supported information is shown in Table 3:

[0101] Table 3

[0102]

[0103]

[0104] In one embodiment, Figure 6 This is a flow chart of an information monitoring method provided according to the third embodiment of the present invention. Figure 6 The following are the core steps of monitoring information. The method includes: generating data processing flow logs, reading log data through asynchronous threads, classifying and grading log data based on monitoring indicator definitions, and performing operations such as notification, alarm, termination, and recording.

[0105] In one embodiment, Figure 7 This is a flow chart of a data insertion method provided according to the third embodiment of the present invention. Figure 7 Use processes for plugins, such as Figure 7 As shown, the data loading process requires a plug-in. You can check whether the current plug-in is loaded. If not, the binary plug-in file is read from the cache, the plug-in is compiled and loaded dynamically, and the loaded plug-in is placed in the cache. If it is, the plug-in is directly obtained from the cache to perform data conversion tasks or database insertion tasks.

[0106] The system of this embodiment has built-in support for multiple API authentication methods (such as OAuth2.0, API keys), can flexibly configure parameterized requests and paging mechanisms, and automatically handle complex data parsing (such as nested JSON and dynamic fields); at the same time, it provides highly automated data processing, providing a one-stop solution from data extraction to cleaning, conversion, and loading. Through a visual configuration interface and predefined templates, it greatly reduces the complexity of development, allowing non-technical personnel to quickly get started. To achieve unified monitoring and operation and maintenance, the system has built-in unified monitoring and alarm functions, which can track the execution status of data collection tasks in real time, automatically record error logs and generate performance analysis reports, providing strong guarantees for the efficient operation and maintenance of data links. In addition, it supports multiple data sources and target library types, has a powerful plug-in mechanism, can quickly adapt to new API interfaces and business requirements, meet the continuous expansion needs of enterprises in dynamic environments, and are familiar with the dynamic expansion capabilities of the plug-in mechanism. The system provides a page-based configuration interface, and users can configure data collection tasks through a graphical interface without writing code, which significantly reduces the threshold for deployment and configuration.

[0107] Example 4

[0108] Figure 8 1 is a schematic diagram of the structure of a data synchronization device provided according to the fourth embodiment of the present invention. Figure 8 As shown, the device includes: an information extraction module 81, an operation determination module 82, a data determination module 83 and a data storage module 84.

[0109] The information extraction module 81 is used to obtain the process to be processed and obtain the preset application programming interface configuration information according to the identification information of the process to be processed; wherein the preset application programming interface configuration information is based on the visual page configuration;

[0110] an operation determining module 82, configured to determine a network communication configuration template associated with the preset application programming interface configuration information, and determine a network communication operation based on the preset application programming interface configuration information and the network communication configuration template;

[0111] A data determination module 83 is configured to obtain return data corresponding to the process to be processed based on the network communication operation;

[0112] The data storage module 84 is configured to determine target data according to the returned data and the preset application programming interface configuration information, and store the target data in a target database.

[0113] In an embodiment of the present invention, an information extraction module is used to obtain a process to be processed, preset application programming interface configuration information is obtained according to identification information of the process to be processed, an operation determination module determines a network communication configuration template associated with the preset application programming interface configuration information, and determines a network communication operation according to the preset application programming interface configuration information and the network communication configuration template. A data determination module obtains return data corresponding to the process to be processed based on the network communication operation, a data storage module determines target data according to the return data and the preset application programming interface configuration information, and stores the target data in a target database, thereby automatically obtaining API data and writing it into the target database, avoiding the need to redevelop each data link. At the same time, the preset application programming interface configuration information is configured based on a visual page. Through the visual configuration page and predefined templates, the development complexity is greatly reduced, and the user experience is improved.

[0114] In one embodiment, the information extraction module 81 includes:

[0115] An identification information acquisition unit, configured to trigger a center to monitor and receive a process to be processed based on a preset task, and determine identification information of the process to be processed;

[0116] A configuration extraction unit is used to search for preset application programming interface configuration information that matches the identification information; wherein the preset application programming interface configuration information at least includes: application programming interface configuration information access configuration, application programming interface configuration information parameter configuration, data processing configuration and target database configuration.

[0117] In one embodiment, a data synchronization device further includes:

[0118] A data reading module is used to obtain access configuration information and parameter configuration information of the application programming interface input by the user through the visual page and target database configuration information;

[0119] a configuration generation module, configured to use the access configuration information as application programming interface configuration information access configuration, use the parameter configuration information as application programming interface configuration information parameter configuration, and use the target database configuration information as target database configuration;

[0120] The information generation module is not used to extract the data processing configuration associated with the configuration item parameters, and uses the application programming interface configuration information access configuration, the application programming interface configuration information parameter configuration, the data processing configuration and the target database configuration as preset application programming interface configuration information.

[0121] In one embodiment, the operation determination module 82 includes:

[0122] a template extraction unit, configured to extract a network communication configuration template associated with the preset application programming interface configuration information;

[0123] An operation determination unit is used to fill the application programming interface configuration information parameter configuration and application programming interface configuration information access configuration in the preset application programming interface configuration information into the network communication configuration template to obtain a network communication operation; wherein, the network communication operation at least includes a hypertext transfer protocol request and establishing communication.

[0124] In one embodiment, the data determination module 83 includes:

[0125] a communication connection unit, configured to establish a communication connection of a target application programming interface corresponding to the network communication operation based on the network communication operation;

[0126] The data returning unit is configured to access the target application programming interface through the network communication operation to obtain task data, and use the task data as return data.

[0127] In one embodiment, the data storage module 84 includes:

[0128] a data parsing unit, configured to determine a data processing configuration in the preset application programming interface configuration information, and parse the returned data according to the data processing configuration to obtain target data;

[0129] The data storage unit is configured to determine a target database based on the target database configuration in the preset application programming interface configuration information, and to store and insert the target data into the target database.

[0130] In one embodiment, a data synchronization device further includes:

[0131] An abnormality judgment module is used to determine the number of target data successfully stored in the target database, and determine the abnormality of data synchronization according to the number of target data;

[0132] The abnormal alarm module is used to perform an alarm operation according to a preset alarm strategy when the abnormal situation is abnormal.

[0133] A data synchronization device provided by an embodiment of the present invention can execute a data synchronization method provided by any embodiment of the present invention, and has corresponding functional modules and beneficial effects of the execution method.

[0134] Example 5

[0135] Figure 91 is a schematic diagram of the structure of an electronic device that implements a data synchronization method of an embodiment of the present invention. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or claimed herein.

[0136] like Figure 9 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., which is communicatively connected to the at least one processor 11. The memory stores a computer program that can be executed by the at least one processor. The processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 into the random access memory (RAM) 13. Various programs and data required for the operation of the electronic device 10 can also be stored in the RAM 13. The processor 11, ROM 12, and RAM 13 are connected to each other via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0137] Multiple components in the electronic device 10 are connected to the I / O interface 15, including an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a magnetic disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0138] The processor 11 can be any general-purpose and / or specialized processing component with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 executes the various methods and processes described above, such as a data synchronization method.

[0139] In some embodiments, a data synchronization method may be implemented as a computer program tangibly embodied in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of a data synchronization method X described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to perform a data synchronization method in any other suitable manner (e.g., via firmware).

[0140] Various embodiments of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system comprising at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0141] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when the computer program is executed by the processor, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The computer program may be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0142] In the context of the present invention, computer-readable storage media can be tangible media that can contain or store a computer program for use with an instruction execution system, device or equipment or used in combination with an instruction execution system, device or equipment. Computer-readable storage media can include but are not limited to electronic, magnetic, optical, electromagnetic, infrared or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, computer-readable storage media can be machine-readable signal media. More specific examples of machine-readable storage media can include electrical connections based on one or more lines, portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0143] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0144] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0145] A computing system may include clients and servers. The clients and servers are typically remote from each other and typically interact via a communication network. This client-server relationship arises through computer programs running on the respective computers, creating a client-server relationship. The server may be a cloud server, also known as a cloud computing server or cloud host. This server is a hosting product within the cloud computing service ecosystem that addresses the management difficulties and limited scalability of traditional physical hosting and VPS services.

[0146] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in the present invention can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solution of the present invention can be achieved. This is not limited herein.

[0147] The above specific embodiments do not limit the scope of protection of the present invention. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention are intended to be included within the scope of protection of the present invention.

Claims

1. A data synchronization method, characterized in that: include: Obtaining a process to be processed, and obtaining preset application programming interface configuration information according to identification information of the process to be processed; wherein the preset application programming interface configuration information is based on a visual page configuration; Determining a network communication configuration template associated with the preset application programming interface configuration information, and determining a network communication operation according to the preset application programming interface configuration information and the network communication configuration template; Acquire return data corresponding to the process to be processed based on the network communication operation; Target data is determined according to the returned data and the preset application programming interface configuration information, and the target data is stored in a target database.

2. The method according to claim 1, characterized in that The step of obtaining the process to be processed, and obtaining preset application programming interface configuration information according to the identification information of the process to be processed, includes: The center monitors and receives the pending process based on the preset task triggering, and determines the identification information of the pending process; Search for preset application programming interface configuration information that matches the identification information; wherein the preset application programming interface configuration information at least includes: application programming interface configuration information access configuration, application programming interface configuration information parameter configuration, data processing configuration and target database configuration.

3. The method according to claim 1, characterized in that The configuration process of the preset application programming interface configuration information includes: Acquire access configuration information and parameter configuration information of the application programming interface input by the user through the visualization page, as well as target database configuration information; Using the access configuration information as application programming interface configuration information access configuration, using the parameter configuration information as application programming interface configuration information parameter configuration, and using the target database configuration information as target database configuration; The data processing configuration associated with the configuration item parameters is extracted, and the application programming interface configuration information access configuration, the application programming interface configuration information parameter configuration, the data processing configuration and the target database configuration are used as preset application programming interface configuration information.

4. The method according to claim 1, wherein The determining of a network communication configuration template associated with the preset application programming interface configuration information, and determining a network communication operation according to the preset application programming interface configuration information and the network communication configuration template, includes: extracting a network communication configuration template associated with the preset application programming interface configuration information; The application programming interface configuration information parameter configuration and the application programming interface configuration information access configuration in the preset application programming interface configuration information are filled into the network communication configuration template to obtain a network communication operation; wherein, the network communication operation at least includes a hypertext transfer protocol request and establishing communication.

5. The method according to claim 1, wherein The obtaining, based on the network communication operation, return data corresponding to the process to be processed includes: establishing, based on the network communication operation, a communication connection of a target application programming interface corresponding to the network communication operation; The target application programming interface is accessed through the network communication operation to obtain task data, and the task data is used as return data.

6. The method according to claim 1, characterized in that Determining target data according to the returned data and the preset application programming interface configuration information, and storing the target data in a target database, comprises: Determining a data processing configuration in the preset application programming interface configuration information, and parsing the returned data according to the data processing configuration to obtain target data; A target database is determined based on the target database configuration in the preset application programming interface configuration information, and the target data is transmitted and inserted into the target database.

7. The method according to claim 1, characterized in that After determining the target data according to the returned data and the preset application programming interface configuration information and storing the target data in the target database, the method further includes: Determine the number of target data items successfully stored in the target database, and determine an abnormality in data synchronization according to the number of target data items; When the abnormal situation is abnormal, an alarm operation is performed according to a preset alarm strategy.

8. A data synchronization device, characterized in that: include: An information extraction module is used to obtain a process to be processed, and obtain preset application programming interface configuration information according to identification information of the process to be processed; wherein the preset application programming interface configuration information is based on a visual page configuration; an operation determination module, configured to determine a network communication configuration template associated with the preset application programming interface configuration information, and determine a network communication operation according to the preset application programming interface configuration information and the network communication configuration template; A data determination module, configured to obtain return data corresponding to the process to be processed based on the network communication operation; The data storage module is used to determine target data according to the returned data and the preset application programming interface configuration information, and store the target data in a target database.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor. The computer program is executed by the at least one processor to enable the at least one processor to perform a data synchronization method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement a data synchronization method according to any one of claims 1 to 7 when executed.

Images