Data integration method, device, electronic device and storage medium

Abstract

The present invention discloses a data integration method, device, electronic device, and storage medium. The method includes responding to a data source configuration instruction, configuring a first type of fields to be collected from a first type of data source that matches the data integration business, and configuring a second type of fields to be collected from a second type of data source; responding to a task definition instruction, defining a data processing task according to a predefined data structure; responding to a static data collection instruction, collecting field values ​​of the first type of fields to be collected from the first type of data source, and obtaining static data that matches the data integration business; responding to a dynamic data collection instruction, collecting field values ​​of the second type of fields to be collected from the second type of data source, and obtaining dynamic data that matches the data integration business; executing the data processing task to process the static data and dynamic data into a predefined data structure, and writing the processed data to a target storage medium to achieve data integration. The technical solution of the present invention improves the business coverage of the data integration method.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a data integration method, device, electronic equipment and storage medium. Background Art

[0002] With the continuous deepening of digital economic construction, there are more and more demands for sharing government data across industries, departments and fields. However, problems such as the inability of systems to communicate with each other, scattered data without aggregation, diverse and inconsistent data types, and approval and management of data sharing between systems have not been well resolved, resulting in inefficient digital economic construction in various fields. Summary of the Invention

[0003] The present invention provides a data integration method, device, electronic equipment and storage medium to improve the business coverage of the data integration method.

[0004] According to one aspect of the present invention, a data integration method is provided, the method comprising:

[0005] In response to the data source configuration instruction, a first type of fields to be collected are configured from a first type of data source that matches the data integration business, and a second type of fields to be collected are configured from a second type of data source;

[0006] Responding to task definition instructions, defining data processing tasks according to predefined data structures;

[0007] In response to the static data collection instruction, the field values ​​of the first type of to-be-collected fields in the first type of data source are collected to obtain static data matching the data integration business;

[0008] In response to the dynamic data collection instruction, the field values ​​of the second type of to-be-collected fields in the second type of data source are collected to obtain dynamic data matching the data integration business;

[0009] The data processing task is executed to process the static data and the dynamic data into the predefined data structure, and the processed data is written into a target storage medium to achieve data integration.

[0010] According to another aspect of the present invention, there is provided a data integration device, the device comprising:

[0011] A module for configuring fields to be collected, configured to respond to a data source configuration instruction, configure a first type of fields to be collected from a first type of data source that matches the data integration business, and configure a second type of fields to be collected from a second type of data source;

[0012] A task definition module is used to respond to task definition instructions and define data processing tasks according to predefined data structures;

[0013] A static data collection module, configured to respond to a static data collection instruction, collect field values ​​of the first type of to-be-collected fields in the first type of data source, and obtain static data matching the data integration business;

[0014] A dynamic data collection module, configured to respond to a dynamic data collection instruction, collect field values ​​of the second type of fields to be collected in the second type of data source, and obtain dynamic data matching the data integration business;

[0015] The data storage module is used to execute the data processing task to process the static data and the dynamic data into the predefined data structure, and write the processed data into the target storage medium to achieve data integration.

[0016] According to another aspect of the present invention, an electronic device is provided, comprising:

[0017] at least one processor; and

[0018] a memory communicatively connected to the at least one processor; wherein,

[0019] The memory stores a computer program that can be executed by the at least one processor. The computer program is executed by the at least one processor to enable the at least one processor to perform the data integration method described in any embodiment of the present invention.

[0020] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the data integration method described in any embodiment of the present invention when executed.

[0021] The technical solution of the embodiment of the present invention configures a first type of fields to be collected from a first type of data source that matches the data integration business, and configures a second type of fields to be collected from a second type of data source in response to a data source configuration instruction; defines a data processing task according to a predefined data structure in response to a task definition instruction; collects field values ​​of the first type of fields to be collected in the first type of data source in response to a static data collection instruction to obtain static data that matches the data integration business; collects field values ​​of the second type of fields to be collected in the second type of data source in response to a dynamic data collection instruction to obtain dynamic data that matches the data integration business; executes a data processing task to process static data and dynamic data into a predefined data structure, and writes the processed data into a target storage medium to achieve data integration. The technical means of collecting different field values ​​from different data sources to obtain static data and dynamic data required by the data integration business and processing the static data and dynamic data into a predefined data structure solves the problem that the prior art does not combine static data sources with dynamic data sources for integration and cannot meet specific business scenarios, thereby improving the business coverage of the data integration method.

[0022] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present invention, nor is it intended to limit the scope of the present invention. Other features of the present invention will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0024] Figure 1 A flowchart of a data integration method provided by an embodiment of the present invention;

[0025] Figure 2a A directed acyclic diagram of task scheduling is provided for an embodiment of the present invention;

[0026] Figure 2b A flow chart of a static data collection method provided by an embodiment of the present invention;

[0027] Figure 2c A flow chart of a dynamic data acquisition method provided by an embodiment of the present invention;

[0028] Figure 3 A schematic structural diagram of a data integration device provided by an embodiment of the present invention;

[0029] Figure 4It is a structural diagram of an electronic device for implementing the data integration method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0030] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0031] It should be noted that the terms "first", "second", etc. in the description and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0032] Figure 1 This is a flow chart of a data integration method provided by an embodiment of the present invention. This embodiment is applicable to the integration of data from multiple structured data sources. The method can be executed by a data integration device, which can be implemented in the form of hardware and / or software. The data integration device can be configured in a server integrated with a data integration platform. Figure 1 As shown, the method includes:

[0033] S110 . In response to the data source configuration instruction, configure a first type of fields to be collected from a first type of data source that matches the data integration business, and configure a second type of fields to be collected from a second type of data source.

[0034] The data integration platform of this embodiment can provide a human-computer interaction interface through which data source configuration instructions can be issued to the data integration platform. A data integration business can refer to an integration business that the data integration platform currently needs to execute. The first type of data source can refer to a data source used to provide static data. Static data can refer to data that is static during the data collection and processing time. The first type of data source can be, for example, a data source such as a database table or a non-relational database, where data does not change in the short term. The second type of data source can refer to a data source used to provide dynamic data. Dynamic data can refer to data that is subject to continuous change or continuous appending. The second type of data source can be a data source such as a database CDC (Change Data Capture) configuration or a message middleware, where data changes dynamically. The first type of fields to be collected can refer to fields that need to be collected from the first type of data source corresponding to the data integration business. The second type of fields to be collected can refer to fields that need to be collected from the second type of data source corresponding to the data integration business. The terms "first" and "second" can be used to distinguish similar objects and are not intended to limit their order.

[0035] In an optional implementation, before responding to the data source configuration instruction, it may also include: responding to the data source registration instruction, registering the first information of the first type of data source that matches the data integration business, and registering the second information of the second type of data source; responding to the data preview instruction, displaying the first type of field information of the first type of data source, and displaying the second type of field information of the second type of data source; the first type of data source is displayed after obtaining the access permission of the first type of data source based on the first information, and the second type of data source is displayed after obtaining the access permission of the second type of data source based on the second information.

[0036] The first information can refer to information that grants access rights to the first type of data source to the data integration platform, such as the data source's IP address, account number and password, port number, file format (only CSV, JSON, Excel, and Text are supported), and file path. The second information can refer to information that grants access rights to the second type of data source to the data integration platform, such as configuration information for message middleware topics, database CDC, and account number and password. Optionally, the data integration platform can encrypt and store key information (such as account number and password) for the first and second type of data sources.

[0037] In this embodiment, the relevant data sources can be determined based on the data integration business, and the data source party (i.e., the administrator of the data source) can be instructed to register the data source information on the data integration platform. In this case, the relevant data sources of the data integration business can grant the data source information to the administrator of the data integration platform, even if the administrator of the data integration platform can obtain access rights to the relevant data sources. Furthermore, the data integration platform can respond to the data preview instruction of its administrator and display the first category of field information of the first category of data sources and the second category of field information of the second category of data sources. The advantage of this setting is that it allows the administrator of the data integration platform to view the data source fields and their corresponding field values ​​before implementing the data integration business, making it easier to pre-check the data quality of the relevant data sources.

[0038] Based on the above optional implementation manner, the data integration method may further include sending a first prompt message to the responsible party of the first type of data source when the first type of field information is abnormal; and / or sending a second prompt message to the responsible party of the second type of data source when the second type of field information is abnormal.

[0039] In this embodiment, after previewing the data quality of the data source, a prompt message may be sent to the data source of data with obvious abnormalities to instruct the data source to perform quality maintenance on the data of the data source in a timely manner.

[0040] S120 , responding to the task definition instruction, defining the data processing task according to the predefined data structure.

[0041] The predefined data structure may refer to the final structure of the data after the data integration service is completed.

[0042] In this embodiment, the data processing task may be a task set, and the data integration platform may support multiple scripting languages ​​to define the data processing tasks.

[0043] S130 . Respond to the static data collection instruction, collect the field values ​​of the first type of to-be-collected fields in the first type of data source, and obtain static data matching the data integration business.

[0044] In an optional implementation, before responding to a static data collection instruction, it may also include: responding to a static data collection rule configuration instruction, configuring automatic collection rules for field values ​​of the first category of fields to be collected, and triggering an operation of responding to the static data collection instruction according to the automatic collection rules; or receiving a static data collection instruction sent by a user, triggering an operation of responding to the static data collection instruction.

[0045] The automatic collection rule may refer to a regular static data collection rule such as collecting data every five minutes or collecting data at 1 a.m. every day.

[0046] In this embodiment, since the data of the first type of data source is fixed and does not change in the short term, static data collection can generally be carried out according to fixed rules. For example, static data automatic collection rules can be pre-configured to trigger static data collection, or static data automatic collection rules can be not pre-configured, and static data collection can be manually triggered according to data business collection needs.

[0047] S140 . Respond to the dynamic data collection instruction, collect the field values ​​of the second type of to-be-collected fields in the second type of data source, and obtain dynamic data matching the data integration business.

[0048] In an optional implementation, before responding to the dynamic data collection instruction, the method may further include: receiving a data change notification sent by the second type of data source to trigger an operation of responding to the dynamic data collection instruction.

[0049] The data change notification may refer to notification information generated by the second type of data source itself and sent to the data integration platform when the data of the second type of data source itself changes.

[0050] In this embodiment, since the data in the second-category data source changes dynamically, dynamic data collection is performed based on real-time monitoring. It should be noted that in this embodiment, real-time monitoring of the second-category data source can be achieved through proactive notification from the second-category data source. That is, when the second-category data source changes its own data, it can promptly send a data change notification to the data integration platform. It is understood that the collection of dynamic data does not necessarily require regularity.

[0051] S150: Execute data processing tasks to process static data and dynamic data into a predefined data structure, and write the processed data into a target storage medium to achieve data integration.

[0052] In an optional embodiment, executing a data processing task to process static data and dynamic data into a predefined data structure may include: executing a data preprocessing task in the data processing task to perform abnormal data processing on the static data and dynamic data to obtain preprocessed static data and preprocessed dynamic data; executing a data conversion task in the data processing task to process the preprocessed static data and preprocessed dynamic data into a predefined data structure through mapping and calculation operations.

[0053] Data preprocessing tasks can be a set of preparation tasks to improve the usability of collected data, such as processing missing values, duplicate values, and compensation processing. Data conversion tasks can refer to a set of tasks such as mapping, conversion, and aggregation between collected data from different data sources.

[0054] In this embodiment, the collected static data and dynamic data can be cleaned and compensated, and other preparatory operations can be performed by executing data preprocessing tasks. The preprocessed static data and preprocessed dynamic data can be processed into predefined data structures through mapping and calculation operations (for example, mapping and aggregation of multiple source data sources into a temporary table, etc.), and then the processed data can be written into the target storage medium to complete the data integration business.

[0055] The technical solution of the embodiment of the present invention configures a first type of fields to be collected from a first type of data source that matches the data integration business, and configures a second type of fields to be collected from a second type of data source in response to a data source configuration instruction; defines a data processing task according to a predefined data structure in response to a task definition instruction; collects field values ​​of the first type of fields to be collected in the first type of data source in response to a static data collection instruction to obtain static data that matches the data integration business; collects field values ​​of the second type of fields to be collected in the second type of data source in response to a dynamic data collection instruction to obtain dynamic data that matches the data integration business; executes a data processing task to process static data and dynamic data into a predefined data structure, and writes the processed data into a target storage medium to achieve data integration. The technical means of collecting different field values ​​from different data sources to obtain static data and dynamic data required by the data integration business and processing the static data and dynamic data into a predefined data structure solves the problem that the prior art does not combine static data sources with dynamic data sources for integration and cannot meet specific business scenarios, thereby improving the business coverage of the data integration method.

[0056] Optionally, the data integration method of this embodiment may further include: generating snapshots for data collection, data processing, and data storage operations, and recording information of each link to provide real-time log query information.

[0057] To help those skilled in the art better understand the data integration platform of this embodiment, the following provides an introduction to the functional modules of the data integration platform:

[0058] 1. Data source module: registers the data source information (for example, information about databases, non-relational databases, database CDC configuration, message middleware, etc.).

[0059] 2. Data preview module: used to preview the fields and field values ​​of the data source. The operators of the data integration platform can view the data quality of the registered data sources.

[0060] 3. Static data collection module: According to the data source information registered by the data source module, configure one or more source data information to be collected (for example: database table fields, non-relational database fields, file paths, etc.).

[0061] 4. Dynamic data acquisition module: According to the data source information provided by the data source module, configure one or more source data information that needs to be collected (for example: database CDC configuration, message middleware, etc.).

[0062] 5. Data Scheduling Module: Configure the scheduling time for collecting static data sets (for example, every five minutes or at 1:00 AM every day).

[0063] 6. Data preprocessing module: Configure data cleaning and compensation rules to improve data availability (for example, define rules and compensation processing rules for handling missing values, duplicate values ​​and other abnormal values).

[0064] 7. Data conversion module: maps and converts multi-source heterogeneous data into predefined data structures (for example, mapping and aggregating multi-source data sources into a temporary table, etc.).

[0065] 8. Data storage module: stores the converted and integrated data in the target storage medium. (For example, storing the converted data in a relational database).

[0066] 9. Process link module: records information of each link in the process from the data acquisition module to the data storage module (for example: data collection time, data cleaning and compensation rules, target information of data storage, etc.).

[0067] 10. Snapshot recording module: records snapshots during the data collection, conversion, and storage process, thereby achieving fault tolerance in the process.

[0068] 11. Task scheduling module: The task scheduling module is nested in the data acquisition, preprocessing, conversion and storage modules. Multiple tasks can be defined in the task scheduling module. The module uses a directed acyclic graph (such as Figure 2a The tasks shown have dependencies: Task 3 and Task 4 depend on the processing results of Task 1, and Task 5 depends on the processing results of Task 2, Task 3, and Task 4) for task orchestration (for example: the data cleaning and compensation module supports multiple task orchestrations and tasks support multiple scripting languages, etc.).

[0069] In order to enable those skilled in the art to better understand the data integration method of this embodiment, the following describes the collection processes of static data and dynamic data separately based on the functional modules of the above-mentioned data integration platform.

[0070] Figure 2b A flow chart of a static data collection method provided by an embodiment of the present invention:

[0071] 1. The data source registers data source related information in the data source module, such as data source IP, database account password, port, file format (supports CSV, JSON, Excel, text, etc.), file path, etc., and key data source information (such as password) will be encrypted and stored.

[0072] 2. After the data source party completes the configuration of the data source related information, it can grant the data source information to the data users of the data integration platform for selection. The data users will have the authority to access the data source and preview its data.

[0073] 3. Users can configure multiple data sources and select the required field sets for each registered data source based on data integration business needs. They can also define pre-processing tasks and transformation tasks such as field mapping and aggregation operations for each data source.

[0074] 4. Based on the needs of the data integration business, use DataFront to configure data scheduling rules. If data scheduling rules are not configured, data collection must be triggered manually; if scheduling rules are configured, data collection is triggered automatically.

[0075] 5. After triggering data collection, a series of preprocessing operations, including cleaning and compensation, are performed on the collected data. These preprocessing operations are performed within the preprocessing task orchestration set. The data integration platform automatically generates snapshots of data collection and preprocessing operations, records operation details, and provides real-time log query information.

[0076] 6. After data preprocessing is complete, it will be transformed within the task orchestration set defined in the data transformation definition. The platform automatically generates snapshots of data transformation operations, records operation details, and provides real-time log query information.

[0077] 7. After the data conversion is complete, the data is written to the target storage medium. The platform can automatically generate snapshots for data storage operations, record operation information, and provide real-time log query information.

[0078] 8. Once the data is written to the target storage medium, the static data collection process of the entire data integration business is completed.

[0079] Figure 2c A flow chart of a dynamic data acquisition method provided by an embodiment of the present invention:

[0080] 1. The data source registers data source related information (message middleware topic and database CDC configuration, account and password) in the data source module, and key data source information (such as password) is encrypted and stored.

[0081] 2. After the data source completes the configuration of the data source-related information, it can grant the data source information to the data user for selection. The data user will then have the authority to access the data source and preview its historical data.

[0082] 3. Users can configure multiple data sources and determine whether to configure a static dataset. If so, preload the static dataset. Select the required field set for each data source as needed. Define preprocessing tasks and transformation tasks such as field mapping and aggregation operations for each data source.

[0083] 4. Data changes in dynamic data sources trigger collection operations, which then undergo a series of preprocessing operations, including cleaning and compensation. These preprocessing operations are performed within the defined task orchestration set. The platform automatically generates snapshots for data collection and preprocessing operations, records operational information, and provides real-time log query information.

[0084] 5. After data preprocessing is complete, it will be transformed within the task orchestration set defined in the data transformation definition. The data integration platform automatically generates snapshots for data collection and preprocessing operations, records operational information, and provides real-time log query information.

[0085] 6. After data conversion is complete, write the data to the target storage medium. The data integration platform can automatically generate snapshots for data collection and data preprocessing operations. The platform records operation information and provides real-time log query information.

[0086] 7. Once the data is written to the target storage medium, the dynamic data collection process of the entire data integration business is completed.

[0087] In order to enable those skilled in the art to better understand the data integration method of this embodiment, meteorological data integration is used as an example for description below.

[0088] For meteorological data integration, daily data collection involves collecting information such as temperature, humidity, and rainfall probability from sensors in designated areas. During this process, the platform can, for example, obtain two tables: one that maps sensor codes to sensor data, and this table's data can be dynamic; the other that maps sensor numbers to region codes, and this table's data can be static. The data integration service in this example essentially integrates these two tables into a single table.

[0089] Figure 3 This is a structural diagram of a data integration device provided by an embodiment of the present invention. Figure 3As shown, the device includes: a field configuration module 310 to be collected, a task definition module 320, a static data collection module 330, a dynamic data collection module 340 and a data storage module 350. Among them:

[0090] The to-be-collected fields configuration module 310 is configured to respond to a data source configuration instruction, configure a first type of to-be-collected fields from a first type of data source that matches the data integration business, and configure a second type of to-be-collected fields from a second type of data source;

[0091] The task definition module 320 is used to respond to the task definition instruction and define the data processing task according to the predefined data structure;

[0092] A static data collection module 330 is configured to respond to a static data collection instruction, collect field values ​​of the first type of to-be-collected fields in the first type of data source, and obtain static data matching the data integration business;

[0093] A dynamic data collection module 340 is configured to respond to a dynamic data collection instruction, collect field values ​​of the second type of to-be-collected fields in the second type of data source, and obtain dynamic data matching the data integration business;

[0094] The data storage module 350 is used to execute the data processing task to process the static data and the dynamic data into the predefined data structure, and write the processed data into the target storage medium to achieve data integration.

[0095] The technical solution of the embodiment of the present invention configures a first type of fields to be collected from a first type of data source that matches the data integration business, and configures a second type of fields to be collected from a second type of data source in response to a data source configuration instruction; defines a data processing task according to a predefined data structure in response to a task definition instruction; collects field values ​​of the first type of fields to be collected in the first type of data source in response to a static data collection instruction to obtain static data that matches the data integration business; collects field values ​​of the second type of fields to be collected in the second type of data source in response to a dynamic data collection instruction to obtain dynamic data that matches the data integration business; executes a data processing task to process static data and dynamic data into a predefined data structure, and writes the processed data into a target storage medium to achieve data integration. The technical means of collecting different field values ​​from different data sources to obtain static data and dynamic data required by the data integration business and processing the static data and dynamic data into a predefined data structure solves the problem that the prior art does not combine static data sources with dynamic data sources for integration and cannot meet specific business scenarios, thereby improving the business coverage of the data integration method.

[0096] Optionally, the data integration device further includes a field display module configured to:

[0097] In response to the data source registration instruction, register first information of a first type of data source matching the data integration service, and register second information of the second type of data source;

[0098] In response to the data preview instruction, display the first category of field information of the first category of data source, and display the second category of field information of the second category of data source;

[0099] The first type of data source is displayed after obtaining access rights to the first type of data source according to the first information, and the second type of data source is displayed after obtaining access rights to the second type of data source according to the second information.

[0100] Optionally, the data integration device further includes a prompt information sending module, configured to:

[0101] When the first-category field information is abnormal, sending a first prompt message to the party responsible for the first-category data source; and / or,

[0102] When the second-category field information is abnormal, a second prompt message is sent to a responsible party of the second-category data source.

[0103] Optionally, the data integration device further includes a static data collection triggering module configured to:

[0104] In response to the static data collection rule configuration instruction, configure an automatic collection rule for the field values ​​of the first type of fields to be collected, and trigger an operation in response to the static data collection instruction according to the automatic collection rule; or

[0105] Receive static data collection instructions sent by users and trigger operations that respond to static data collection instructions.

[0106] Optionally, the data integration device further includes a dynamic data collection triggering module configured to:

[0107] Receive the data change notification sent by the second type of data source, and trigger an operation of responding to the dynamic data collection instruction.

[0108] Optionally, the data storage module 350 may be used to:

[0109] Executing a data preprocessing task in the data processing task to perform abnormal data processing on the static data and the dynamic data to obtain preprocessed static data and preprocessed dynamic data;

[0110] The data conversion task in the data processing task is executed to process the preprocessed static data and the preprocessed dynamic data into the predefined data structure through mapping and operation operations.

[0111] Optionally, the data integration device further includes an operation recording module, configured to:

[0112] Generate snapshots for data collection, data processing, and data storage operations, and record information for each link to provide real-time log query information.

[0113] The data integration device provided in the embodiment of the present invention can execute the data integration method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0114] Figure 4 A schematic diagram of an electronic device 400 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers or various forms of mobile devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or claimed herein.

[0115] like Figure 4 As shown, the electronic device 400 includes at least one processor 401, and a memory connected to the at least one processor 401 in communication, such as a read-only memory (ROM) 402, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by the at least one processor, and the processor 401 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 402 or the computer program loaded from the storage unit 408 to the random access memory (RAM) 13. In RAM 403, various programs and data required for the operation of the electronic device 400 can also be stored. The processor 401, ROM 402 and RAM 403 are connected to each other via a bus 404. An input / output (I / O) interface 405 is also connected to the bus 404.

[0116] Multiple components in the electronic device 400 are connected to the I / O interface 405, including an input unit 406, such as a keyboard, a mouse, etc.; an output unit 407, such as various types of displays, speakers, etc.; a storage unit 408, such as a magnetic disk, an optical disk, etc.; and a communication unit 409, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 409 allows the electronic device 400 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0117] Processor 401 can be various general-purpose and / or specialized processing components with processing and computing capabilities. Some examples of processor 401 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. Processor 401 performs the various methods and processes described above, such as the data integration method.

[0118] In some embodiments, the data integration method can be implemented as a computer program that is tangibly contained in a computer-readable storage medium, such as storage unit 408. In some embodiments, part or all of the computer program can be loaded and / or installed on electronic device 400 via ROM 402 and / or communication unit 409. When the computer program is loaded into RAM 403 and executed by processor 401, one or more steps of the data integration method described above can be performed. Alternatively, in other embodiments, processor 401 can be configured to perform the data integration method in any other suitable manner (e.g., by means of firmware).

[0119] Various embodiments of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0120] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when the computer program is executed by the processor, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The computer program may be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0121] In the context of the present invention, computer-readable storage media can be tangible media that can contain or store a computer program for use with an instruction execution system, device or equipment or used in combination with an instruction execution system, device or equipment. Computer-readable storage media can include but are not limited to electronic, magnetic, optical, electromagnetic, infrared or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, computer-readable storage media can be machine-readable signal media. More specific examples of machine-readable storage media can include electrical connections based on one or more lines, portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0122] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0123] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0124] A computing system may include clients and servers. The clients and servers are typically remote from each other and typically interact via a communication network. This client-server relationship arises through computer programs running on the respective computers, creating a client-server relationship. The server may be a cloud server, also known as a cloud computing server or cloud host. This server is a hosting product within the cloud computing service ecosystem that addresses the management difficulties and limited scalability of traditional physical hosting and VPS services.

[0125] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in the present invention can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solution of the present invention can be achieved. This is not limited herein.

[0126] The above specific embodiments do not limit the scope of protection of the present invention. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention are intended to be included within the scope of protection of the present invention.

Claims

1. A data integration method, characterized in that: include: In response to the data source registration instruction, register first information of a first type of data source that matches the data integration business, and register second information of a second type of data source; In response to the data preview instruction, display the first category of field information of the first category of data source, and display the second category of field information of the second category of data source; The first type of data source is displayed after obtaining access rights to the first type of data source based on the first information, and the second type of data source is displayed after obtaining access rights to the second type of data source based on the second information; In response to the data source configuration instruction, a first type of fields to be collected is configured from a first type of data source that matches the data integration business, and a second type of fields to be collected is configured from a second type of data source; Responding to task definition instructions, defining data processing tasks according to predefined data structures; In response to the static data collection instruction, the field values ​​of the first type of to-be-collected fields in the first type of data source are collected to obtain static data matching the data integration business; In response to the dynamic data collection instruction, the field values ​​of the second type of to-be-collected fields in the second type of data source are collected to obtain dynamic data matching the data integration business; Executing a data preprocessing task in the data processing task to perform abnormal data processing on the static data and the dynamic data to obtain preprocessed static data and preprocessed dynamic data; Execute the data conversion task in the data processing task to process the preprocessed static data and the preprocessed dynamic data into the predefined data structure through mapping and operation operations, and write the processed data into the target storage medium to achieve data integration.

2. The method according to claim 1, characterized in that Also includes: When the first-category field information is abnormal, sending a first prompt message to the responsible party of the first-category data source; and / or, When the second-category field information is abnormal, a second prompt message is sent to a responsible party of the second-category data source.

3. The method according to claim 1, characterized in that Before responding to the static data collection instruction, it also includes: In response to the static data collection rule configuration instruction, configure an automatic collection rule for the field values ​​of the first type of fields to be collected, and trigger an operation in response to the static data collection instruction according to the automatic collection rule; or Receive static data collection instructions sent by users and trigger operations that respond to static data collection instructions.

4. The method according to claim 1, wherein Before responding to the dynamic data collection instruction, it also includes: Receive the data change notification sent by the second type of data source, and trigger an operation of responding to the dynamic data collection instruction.

5. The method according to claim 1, wherein Also includes: Generate snapshots for data collection, data processing, and data storage operations, and record information for each link to provide real-time log query information.

6. A data integration device, characterized in that: include: a field display module configured to respond to a data source registration instruction, register first information of a first-category data source that matches the data integration business, and register second information of a second-category data source; respond to a data preview instruction, display first-category field information of the first-category data source, and display second-category field information of the second-category data source; the first-category data source is displayed after obtaining access rights to the first-category data source based on the first information, and the second-category data source is displayed after obtaining access rights to the second-category data source based on the second information; A to-be-collected fields configuration module, configured to respond to a data source configuration instruction, configure a first type of to-be-collected fields from a first type of data source matching the data integration business, and configure a second type of to-be-collected fields from a second type of data source; A task definition module is used to respond to task definition instructions and define data processing tasks according to predefined data structures; A static data collection module, configured to respond to a static data collection instruction, collect field values ​​of the first type of to-be-collected fields in the first type of data source, and obtain static data matching the data integration business; A dynamic data collection module, configured to respond to a dynamic data collection instruction, collect field values ​​of the second type of fields to be collected in the second type of data source, and obtain dynamic data matching the data integration business; A data storage module is used to perform a data preprocessing task in the data processing task to perform abnormal data processing on the static data and the dynamic data to obtain preprocessed static data and preprocessed dynamic data; Execute the data conversion task in the data processing task to process the preprocessed static data and the preprocessed dynamic data into the predefined data structure through mapping and operation operations, and write the processed data into the target storage medium to achieve data integration.

7. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor. The computer program is executed by the at least one processor to enable the at least one processor to perform the data integration method according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the data integration method according to any one of claims 1 to 5 when executed.

Images