Method, device, electronic device and storage medium for data management

Abstract

A method, apparatus, device, and storage medium for data management are provided. The data management method includes: obtaining configuration information of a database instance, the database instance including at least one data source; accessing at least one raw data layer based on the configuration information, each raw data layer including description information of a data source of the database instance; accessing at least one operational data storage layer corresponding to the at least one raw data layer, each operational data storage layer including processing rules for a data source of the database instance; constructing a data access task corresponding to the database instance based on the at least one raw data layer and the at least one operational data storage layer; and initiating an incremental data task for the database instance based on the data access task. By reusing the data access task of a database instance, access to at least one data source in the database instance can be achieved, which is beneficial for efficient data access to data sources in the database instance.

Description

Technical Field

[0001] This application relates to the field of computers, and more specifically, to methods, apparatus, electronic devices, and storage media for data management. Background Technology

[0002] Data warehouses require data to be collected from business databases to achieve centralized data storage and integrated computing capabilities. Typically, industry practice uses timed full / incremental data access, which has a significant drawback: the timeliness of data entering the data warehouse is very low (usually T+1). To address this issue, a new technical solution has emerged: Change Data Capture (CDC). This directly reads change logs from the business database and applies corresponding changes downstream, enabling real-time data access to the data warehouse.

[0003] Data access needs to ensure the real-time nature and accuracy of the data while minimizing the impact on upstream business databases. For a data platform, which serves as the data center for the entire data analysis process, there is a significant amount of data access work. How to efficiently perform data access is a pressing issue that needs to be addressed. Summary of the Invention

[0004] This application provides a data management method, apparatus, device, and storage medium that facilitates efficient data access to data sources in a database instance.

[0005] Firstly, a data management method is provided, including:

[0006] Obtain the configuration information of a database instance, wherein the database instance includes at least one data source;

[0007] Based on the configuration information, at least one raw data layer is accessed, and each raw data layer includes description information of a data source of the database instance;

[0008] Access at least one operational data storage layer corresponding to the at least one raw data layer, each operational data storage layer including processing rules for a data source of the database instance;

[0009] Based on the at least one raw data layer and the at least one operational data storage layer, construct the data access task corresponding to the database instance;

[0010] Based on the data access task, initiate the incremental data task for the database instance.

[0011] Secondly, a data management apparatus is provided, comprising:

[0012] The acquisition unit is used to acquire configuration information of a database instance, wherein the database instance includes at least one data source;

[0013] An access unit is configured to access at least one raw data layer according to the configuration information, wherein each raw data layer includes description information of a data source of the database instance;

[0014] The access unit is also used to access at least one operational data storage layer corresponding to the at least one raw data layer, and each operational data storage layer includes processing rules for a data source of the database instance;

[0015] A construction unit is used to construct a data access task corresponding to the database instance based on the at least one raw data layer and the at least one operational data storage layer;

[0016] The startup unit is used to start the incremental data task of the database instance according to the data access task.

[0017] Thirdly, this application provides an electronic device, comprising:

[0018] Processor, adapted to implement computer instructions; and,

[0019] A memory that stores computer instructions adapted for loading by a processor and executing the method described in the first aspect above.

[0020] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer instructions that, when read and executed by a processor of a computer device, cause the computer device to perform the method described in the first aspect.

[0021] Fifthly, embodiments of this application provide a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the method described in the first aspect.

[0022] Based on the above technical solutions, the embodiments of this application can achieve access to at least one data source in a database instance by reusing the data access task of a single database instance. This allows for data access architecture design using the database instance as the synchronization unit, which is beneficial for efficient data access to data sources within the database instance. For example, a single change log can support change capture and distribution from multiple data tables. Attached Figure Description

[0023] Figure 1 This is a schematic diagram illustrating an application scenario according to an embodiment of this application;

[0024] Figure 2 This is a schematic flowchart illustrating a data management method according to an embodiment of this application;

[0025] Figure 3 This is a schematic flowchart illustrating another data management method according to an embodiment of this application;

[0026] Figure 4 This is a schematic diagram of an optional data access process according to an embodiment of this application;

[0027] Figure 5 This is a schematic diagram of an optional data distribution architecture according to an embodiment of this application;

[0028] Figure 6 A schematic block diagram of a data management device provided for embodiments of this application;

[0029] Figure 7 This is a schematic block diagram of the electronic device provided in the embodiments of this application. Detailed Implementation

[0030] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0031] First, the application scenarios of the embodiments of this application will be described.

[0032] Figure 1 This is a schematic diagram of an application scenario involved in an embodiment of this application.

[0033] like Figure 1 As shown, the system includes a data management platform 100, which includes a data access module 110 and a data center 120. The data access module 110 can be used to access data from the business database, and the data center 120 can be used to store the data accessed by the data access module 110.

[0034] The data management platform 110, for example, is a real-time data management platform, which includes a development interface for real-time data access and management, and can integrate data access, management and computing into one.

[0035] For example, the data management platform 100 can be deployed on a server. The server can be one or more servers. When there are multiple servers, at least two servers are used to provide different services, and / or at least two servers are used to provide the same service, such as providing the same service in a load-balanced manner. This application embodiment does not limit this.

[0036] The server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms. A server can also become a node in a blockchain.

[0037] Optionally, the server can also display the management interface of the data management platform 100. For example, administrators can use this management page to monitor and manage various aspects of data access and storage.

[0038] In related technologies, existing data access solutions are mainly based on real-time database synchronization tools. Currently, there are two main types of commonly used technical solutions on the market:

[0039] (a) Using open-source data synchronization tools: debezium / canal / maxwell. These data synchronization tools have similar architectures and principles, namely, simulating incremental synchronization of data changes by a database (DB) slave. For existing data, an additional data dumping tool is required. Incremental data is generally sent to a message middleware, and downstream computations access the message middleware for data consumption and processing.

[0040] (ii) Using the data synchronization tool: Flink-CDC. As one of the data sources for Flink-SQL (Structured Query Language), Flink-CDC integrates data acquisition and computation, processing data on the acquisition side and significantly shortening the data synchronization chain. Furthermore, Flink-CDC internally supports both incremental and full data processing, making it a convenient method for single-table synchronous computation.

[0041] However, both of the above solutions have some drawbacks. The main disadvantages of Solution (I) are: 1. Lack of a flexible method for synchronizing existing data; 2. Difficulty in interfacing with downstream computation, requiring an additional conversion layer to meet application requirements, resulting in a long data synchronization link. Solution (I) is suitable for scenarios requiring incremental synchronization. Solution (II) integrates data acquisition and computation, shortening the data synchronization link, but still has some problems: 1. Lack of a flexible method for synchronizing existing data; 2. Unstable performance when pulling existing data from large tables; 3. Inadequate support for synchronizing multiple tables within the same database. Solution (II) is suitable for single-table synchronization computation, exploratory scenarios, or small table association computation tasks in a production environment.

[0042] For data management platforms (such as data middleware), there is a great demand for real-time synchronous collection (full + incremental) of data tables. Therefore, there is a need for an efficient data access solution to support data synchronization scenarios with large data volumes and many data tables.

[0043] Based on this, embodiments of this application provide a data management method, apparatus, device, and storage medium. By accessing at least one raw data layer and at least one operational data storage layer according to the configuration information of a database instance, each raw data layer includes description information of a data source for the database instance, and each operational data storage layer includes processing rules for a data source of the database instance. Then, based on the at least one raw data layer and the at least one operational data storage layer, a data access task corresponding to the database instance is obtained. This allows incremental data tasks for the database instance to be initiated based on the data access task, thereby achieving access to at least one data source in the database instance. Therefore, embodiments of this application can achieve access to at least one data source in a database instance by reusing the data access task of a single database instance, which is beneficial for efficient data access to data sources in a database instance.

[0044] Specific embodiments of this application are described below with reference to the accompanying drawings.

[0045] Figure 2 This illustration shows a schematic flowchart of a data management method 200 provided in an embodiment of this application. The method 200 can be executed by any electronic device with data processing capabilities; for example, the electronic device can be implemented as a server, such as a data management platform on the server, etc., and this application does not limit this. Figure 2 As shown, method 200 includes steps 210 to 250.

[0046] 210. Obtain the configuration information of the database instance, which includes at least one data source.

[0047] The database instance may include the data source that needs to be accessed, such as one or more data sources. The data source may be, for example, a raw data table, which is not limited in this application.

[0048] For example, users can register the configuration information of database instances that need to access data on the data management platform. This allows the data management platform to obtain the configuration information of the database instance and reuse it when repeatedly accessing data sources within the same database instance. This configuration information can be used to obtain the data source of the database instance.

[0049] In some embodiments, the configuration information of a database instance includes at least one of the address information for connecting to the database instance and the account information for the database instance. For example, the account information includes a username and password.

[0050] 220. Based on the configuration information, access at least one raw data layer, each of which includes a description of a data source for the database instance.

[0051] For example, the raw data (RAW) layer can correspond to a specific data table, containing descriptive information about the data source (i.e., the raw data table). As a specific example, the descriptive information may include a schema description, which is not limited in this application.

[0052] Specifically, in step 220, a corresponding database instance can be obtained based on the configuration information, and then at least one RAW layer can be accessed based on that database instance. Each RAW layer can correspond to a data source within the database instance and includes description information for that data source. As an example, RAW layers corresponding to some or all data sources in the database instance can be accessed according to business requirements.

[0053] In some embodiments, RAW layer access can be performed once or multiple times based on the above configuration information. The number of RAW layers accessed each time can be one or more, and this application does not limit this. For example, based on the first business requirements and the configuration information of the database instance, RAW layer access can be performed on the first data source (or the first original data table) in the database instance, and the schema description of the first data source (or the first original data table) can be entered. Optionally, subsequently, based on the second business requirements and the configuration information, RAW layer access can be performed on the second data source (or the second original data table) in the database instance, and the schema description of the second data source (or the second original data table) can be entered.

[0054] 230. Access at least one operational data storage layer corresponding to at least one raw data layer, each operational data storage layer including processing rules for a data source of the database instance.

[0055] The Operational Data Store (ODS) layer, also known as the source layer, is specifically designed to store data from business systems. For example, data from the data source undergoes at least one of the following processes: extraction, cleaning, transmission, and extract-transform-load (ETL) before entering the ODS layer. The ODS layer primarily prepares the data for subsequent use, minimizing its impact on the business system.

[0056] In step 230, the at least one RAW layer can correspond one-to-one with at least one ODS layer. For example, the one-to-one correspondence between the RAW layer and the ODS layer can each include descriptive information of the same data source in the database instance and processing rules for that same data source. Processing rules for the data source in the database instance can be configured at this layer. By accessing the ODS layer corresponding to each RAW layer, the processing rules for the data source corresponding to each RAW layer can be obtained.

[0057] For example, after accessing a RAW layer, the corresponding ODS layer can be accessed. For instance, after accessing the RAW layer corresponding to a first data source (or a first original data table), the corresponding ODS layer can be accessed, which contains the processing rules for the first data source (or the first original data table). Optionally, after accessing the RAW layer corresponding to a second data source (or a second original data table), the corresponding ODS layer can be accessed, which contains the processing rules for the second data source (or the second original data table).

[0058] In some embodiments, the above processing rules may include desensitization rules. For example, sensitive fields in the data source may be hashed or removed directly.

[0059] As one implementation approach, the processing rules can be pre-built into the data management platform. Specifically, the data management platform can adopt a customized design approach based on the actual data access scenario, such as configuring de-identification rules through custom SQL. Optionally, for frequently used processing rules (such as hash rules), their corresponding field processing functions can be pre-built into the platform, thereby flexibly and conveniently implementing field processing, such as dynamic pruning, encryption, and custom processing methods. Optionally, abstract processing can also be performed on fields and constants. Therefore, the embodiments of this application can provide users with fast data processing operations; for example, users can achieve one-click de-identification processing of data sources through the de-identification rule button displayed on the platform.

[0060] 240. Based on the above-mentioned at least one raw data layer and at least one operational data storage layer, construct the data access task corresponding to the database instance.

[0061] Specifically, one database instance can correspond to one data access task. This data access task can be used to access at least one data source for that database instance. For example, the data access task can be used to access the data sources corresponding to the RAW layer and ODS layer accessed in steps 220 and 230.

[0062] In some embodiments, after accessing the new RAW layer and ODS layer, the data access task corresponding to the database instance can be updated according to the updated RAW layer and ODS layer.

[0063] For example, after accessing the RAW layer and ODS corresponding to the first data source (or the first original data table), a data access task corresponding to the database instance can be established, which can be used to access the first data source (or the first original data table). Optionally, after accessing the RAW layer and ODS corresponding to the second data source (or the second original data table), the data access task corresponding to the database instance can be updated. The updated data access task is used to access the first data source (or the first original data table) and the second data source (or the second original data table).

[0064] 250. Based on this data access task, start the incremental data task for the database instance.

[0065] Specifically, the data access task may include an incremental data task on a data source in the database instance, where the data source can be the data source corresponding to the RAW layer and ODS layer accessed in steps 220 and 230. In step 250, an incremental data task on the database instance can be started based on this data access task, for example, incremental data can be captured in real time.

[0066] For example, after accessing the RAW layer and ODS corresponding to the first data source (or the first original data table), a data access task corresponding to the database instance can be established, and an incremental data task for the first data source (or the first original data table) can be started. Optionally, after accessing the RAW layer and ODS corresponding to the second data source (or the second original data table), the data access task corresponding to the database instance can be updated, and incremental data tasks for the first data source (or the first original data table) and the second data source (or the second original data table) can be started.

[0067] Therefore, this embodiment of the application accesses at least one raw data layer and at least one operational data storage layer based on the configuration information of the database instance. Each raw data layer includes description information of a data source of the database instance, and each operational data storage layer includes processing rules for a data source of the database instance. Then, based on the at least one raw data layer and the at least one operational data storage layer, a data access task corresponding to the database instance is obtained. This allows the incremental data task of the database instance to be initiated based on the data access task, thereby achieving access to at least one data source in the database instance. This embodiment of the application can achieve access to at least one data source in the database instance by reusing the data access task of a single database instance. It achieves data access architecture design based on the database instance as the synchronization unit, which is beneficial for efficient data access to data sources in the database instance. For example, for one change log, it can support change capture and distribution of multiple data tables.

[0068] In addition, when data is synchronized at the database instance level, adding or deleting original data tables in that database instance will not increase or decrease the database load or the number of connections, which can help reduce the impact on upstream and downstream businesses.

[0069] In some embodiments, the incremental data obtained by the incremental data task can also be processed according to a pre-set data structure.

[0070] Specifically, by processing the incremental data according to a pre-defined standard data structure, it is possible to unify the incremental data reported from various data sources (such as binlog / data / event) through a custom standard data structure. Furthermore, it can be easily extended to support new data reporting styles, ensuring that subsequent field processing is based on the unified data structure, achieving logical reuse of the data structure, and supporting the access of data in multiple formats.

[0071] In some embodiments, a label can be added to the incremental data acquired by the incremental data task, and the label is used to indicate the data source corresponding to the incremental data.

[0072] Specifically, besides the essential data parsing and field processing, another major performance bottleneck in the entire data access process is data distribution. Here, data distribution refers to the downstream business's retrieval of the required data tables. If a standard filter is used, the performance of data distribution decreases linearly as the number of accessed data tables increases. Therefore, this embodiment of the application can add tags to incremental data according to rules during the data distribution stage. These tags indicate the data source corresponding to the incremental data, allowing for rapid data distribution after a cold start during subsequent data distribution. Downstream businesses can directly retrieve the bypass output data corresponding to the tags.

[0073] In some embodiments, descriptive information, such as schema information, of all data sources in the database instance can also be obtained. For example, binlog parsing requires explicit schema information of the data tables in the database. This allows the binlog acquisition side to collect schema information of all tables in the database instance and further filter the required data after parsing before distributing it. This ensures that data distribution does not affect each other in scenarios where tables are frequently added.

[0074] In some embodiments, the at least one data source mentioned above includes a first data source, and the data to be accessed by the data access task includes the existing data corresponding to the first data source. Synchronization of the existing data is mainly for accessing existing data prior to the start of binlog synchronization. For example, existing data can be pulled via Java Database Connectivity (JDBC), but this application does not limit this approach.

[0075] In some embodiments, existing data can be processed according to a pre-set data structure to convert it into a predefined standard data structure. Optionally, the data structure corresponding to the existing data can be the same as the data structure corresponding to the incremental data, allowing the existing data to reuse the processing logic of the incremental data. Optionally, for flexibility, existing data synchronization with custom conditions can also be supported, which is not limited in this application. For example, custom conditions can be embedded filtering conditions for data tables. Taking an order table as an example, custom conditions can be used to filter data with an order date of 20230101, or to filter data with a status of "completed", or to filter data with an order date of 20230101 and completed.

[0076] In other embodiments, the data that the data access task needs to access may not include the existing data corresponding to the first data source, and this application does not limit this.

[0077] In some embodiments, such as when it is determined that a data access task requires existing data, see [reference needed]. Figure 3 The corresponding existing data task can be started through the following steps 251 to 253.

[0078] 251. Stop the incremental data retrieval task for the database instance and obtain the stop point of the incremental data retrieval task. For example, when the incremental data retrieval task stops after synchronizing the 1003rd data record, the stop point of the incremental data retrieval task is 1003.

[0079] 252, Initiate the task of ingesting existing data from the first data source.

[0080] For example, the task of ingesting existing data from the first data source can be reconstructed. For instance, a corresponding task identifier (ID) can be assigned to this existing data task. This existing data task has a different task identifier (ID) than the incremental data task described above.

[0081] 253. When the existing data task is completed, the incremental data acquisition task is restarted according to the aforementioned stop point. At this time, the incremental data task can start from the aforementioned stop point 1003, and the incremental data task is also used to access the incremental data from the first data source.

[0082] Therefore, by restarting the data acquisition task based on the aforementioned stop points, it is beneficial to ensure the accuracy of incremental data acquisition while acquiring the existing data from the data source.

[0083] Figure 4 This illustration shows an optional data access process provided in an embodiment of this application. It should be understood that... Figure 4 The steps or operations of the data access process are illustrated, but these steps or operations are merely examples, and other operations may be performed in the embodiments of this application. Figure 4 Variations of various operations within it. Furthermore... Figure 4 The various steps in can be followed according to... Figure 4 The different orders in which they are presented may be executed, and it is possible that they are not intended to be executed. Figure 4 All operations.

[0084] 401, Is the database instance registered?

[0085] 402. Register the database instance if it is not already registered.

[0086] For specific details on database registration, please refer to [link / reference]. Figure 2 The relevant description of step 210 will not be repeated here.

[0087] 403, accessing the RAW layer when the database instance is registered.

[0088] Specifically, step 403 can be referred to. Figure 2 The relevant description of step 220 will not be repeated here.

[0089] 404, accessing the ODS layer.

[0090] Specifically, step 404 can be referenced. Figure 2 The relevant description of step 230 will not be repeated here.

[0091] 405, Build data access task.

[0092] Specifically, step 405 can be referenced. Figure 2 The relevant description of step 240 will not be repeated here.

[0093] 406. Do you need existing data? If you need existing data, proceed to step 407; if you do not need existing data, proceed to step 409.

[0094] For example, when accessing the RAW layer and ODS layer of data source #1 in steps 403 and 404, it can be determined whether it is necessary to access the existing data of data source #1.

[0095] 407, Stop incremental data task.

[0096] Specifically, when it is determined that existing data from data source #1 needs to be accessed, the incremental data task currently being executed by the data access task is stopped. Optionally, the stop point of this incremental data task can also be recorded, for example, 1003, meaning that the incremental data task stops synchronizing the 1003rd data item.

[0097] 408, Initiate the task of acquiring existing data.

[0098] Specifically, you can start the task of ingesting the existing data corresponding to data source #1.

[0099] 409, Incremental data task started.

[0100] Specifically, step 409 can be referenced. Figure 2 The description of step 250 is omitted here. After starting the incremental data task, the previously executed incremental data can be accessed based on the stop point in step 407. Here, the incremental data also includes the incremental data from data source #1.

[0101] Therefore, the embodiments of this application can achieve access to at least one data source in the database instance by reusing the data access task of a database instance, and realize that the database instance is the synchronization unit in the data access architecture design, which is conducive to efficient data access to the data source in the database instance.

[0102] In some embodiments, the status information of the existing data task can be obtained from the stream processing platform and transmitted to the client via a network protocol, allowing the client to display the status information and thus providing users with a real-time visual display of the execution status of the existing data task. The stream processing platform used to run the existing data task can be, for example, Flink. Optionally, Kafka can be used as a message middleware for data interaction between the stream processing platform and the data management platform. The network protocol can be the WebSocket real-time transport protocol. The data management platform and the stream processing platform can agree to use the task ID passed when starting the existing data task as their unique identifier.

[0103] In some embodiments, the status information of the existing data task may include at least one of the following: task start, in progress, end, and data acquisition progress.

[0104] In some embodiments, the ownership of each data source can be obtained, and the access data of each data source can be aggregated to the corresponding data center based on the ownership of each data source.

[0105] Specifically, for data analysis scenarios, regardless of where the upstream data originates, it must be aggregated in the same data center to enable data correlation, analysis, and operation, while also meeting legal compliance requirements. Based on this, the ownership of each data source can be identified, such as which platform or merchant the data source belongs to. Then, the data sources are distributed according to their ownership, aggregating the data to the correct data center to meet compliance requirements for data aggregation and analysis. For detailed procedures, please refer to [link to relevant documentation]. Figure 5 During data access, each access point (e.g., access point A or access point B) can enter the data's ownership subject and then synchronize the data to the access point's Kafka. Afterward, the ownership subject of the data can be determined, and the data can be distributed to the corresponding transit zone or data center. The transit zone can, for example, include a domestic transit zone or a US-based transit zone, and the corresponding data centers can include domestic data centers and US-based data centers.

[0106] In some embodiments, this application can select a relatively stable Flink-CDC component, such as Flink-CDC version 2.1.1, to automatically recover tasks from common operational anomalies. Flink-CDC 2.1.1 can resolve the issue of infinitely increasing state in Flink-CDC version 1.4. Optionally, Flink-CDC version 2.1.1 can be further modified to support MySQL 5.6, and a blacklist approach is used to resolve the issue of adding tables.

[0107] In some embodiments, in scenarios where a task fails and cannot be recovered, such as when a single database instance accesses an increasing number of data tables, it can support data backtracking by passing in binlog points or timestamps. For example, data after the binlog point or timestamp can be re-accessed to fill in the data from the period of the error.

[0108] The specific embodiments of this application have been described in detail above with reference to the accompanying drawings. However, this application is not limited to the specific details of the above embodiments. Within the scope of the technical concept of this application, various simple modifications can be made to the technical solutions of this application, and these simple modifications all fall within the protection scope of this application. For example, the various specific technical features described in the above embodiments can be combined in any suitable manner without contradiction. To avoid unnecessary repetition, this application will not describe the various possible combinations separately. Furthermore, various different embodiments of this application can also be arbitrarily combined, as long as they do not violate the spirit of this application, they should also be considered as the content disclosed in this application.

[0109] It should also be understood that, in the various method embodiments of this application, the sequence numbers of the above processes do not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application. It should be understood that these sequence numbers can be interchanged where appropriate so that the embodiments of this application described can be implemented in a sequence other than those illustrated or described.

[0110] The method embodiments of this application have been described in detail above. The following description, in conjunction with... Figure 6 and Figure 7 This describes an embodiment of the apparatus described in this application.

[0111] Figure 6 This is a schematic block diagram of a data management apparatus 1000 according to an embodiment of this application. Figure 6 As shown, the data management device 1000 may include an acquisition unit 1010, an access unit 1020, a construction unit 1030, and a startup unit 2040.

[0112] The acquisition unit 1010 is used to acquire configuration information of a database instance, wherein the database instance includes at least one data source;

[0113] Access unit 1020 is used to access at least one raw data layer according to the configuration information, each raw data layer including description information of a data source of the database instance;

[0114] The access unit 1020 is also used to access at least one operational data storage layer corresponding to the at least one raw data layer, and each operational data storage layer includes processing rules for a data source of the database instance;

[0115] Construction unit 1030 is used to construct a data access task corresponding to the database instance based on the at least one raw data layer and the at least one operational data storage layer;

[0116] The startup unit 1040 is used to start the incremental data task of the database instance according to the data access task.

[0117] In some embodiments, the at least one data source includes a first data source, and the data to be accessed by the data access task includes the existing data corresponding to the first data source.

[0118] In some embodiments, the starting unit 1040 is further configured to:

[0119] Stop the incremental data acquisition task of the database instance, and obtain the stop point of the incremental data acquisition task;

[0120] Start the task of ingesting existing data from the first data source;

[0121] When the existing data task is completed, the incremental data acquisition task is restarted according to the stop point.

[0122] In some embodiments, the obtaining unit 1010 is further configured to:

[0123] The status information of the existing data task is obtained from the stream processing platform, wherein the stream processing platform is used to run the existing data task.

[0124] The device 1000 further includes a transmission unit for transmitting the status information to a client via a network protocol, so that the client can display the status information.

[0125] In some embodiments, the apparatus 1000 further includes a processing unit for:

[0126] The incremental data acquired by the incremental data task is processed according to a pre-set data structure.

[0127] In some embodiments, the processing unit is further configured to:

[0128] The incremental data acquired by the incremental data task is tagged, and the tags are used to indicate the data source corresponding to the incremental data.

[0129] In some embodiments, the acquisition unit 1010 is further configured to: acquire the subject to which each of the data sources belongs;

[0130] The processing unit is also used to aggregate the access data corresponding to each data source to the corresponding data center according to the subject to which each data source belongs.

[0131] In some embodiments, the configuration information includes at least one of the address information for connecting to the database instance and the account information for the database instance.

[0132] It should be understood that the device embodiments and method embodiments can correspond to each other, and similar descriptions can be referred to the method embodiments. To avoid repetition, they will not be repeated here. Specifically, in this embodiment, the data management device 1000 can correspond to the device that executes the method 200 of the embodiments of this application, and the foregoing and other operations and / or functions of each module in the device 1000 are respectively to implement the corresponding processes in the various methods 200 above. For the sake of brevity, they will not be repeated here.

[0133] The apparatus and system of this application embodiments have been described above from the perspective of functional modules in conjunction with the accompanying drawings. It should be understood that these functional modules can be implemented in hardware, in software instructions, or in a combination of hardware and software modules. Specifically, the steps of the method embodiments in this application can be completed by integrated logic circuits in the processor's hardware and / or by software instructions. The steps of the methods disclosed in this application embodiments can be directly embodied as being executed by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. Optionally, the software module can reside in a mature storage medium in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps in the above method embodiments.

[0134] like Figure 7 This is a schematic block diagram of the electronic device 1100 provided in the embodiments of this application.

[0135] like Figure 7 As shown, the electronic device 1100 may include:

[0136] The system includes a memory 1110 and a processor 1120. The memory 1110 stores computer programs and transfers the program code to the processor 1120. In other words, the processor 1120 can retrieve and run the computer program from the memory 1110 to implement the methods described in the embodiments of this application.

[0137] For example, the processor 1120 can be used to execute the corresponding steps in the method 200 described above according to the instructions in the computer program.

[0138] In some embodiments of this application, the processor 1120 may include, but is not limited to:

[0139] General-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0140] In some embodiments of this application, the memory 1110 includes, but is not limited to:

[0141] Volatile memory and / or non-volatile memory. Non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static RAM (SRAM), Dynamic RAM (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDR SDRAM), Enhanced Synchronous DRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), and Direct Rambus RAM (DR RAM).

[0142] In some embodiments of this application, the computer program may be divided into one or more modules, which are stored in the memory 1110 and executed by the processor 1120 to complete the encoding method provided in this application. The one or more modules may be a series of computer program instruction segments capable of performing specific functions, which describe the execution process of the computer program in the electronic device 1100.

[0143] Optionally, the electronic device 1100 may also include:

[0144] Transceiver 1130, which can be connected to processor 1120 or memory 1110.

[0145] The processor 1120 can control the transceiver 1130 to communicate with other devices; specifically, it can send information or data to other devices or receive information or data sent by other devices. The transceiver 1130 may include a transmitter and a receiver. The transceiver 1130 may further include antennas, and the number of antennas may be one or more.

[0146] It should be understood that the various components in the electronic device 1100 are connected through a bus system, which includes a data bus, a power bus, a control bus, and a status signal bus.

[0147] According to one aspect of this application, a computer device is provided, including a processor and a memory for storing a computer program, the processor for calling and running the computer program stored in the memory, causing the computer device to perform the method of the above-described method embodiments.

[0148] According to one aspect of this application, a computer storage medium is provided that stores a computer program thereon, which, when executed by a computer, enables the computer to perform the methods of the above-described method embodiments. Alternatively, embodiments of this application also provide a computer program product containing instructions that, when executed by a computer, cause the computer to perform the methods of the above-described method embodiments.

[0149] According to another aspect of this application, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the method described in the above-described method embodiments.

[0150] In other words, when implemented using software, it can be implemented wholly or partially in the form of a computer program product. This computer program product includes one or more computer instructions. When these computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., digital video disc (DVD)), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0151] It should be understood that in the embodiments of this application, "B corresponding to A" means that B is associated with A. In one implementation, B can be determined based on A. However, it should also be understood that determining B based on A does not mean determining B solely based on A; B can also be determined based on A and / or other information.

[0152] In the description of this application, unless otherwise stated, "at least one" means one or more, and "multiple" means two or more. Additionally, "and / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can mean: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can mean: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0153] It should also be understood that the descriptions of "first", "second", etc. appearing in the embodiments of this application are only for illustration and to distinguish the objects being described, and there is no order to them. They do not indicate any special limitation on the number of devices in the embodiments of this application, and cannot constitute any limitation on the embodiments of this application.

[0154] It should also be understood that specific features, structures, or characteristics relating to embodiments in the specification are included in at least one embodiment of this application. Furthermore, these specific features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0155] Furthermore, the terms “comprising” and “having”, and any variations thereof, are intended to cover non-exclusive inclusion, such that a process, method, system, product, or server that includes a series of steps or units is not necessarily limited to those steps or units that are explicitly listed, but may include other steps or units that are not explicitly listed or that are inherent to such processes, methods, products, or devices.

[0156] It is understood that in the specific implementation of this application, when the above embodiments of this application are applied to specific products or technologies and involve user information and other related data, user permission or consent is required, and the collection, use and processing of related data must comply with relevant laws, regulations and standards.

[0157] Those skilled in the art will recognize that the modules and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0158] In the several embodiments provided in this application, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or modules may be electrical, mechanical, or other forms.

[0159] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. For example, the functional modules in the various embodiments of this application may be integrated into one processing module, or each module may exist physically separately, or two or more modules may be integrated into one module.

[0160] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A data management method, characterized in that, include: Obtain the configuration information of the database instance, which includes at least one data source. The data access task corresponding to the database instance is reused, and at least one data source is accessed in the database instance. The database instance is used as the synchronization unit in the data access architecture design. According to the configuration information, at least one raw data layer is accessed, and each raw data layer includes a description of a data source of the database instance, the description information including a schema description; Access at least one operational data storage layer corresponding to the at least one raw data layer, each operational data storage layer including processing rules for a data source of the database instance; Based on the at least one raw data layer and the at least one operational data storage layer, construct the data access task corresponding to the database instance; Based on the data access task, initiate the incremental data task for the database instance; The at least one data source includes a first data source, and the data to be accessed by the data access task includes the existing data corresponding to the first data source. Stop the incremental data acquisition task of the database instance, and obtain the stop point of the incremental data acquisition task; Start the task of ingesting existing data from the first data source; When the existing data task is completed, the incremental data acquisition task is restarted according to the stop point.

2. The method according to claim 1, characterized in that, Also includes: Obtain the status information of the existing data task from the stream processing platform, wherein the stream processing platform is used to run the existing data task; The status information is transmitted to the client via a network protocol so that the client can display the status information.

3. The method according to any one of claims 1-2, characterized in that, Also includes: The incremental data acquired by the incremental data task is processed according to a pre-set data structure.

4. The method according to any one of claims 1-2, characterized in that, Also includes: The incremental data acquired by the incremental data task is tagged, and the tags are used to indicate the data source corresponding to the incremental data.

5. The method according to any one of claims 1-2, characterized in that, Also includes: Obtain the entity to which each of the data sources belongs; Based on the entity to which each data source belongs, the access data corresponding to each data source is aggregated to the corresponding data center.

6. The method according to any one of claims 1-2, characterized in that, The configuration information includes at least one of the address information for connecting to the database instance and the account information for the database instance.

7. A data management device, characterized in that, include: The acquisition unit is used to acquire configuration information of a database instance, wherein the database instance includes at least one data source, and the data access task corresponding to the database instance is reused to access at least one data source in the database instance. In the data access architecture design, the database instance is used as the synchronization unit. An access unit is configured to access at least one raw data layer according to the configuration information, wherein each raw data layer includes description information of a data source of the database instance, and the description information includes a schema description; The access unit is also used to access at least one operational data storage layer corresponding to the at least one raw data layer, and each operational data storage layer includes processing rules for a data source of the database instance; A construction unit is used to construct a data access task corresponding to the database instance based on the at least one raw data layer and the at least one operational data storage layer; The startup unit is used to start the incremental data task of the database instance according to the data access task.

8. An electronic device, characterized in that, The method includes a processor and a memory, wherein the memory stores instructions, and when the processor executes the instructions, it causes the processor to perform the method according to any one of claims 1-6.

9. A computer storage medium, characterized in that, Used for storing computer programs, said computer programs including methods for performing any one of claims 1-6.

Images