Data lake entering method and device, computer equipment and storage medium

Abstract

The invention provides a data lake entering method and device, computer equipment and a storage medium, and belongs to the technical field of data management. The method comprises the steps of obtaining system metadata of a plurality of data sources, creating a data mapping table based on the system metadata, performing data prefetching based on the system metadata, storing prefetched first data into a data lake of a data management system, responding to an access request for a virtual access address of second data, and storing the second data into the data lake of the data management system based on the data mapping table. And acquiring second data corresponding to the access request from the data source where the second data is located, and storing the second data into the data lake, so that the time delay of the data service provided for the user is controllable.

Description

Technical Field

[0001] The present application relates to the field of digital governance, and in particular to a method, device, computer equipment and storage medium for entering data into a lake. Background Art

[0002] With the development of information technology, the process of digitalization is accelerating. Digitalization usually requires a data lake. As a centralized repository, the data lake is responsible for the aggregation of structured raw data and unstructured raw data. How to efficiently enter the raw data into the lake is the key to better providing data services.

[0003] At present, physical and virtual lake entry technologies are commonly used. Physical lake entry refers to extracting data from the data source through ETL (Extract-Transform-Load) tools, restful API (Application Programming Interface) interface calls or push, and then loading it into the data lake after conversion. Virtual lake entry refers to establishing virtualized tables / views in the data lake to establish a virtual mapping from the data source to the data lake. When providing data services, data is directly accessed from the data source through virtual mapping.

[0004] The above technical solutions all need to load or access data in the data source through the network. Therefore, the delay of data entering the lake or data service is uncontrollable due to factors such as data volume, network latency and bandwidth. In addition, in physical entry into the lake, data can only be accessed after entering the lake, so it is impossible to provide data services with the latest data in real time. Summary of the invention

[0005] The embodiments of the present application provide a method, device, computer equipment and storage medium for entering data into a lake, which can achieve the effect of controlling the delay of data services provided to users. The technical solution is as follows:

[0006] On the one hand, a method for entering data into a data lake is provided, the method comprising: obtaining system metadata of multiple data sources, the system metadata being used to indicate a storage address of data in the data source; creating a data mapping table based on the system metadata, the data mapping table being used to indicate a virtual access address of the data and a storage address of the data in the data source; performing data pre-fetching based on the system metadata, and storing the pre-fetched first data in a data lake of a data management system; in response to an access request for a virtual access address of second data, obtaining second data corresponding to the access request from the data source where the second data is located based on the data mapping table, and storing the second data in the data lake.

[0007] The embodiment of the present application provides a method for entering a data lake, which can achieve the purpose of controlling the latency of the data services provided to users. In the process of entering the data lake, firstly, the system metadata of multiple data sources are obtained, a data mapping table is created based on the system metadata, data is pre-fetched based on the system metadata, and the pre-fetched first data is stored in the data lake of the data management system; in response to an access request to a virtual access address of the second data, based on the data mapping table, the second data corresponding to the access request is obtained from the data source where the second data is located, and the second data is stored in the data lake, so that the latency of the data services provided to users can be controlled.

[0008] In some embodiments, the method further includes: registering adapters corresponding to multiple data sources, the adapters being used to receive data packets from the corresponding data sources and encode and decode the received data packets; associating multiple data sources and corresponding adapters to obtain system metadata of the data sources.

[0009] By registering adapters, unified access to heterogeneous data sources can be achieved. It can support the docking of various protocols and the loading of heterogeneous data source systems with different data types, and meet the access of various data types such as structured data, unstructured data, semi-structured data, various databases, file storage, object storage and other different storage methods, and has good scalability.

[0010] In some embodiments, obtaining the system metadata of the data source includes: receiving data packets from the corresponding data source by an adapter corresponding to the data source, encoding and decoding the received data packets to obtain the system metadata of the data source, and storing the system metadata of the data source in a storage subsystem of the data management system.

[0011] In some embodiments, creating the data mapping table based on the system metadata includes: creating a virtual image of the system metadata based on the system metadata; and generating the data mapping table based on the virtual image of the system metadata.

[0012] The data mapping table is used to establish a unified data view, and the unified data view can shield the actual storage address of the data from the outside.

[0013] In some embodiments, pre-fetching data based on system metadata and storing the pre-fetched first data into a data lake of the data management system includes:

[0014] Based on the background pre-fetch algorithm, multiple copy tasks are created;

[0015] Based on the multiple copy tasks, first data is pre-fetched from multiple data sources through virtual mirroring, the first data is stored in the data lake, and the storage address of the first data in the data mapping table is updated to the address in the data lake.

[0016] In some embodiments, in response to an access request to a virtual access address of second data, obtaining the second data corresponding to the access request from a data source where the second data is located based on a data mapping table, and storing the second data in a data lake includes:

[0017] In response to an access request to a virtual access address of second data, querying a storage address of the second data in a data mapping table;

[0018] If the storage address of the second data is an address in the data source, the second data corresponding to the access request is obtained from the data source where the second data is located, and the second data is stored in the data lake.

[0019] In some embodiments, if the storage address of the second data is an address in the data lake, the second data corresponding to the access request is obtained from the data lake.

[0020] On the other hand, a computer device is provided, which includes a processor and a memory, wherein the memory is used to store at least one computer program, and the at least one computer program is loaded and executed by the processor to implement the operations performed by the data entry method in the embodiment of the present application.

[0021] On the other hand, a computer-readable storage medium is provided, in which at least one computer program is stored. The at least one computer program is loaded and executed by a processor to implement the operations performed by the data entry method in the embodiment of the present application.

[0022] On the other hand, a computer program product or a computer program is provided, which includes a computer program code, and the computer program code is stored in a computer-readable storage medium. A processor of a computer device reads the computer program code from the computer-readable storage medium, and the processor executes the computer program code, so that the computer device executes the data entry method provided in the above-mentioned first aspect or various optional implementations of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0024] Figure 1 It is an implementation environment of a data entry method provided according to an embodiment of the present application;

[0025] Figure 2It is a flow chart of a method for entering data into a lake provided according to an embodiment of the present application;

[0026] Figure 3 is a flow chart of a method for obtaining system metadata provided according to an embodiment of the present application;

[0027] Figure 4 It is a block diagram of a data entry device provided according to an embodiment of the present application;

[0028] Figure 5 It is a structural schematic diagram of a computer device provided according to an embodiment of the present application. DETAILED DESCRIPTION

[0029] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0030] In this application, the terms "first", "second", etc. are used to distinguish identical or similar items with basically the same effects and functions. It should be understood that there is no logical or temporal dependency between "first", "second", and "nth", nor is there any limitation on quantity and execution order.

[0031] In the present application, the term "at least one" means one or more, and the term "plurality" means two or more.

[0032] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.) and signals involved in this application are all authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions. For example, the multimedia resources involved in this application are all obtained with full authorization.

[0033] Figure 1 Schematic diagram of the implementation environment of a data entry method provided in an embodiment of the present application. Figure 1 The implementation environment includes: multiple data sources 101 and a data management platform 102. The data management platform 102 includes: a virtual access gateway, a virtual image, a storage subsystem, a unified data view, and a data lake. The virtual access gateway includes: a protocol connection layer, an adapter, a core processing module, and a unified northbound interface.

[0034] The multiple data sources 101 are connected to the data management platform 102 through a virtual access gateway. The data source 101 transmits metadata to the data management platform 102 through the virtual access gateway. The data management platform 102 stores the metadata in the storage subsystem to establish a virtual image. Through the virtual image, the data management platform 102 obtains data from the data source 101 and stores it in the data lake. The data management platform 102 provides data to the outside through a unified data view.

[0035] In some embodiments, the multiple data sources 101 include Oracle database, MySQL (My Structured Query Language, relational database) database, object storage database, file system, third-party application system, MongoDB and other NoSQL (Not Only Structured Query Language, non-relational database) databases and other structured databases, unstructured databases, time series databases, etc., but are not limited to these.

[0036] In some embodiments, the adapter includes an ODBC (Open Database Connectivity) adapter, an S3 (Simple Storage Service) adapter, an NFS (Network File System) adapter, etc., but is not limited thereto.

[0037] The data entry method provided in the embodiment of the present application can achieve the purpose of providing data services to users with controllable latency. Figure 2 is a flow chart of a method for entering data into a lake according to an embodiment of the present application. The method is executed by a virtual access gateway in the above system. The virtual access gateway is a server device, such as Figure 2 As shown, the method comprises the following steps:

[0038] 201. The virtual access gateway registers adapters corresponding to multiple data sources, where the adapters are used to receive data packets from corresponding data sources, encode and decode the received data packets, and obtain system metadata of the data sources.

[0039] In the embodiment of the present application, the adapters corresponding to the multiple data sources are registered in the virtual access gateway in a plug-in manner. Accordingly, the virtual access gateway includes a protocol connection layer, a core processing module, multiple registered adapters and a northbound interface.

[0040] Among them, the protocol connection layer includes communication protocols corresponding to multiple data sources, which are used to realize communication between multiple data sources and the virtual access gateway. The core processing module is used to receive data packets from multiple data sources and distribute the data packets from multiple data sources to the corresponding adapters. The northbound interface is used to transmit data packets to the storage subsystem in the data management system.

[0041] In some embodiments, before implementing the present application, adapters corresponding to multiple data sources are developed according to the protocol specifications corresponding to the multiple data sources, so that the adapters corresponding to the multiple data sources can be registered in the virtual access gateway in a plug-in manner to achieve access to heterogeneous data sources.

[0042] 202. The virtual access gateway associates multiple data sources and corresponding adapters.

[0043] In an embodiment of the present application, a communication protocol corresponding to a data source is configured in a protocol connection layer in a virtual access gateway, a core processing module in the virtual access gateway connects the above-mentioned multiple adapters and the protocol connection layer, multiple data sources are associated with corresponding communication protocols in the protocol connection layer, and based on the connection between the above-mentioned multiple adapters and the protocol connection layer, multiple data sources and corresponding adapters are associated.

[0044] The above communication protocol is used to realize the communication between the data source and the virtual access gateway, and is at least one of the existing communication protocols or the user-defined communication protocol.

[0045] 203. The virtual access gateway obtains system metadata of multiple data sources, where the system metadata is used to indicate storage addresses of data in the data sources.

[0046] In some embodiments, the above step 203 includes: receiving a data packet from the corresponding data source by an adapter corresponding to the data source in the virtual access gateway, encoding and decoding the received data packet to obtain system metadata of the data source, and storing the system metadata of the data source in a storage subsystem of the data management system.

[0047] In an embodiment of the present application, the data source is associated with the corresponding communication protocol of the protocol connection layer in the virtual access gateway. Based on the communication protocol, the data packet is passed to the core processing module in the virtual access gateway. The core processing module identifies which data source the data packet comes from based on the communication protocol and assigns the data packet to the adapter corresponding to the data source. The adapter receives the data packet from the corresponding data source from the core processing module, calls the interface function in the adapter to encode and decode the data packet, and obtains the system metadata of the corresponding data source. The adapter is called to convert the communication protocol of the system metadata into a unified communication protocol, and the system metadata after the communication protocol conversion is provided to the storage subsystem in the data management system through the northbound interface in the virtual access gateway.

[0048] Among them, the protocol conversion function of the adapter can shield the upper storage subsystem from data source differences.

[0049] In some embodiments, see below Figure 3 The interface functions include: MetaDataList(), MetaDataDecode(), DataEncode(), DataList() and DataDecode().

[0050] In the above process, the core of the virtual access gateway is the core processing module, which receives data packets from multiple data sources and distributes the data packets from multiple data sources to corresponding adapters. The distribution of the above data packets is scheduled according to the general task scheduling, scheduling the data packet encoding and decoding tasks, and assigning the data packet encoding and decoding tasks to the corresponding adapters that are idle.

[0051] In some embodiments, the data source of the third-party business system data is accessed through Restful API (Application Programming Interface), SDK (Software Development Kit), etc.

[0052] The steps of associating multiple data sources and corresponding adapters and obtaining system metadata of multiple data sources are as follows: Figure 3As shown. In the virtual access gateway, the adapters corresponding to the Oracle database, the adapters corresponding to the MySQL (My Structured Query Language, relational database) database, the Hive database, the HDFS (Hadoop Distributed File System, distributed file system) / OSS (Operation Support Systems, operation support system) database, the device / sensor and other data sources are connected to the protocol connection layer through the core processing module, and the Oracle database, MySQL (My Structured Query Language, relational database) database, Hive database, HDFS (Hadoop Distributed File System, distributed file system) / OSS (Operation Support Systems, operation support system) database, the device / sensor and other data sources establish a communication relationship with the virtual access gateway through the corresponding communication protocol of the protocol connection layer in the virtual access gateway, and the corresponding adapters are associated in the virtual access gateway. The data source passes the data packet to the virtual access gateway based on the protocol connection layer. The core processing module in the virtual access gateway receives the data packet and distributes it to the corresponding adapter. The adapter calls the interface function to encode and decode the data packet to obtain the system metadata. The adapter then converts the communication protocol of the system metadata into the communication protocol corresponding to the unified Restful API (Application Programming Interface) interface, and provides the system metadata to the storage subsystem through the interface.

[0053] The corresponding communication protocols in the above-mentioned protocol connection layer include: JDBC (Java Database Connectivity), TCP (Transmission Control Protocol), UDP (User Datagram Protocol), Socket communication protocol, Http (Hypertext Transfer Protocol), Mqtt (Message Queuing Telemetry Transport), etc.

[0054] 204. Create a data mapping table based on the system metadata, where the data mapping table is used to indicate a virtual access address of the data and a storage address of the data in the data source.

[0055] The virtual access address is a virtual storage address pointing to data provided by the user, and is used to provide data access services for the user.

[0056] In some embodiments, step 204 first creates a virtual image of the system metadata based on the system metadata, and then generates a data mapping table based on the virtual image of the system metadata.

[0057] The virtual image is a copy of the system metadata, stored in the storage subsystem, and is the storage address of the data in the data source, which can be expressed in the form of a link. It should be noted that the virtual image does not include the data in the data source itself.

[0058] In some embodiments, a unified data view is created based on a data mapping table, and the unified data view is used to provide users with virtual access addresses of data in a data source. The user issues a data access request according to the virtual access address in the unified data view, obtains the data through the actual storage address of the data corresponding to the virtual access address in the data mapping table, and provides the data to the user. The unified data view can shield the actual storage address of the data from the outside.

[0059] 205. Pre-fetch data based on the system metadata, and store the pre-fetched first data in a data lake of the data management system.

[0060] In some embodiments, the above step 205 includes first creating multiple copy tasks based on the background prefetch algorithm, and then prefetching first data from multiple data sources through virtual mirroring based on the multiple copy tasks, storing the first data in the data lake, and updating the storage address of the first data in the data mapping table to the address in the data lake.

[0061] The background prefetching algorithm is at least one of the following background prefetching algorithms:

[0062] CIS (Copy in Consequence) means copying data in order.

[0063] RC (Random Cop) refers to dividing data into several units and randomly selecting a unit to copy when copying data;

[0064] LFU (Labor Frequently Used) is a method for prefetching original data based on the access frequency of adjacent data. It preferentially copies the data around the original data with high access frequency. The core idea is that "if adjacent data is accessed many times, the frequency of access to the data in the future will also be higher". It includes a record-to-table algorithm for data.

[0065] Customized background prefetching algorithm means that users or administrators define how to prefetch data from the data source.

[0066] In some embodiments, the multiple copy tasks correspond to data that has not entered the lake, and the data volume of the multiple copy tasks is less than or equal to the data that has not entered the lake.

[0067] The process of pre-fetching the first data based on multiple copy tasks is run in the background, and the execution of the copy tasks is arranged according to the QoS (Quality of Service) strategy. Multiple copy tasks are executed in a distributed manner until all the data in the data source is stored in the data lake. After the data quality is verified, the data mapping table is updated.

[0068] In an embodiment of the present application, for data in the data source that has not entered the lake, the virtual access address of the data in the data mapping table corresponds to the storage address of the data in the virtual image. After the data enters the lake, the virtual access gateway will modify the data mapping table so that the virtual access address of the data in the data mapping table corresponds to the storage address of the data in the data lake.

[0069] During the execution of the above-mentioned copying task, the data management system provides data access services to users through the data mapping table. Therefore, regardless of whether the background copy is completed, the data management system can respond to the data access request in a timely manner and return the data corresponding to the access request, so that the external data service is not affected by the storage location where the data is actually stored.

[0070] 206. In response to an access request to the virtual access address of the second data, based on the data mapping table, obtain the second data corresponding to the access request from the data source where the second data is located, and store the second data in the data lake.

[0071] In the embodiment of the present application, any of the situations shown in step 2061 and step 2062 may occur:

[0072] 2061. In response to an access request for a virtual access address of the second data, query the storage address of the second data in the data mapping table; if the storage address of the second data is an address in the data source, obtain the second data corresponding to the access request from the data source where the second data is located, and store the second data in the data lake.

[0073] In some embodiments, the access request for the virtual access address of the second data is issued by the user through the unified data view. If the storage address of the second data in the data mapping table is the address in the data source, it indicates that the second data has not been stored in the data lake through step 205. Therefore, the second data corresponding to the access request is obtained from the data source where the second data is located through the system metadata in the virtual image, and the second data is returned to the unified data view, and the second data is stored in the data lake. The storage address of the second data in the data mapping table is updated to the address in the data lake, so that the latency of the data service provided to the user can be controlled.

[0074] In an embodiment of the present application, for data in the data source that has not entered the lake, the virtual access address of the data in the data mapping table corresponds to the storage address of the data in the virtual image. After the data enters the lake, the virtual access gateway will modify the data mapping table so that the virtual access address of the data in the data mapping table corresponds to the storage address of the data in the data lake.

[0075] When an access request issued by the unified data view requires that the second data corresponding to the access request be stored in the data lake, a background copy task is interrupted and the second data is stored preferentially.

[0076] 2062. If the storage address of the second data is an address in the data lake, obtain the second data corresponding to the access request from the data lake.

[0077] In some embodiments, if the storage address of the second data in the data mapping table is an address in the data lake, it indicates that the second data has been stored in the data lake through step 205 or step 2061, and the second data is directly obtained from the data lake.

[0078] 207. Update the data in the data lake based on the data in the data source.

[0079] In some embodiments, the data management system sets an update strategy for the data management system based on the data in the data source and according to the update cycle of the data in multiple data sources, and updates the data in the data lake in real time or regularly, so that the data in the data lake is consistent with the data in the data source. Therefore, when the user accesses the data through the unified data view, the data that is consistent with the data in the data source can be directly obtained from the data lake, avoiding the data in the data lake from being wrong due to the change of the data in the data source.

[0080] The above method provides data access services based on data mapping tables. Data can be accessed based on the data lake entry configuration without waiting for the data to be stored in the data lake, thereby achieving a user experience with controllable latency for the data services provided to users. Even if the data takes too long to be stored in the data lake due to the large amount of data in the data source, high network latency, low bandwidth, low performance of the data source system or the data lake, data services can be provided to the outside based on virtual images and data mapping tables. Moreover, once the data is stored in the data lake, the data service performance will no longer be affected by factors such as the data source system performance and the network status between the data lake and the data source. Even if the data source system or network is abnormal, data services can still be provided to users normally. This method provides different ways of accessing data, such as active background pre-fetching and user access storage, which overcomes the disadvantages of both physical data entry and virtual data entry while combining the advantages of both.

[0081] Figure 4 is a block diagram of a data entry device provided according to an embodiment of the present application. The device is used to execute the steps of the above method. Figure 4 , the data entry device includes:

[0082] The system metadata acquisition module 401 is used to acquire system metadata of multiple data sources, where the system metadata is used to indicate the storage address of data in the data source;

[0083] A data mapping table creation module 402 is used to create a data mapping table based on system metadata, where the data mapping table is used to indicate a virtual access address of data and a storage address of the data in a data source;

[0084] A first data acquisition module 403 is used to pre-fetch data based on the system metadata, and store the pre-fetched first data into a data lake in the data management system;

[0085] The second data acquisition module 404: in response to an access request to a virtual access address of the second data, based on the data mapping table, acquires the second data corresponding to the access request from a data source where the second data is located, and stores the second data in the data lake.

[0086] In some embodiments, the system metadata acquisition module includes:

[0087] The adapter unit is used to register adapters corresponding to multiple data sources, and the adapters are used to receive data packets from the corresponding data sources and encode and decode the received data packets; associate multiple data sources and corresponding adapters to obtain system metadata of the data sources;

[0088] An association unit is used to associate multiple data sources and corresponding adapters.

[0089] In some embodiments, the adapter unit is used to:

[0090] The adapter corresponding to the data source receives the data packet from the corresponding data source, encodes and decodes the received data packet to obtain the system metadata of the data source, and stores the system metadata of the data source in the storage subsystem of the data management system.

[0091] In some embodiments, the data mapping table creation module is used to:

[0092] Based on the system metadata, a virtual image of the system metadata is created;

[0093] Generate a data mapping table based on the virtual image of the system metadata.

[0094] In some embodiments, the first data acquisition module is used to:

[0095] Based on the background pre-fetch algorithm, multiple copy tasks are created;

[0096] Based on the multiple copy tasks, first data is pre-fetched from multiple data sources through virtual mirroring, the first data is stored in the data lake, and the storage address of the first data in the data mapping table is updated to the address in the data lake.

[0097] In some embodiments, the second data acquisition module includes:

[0098] A query unit, configured to query a storage address of the second data in a data mapping table in response to an access request to a virtual access address of the second data;

[0099] The storage unit is used to obtain the second data corresponding to the access request from the data source where the second data is located if the storage address of the second data is an address in the data source, and store the second data in the data lake.

[0100] In some embodiments, the second data acquisition module is further used to:

[0101] If the storage address of the second data is an address in the data lake, the second data corresponding to the access request is obtained from the data lake.

[0102] It should be noted that: the device provided in the above embodiment only uses the division of the above functional modules as an example when data is entered into the lake. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the device and method embodiments provided in the above embodiment belong to the same concept, and the specific implementation process is detailed in the method embodiment, which will not be repeated here.

[0103] Figure 5It is a structural diagram of a server provided according to an embodiment of the present application. The server 500 may have relatively large differences due to different configurations or performances, and may include one or more CPUs (Central Processing Units, processors) 501 and one or more memories 502, wherein the memory 502 stores at least one computer program, and the at least one computer program is loaded and executed by the processor 501 to implement the data entry method provided by the above-mentioned various method embodiments. Of course, the server may also have components such as a wired or wireless network interface, a keyboard, and an input and output interface for input and output. The server may also include other components for implementing device functions, which will not be repeated here.

[0104] In the embodiment of the present application, the computer device can be configured as a server, and the server is used as the execution entity to implement the technical solution provided in the embodiment of the present application.

[0105] The embodiment of the present application also provides a computer-readable storage medium, in which at least one computer program is stored, and the at least one computer program is loaded and executed by a processor of a computer device to implement the operation performed by the computer device in the method of the above embodiment. For example, the computer-readable storage medium can be a ROM (Read-Only Memory), a RAM (Random Access Memory), a CD-ROM (Compact Disc Read-Only Memory), a magnetic tape, a floppy disk, an optical data storage device, etc.

[0106] In some embodiments, the computer program involved in the embodiments of the present application may be deployed and executed on a computer device, or on multiple computer devices located at one location, or on multiple computer devices distributed at multiple locations and interconnected by a communication network. Multiple computer devices distributed at multiple locations and interconnected by a communication network may constitute a blockchain system.

[0107] The embodiment of the present application also provides a computer program product or a computer program, which includes a computer program code, and the computer program code is stored in a computer-readable storage medium. The processor of the computer device reads the computer program code from the computer-readable storage medium, and the processor executes the computer program code, so that the computer device executes the data entry method provided in the above various optional implementations.

[0108] A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware or by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

[0109] The above description is only an optional embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A method for entering data into a lake, characterized in that: The method comprises: Acquire system metadata of multiple data sources, where the system metadata is used to indicate storage addresses of data in the data sources; Based on the system metadata, creating a data mapping table, the data mapping table is used to indicate a virtual access address of data and a storage address of the data in the data source; Pre-fetching data based on the system metadata, and storing the pre-fetched first data in a data lake of the data management system; In response to an access request for a virtual access address of second data, based on the data mapping table, the second data corresponding to the access request is obtained from a data source where the second data is located, and the second data is stored in the data lake.

2. The method according to claim 1, characterized in that The method further comprises: Registering adapters corresponding to the multiple data sources, wherein the adapters are used to receive data packets from the corresponding data sources, encode and decode the received data packets, and obtain system metadata of the data sources; The multiple data sources and corresponding adapters are associated.

3. The method according to claim 2, characterized in that The obtaining of the system metadata of the data source comprises: The adapter corresponding to the data source receives the data packet from the corresponding data source, encodes and decodes the received data packet to obtain the system metadata of the data source, and stores the system metadata of the data source in the storage subsystem of the data management system.

4. The method according to claim 1, characterized in that: The creating a data mapping table based on the system metadata includes: Based on the system metadata, creating a virtual image of the system metadata; The data mapping table is generated based on the virtual image of the system metadata.

5. The method according to claim 1, characterized in that The pre-fetching of data based on the system metadata and storing the pre-fetched first data into the data lake of the data management system comprises: Based on the background pre-fetch algorithm, multiple copy tasks are created; Based on the multiple copy tasks, the first data is pre-fetched from the multiple data sources through the virtual mirror, the first data is stored in the data lake, and the storage address of the first data in the data mapping table is updated to an address in the data lake.

6. The method according to claim 1, characterized in that In response to the access request for the virtual access address of the second data, based on the data mapping table, acquiring the second data corresponding to the access request from a data source where the second data is located, and storing the second data in the data lake includes: In response to an access request to the virtual access address of the second data, querying the storage address of the second data in the data mapping table; If the storage address of the second data is an address in the data source, the second data corresponding to the access request is obtained from the data source where the second data is located, and the second data is stored in the data lake.

7. The method according to claim 6, characterized in that The method further comprises: If the storage address of the second data is an address in the data lake, the second data corresponding to the access request is obtained from the data lake.

8. A data lake entry device, characterized in that: The device comprises: A system metadata acquisition module, used to acquire system metadata of multiple data sources, wherein the system metadata is used to indicate storage addresses of data in the data sources; A data mapping table creation module, used to create a data mapping table based on the system metadata, wherein the data mapping table is used to indicate a virtual access address of data and a storage address of the data in the data source; A first data acquisition module, configured to pre-fetch data based on the system metadata, and store the pre-fetched first data into a data lake of the data management system; A second data acquisition module: in response to an access request to a virtual access address of second data, based on the data mapping table, acquires the second data corresponding to the access request from a data source where the second data is located, and stores the second data in the data lake.

9. A computer device, characterized in that: The computer device comprises a processor and a memory, wherein the memory is used to store at least one computer program, and the at least one computer program is loaded by the processor to execute the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium is used to store at least one computer program, and the at least one computer program is used to execute the method according to any one of claims 1 to 7.

11. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the data entry method according to any one of claims 1 to 7 is implemented.

Images