Data lake method, electronic device, and medium

Abstract

The embodiment of the application provides a data lake entering method, an electronic device and a medium, and belongs to the field of data management. The method is applied to a data management system and comprises the following steps: an entering lake executing component in the data management system receives a data lake entering instruction sent by an entering lake task scheduling component in the data management system every interval of a preset time period, and the data lake entering instruction is used to instruct the entering lake executing component to execute an Nth entering lake operation; the entering lake executing component acquires a starting position parameter and an ending position parameter corresponding to the Nth entering lake operation in response to the data lake entering instruction; the entering lake executing component acquires target incremental data from a message middleware according to the starting position parameter and the ending position parameter corresponding to the Nth entering lake operation; and the entering lake executing component imports the target incremental data into a data lake in the data management system through a data lake management component in the data management system. In this way, the resource utilization rate of a data management system device cluster is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of data management, in particular to a data lake entering method, an electronic device and a medium. BACKGROUND

[0002] As a cluster component that can store a large amount of data of any type (including structured, semi-structured and unstructured data), the data lake is widely used in network architectures in various industries. For example, an e-commerce platform generates a large amount of order data every day, and these order data are stored in a database. The change records of the order data in the database need to be consumed by various platform tasks downstream, so the change records of the order data need to be processed as incremental data of the database and entered into the lake to be consumed by various platform tasks downstream.

[0003] At present, data entering into the lake is often implemented by using a platform or component for managing the data lake. The main form of incremental data generated by the database and needing to be entered into the lake is a streaming lake entering mode. Streaming lake entering requires a data entering execution component in the data management system to start a streaming task of a resident process to continuously consume incremental data transmitted by a message middleware in the database.

[0004] However, the streaming task occupies a large amount of resources in the device cluster of the distributed data management system because it is a resident process, so that the occupied resources cannot be reasonably dynamically allocated, which reduces the resource utilization rate of the device cluster. Therefore, how to implement batch lake entering of the incremental data of the database is a problem that needs to be considered by technical personnel at the present stage. SUMMARY

[0005] The main purpose of the embodiments of the present application is to provide a data lake entering method, an electronic device and a medium, which aims to implement batch lake entering of the incremental data of the database.

[0006] To achieve the above purpose, a first aspect of the embodiments of the present application provides a data lake entering method applied to a data management system, and the method comprises:

[0007] The lake entering execution component in the data management system receives a data lake entering instruction sent by a lake entering task scheduling component in the data management system every interval of a preset time period, and the data lake entering instruction is used to instruct the lake entering execution component to perform an Nth lake entering operation, wherein N is a positive integer;

[0008] The lake entering execution component acquires a start position parameter and an end position parameter corresponding to the Nth lake entering operation in response to the data lake entering instruction, and the start position parameter and the end position parameter are used to indicate different data positions in a message middleware in the data management system, and the message middleware is used to acquire incremental data from a database in the data management system.

[0009] The lake entry execution component obtains target incremental data from the message middleware according to a start position parameter and an end position parameter corresponding to the Nth lake entry operation;

[0010] The lake entry execution component imports the target incremental data into a data lake in the data management system through a data lake management component in the data management system.

[0011] In some embodiments, the lake entry execution component, in response to the data lake entry instruction, obtains a start position parameter and an end position parameter corresponding to the Nth lake entry operation, including:

[0012] The lake entry execution component, in response to the data lake entry instruction, obtains a first parameter and an end position parameter corresponding to the Nth lake entry operation, the first parameter being used to indicate a position of incremental data obtained by the message middleware from the database at a preset start time point in the message middleware; or

[0013] The lake entry execution component, in response to the data lake entry instruction, obtains a second parameter and an end position parameter corresponding to the Nth lake entry operation, the second parameter being used to indicate a preset start position of the message middleware; or

[0014] The lake entry execution component, in response to the data lake entry instruction, obtains a third parameter and an end position parameter corresponding to the Nth lake entry operation, the third parameter being used to indicate a position of incremental data last obtained by the lake entry execution component in the message middleware among incremental data that has been obtained by the lake entry execution component.

[0015] In some embodiments, the lake entry execution component, in response to the data lake entry instruction, obtains a first parameter and an end position parameter corresponding to the Nth lake entry operation, including:

[0016] The lake entry execution component, in response to the data lake entry instruction, obtains a first parameter and a fourth parameter corresponding to the Nth lake entry operation, the fourth parameter being used to indicate a position of incremental data obtained by the message middleware at a preset end time point in the message middleware;

[0017] The lake entry execution component, in response to the data lake entry instruction, obtains a second parameter and an end position parameter corresponding to the Nth lake entry operation, including:

[0018] The lake entry execution component acquires, in response to the data lake entry instruction, a second parameter and a fifth parameter corresponding to the Nth lake entry operation, the fifth parameter being used to indicate a preset end position of the message middleware or a position of the latest incremental data obtained by the message middleware from the database in the message middleware.

[0019] The lake entry execution component acquires, in response to the data lake entry instruction, a third parameter and an end position parameter corresponding to the Nth lake entry operation.

[0020] The lake entry execution component acquires, in response to the data lake entry instruction, a third parameter and the fifth parameter corresponding to the Nth lake entry operation.

[0021] In some embodiments, the method further comprises:

[0022] The data compression component in the data management system controls the data lake management component to perform data compression on the data in the data lake.

[0023] In some embodiments, before the data compression component in the data management system controls the data lake management component to perform data compression on the data in the data lake, the method further comprises:

[0024] The data compression component controls the data lake management component to acquire a time point at which the data lake is last updated.

[0025] The data compression component in the data management system controls the data lake management component to perform data compression on the data in the data lake, comprising:

[0026] In the case where the time difference is greater than a preset time difference threshold, the data compression component in the data management system controls the data lake management component to perform data compression on the data in the data lake, wherein the time difference is a difference between a time point at which the data lake is last updated and a time point at which the data lake is last compressed.

[0027] In some embodiments, before the data compression component in the data management system controls the data lake management component to perform data compression on the data in the data lake, the method further comprises:

[0028] The data compression component controls the data lake management component to acquire a data update frequency of the data lake, wherein the data update frequency is a frequency of data update after the data lake is last compressed.

[0029] The data compression component in the data management system controls the data lake management component to perform data compression on data in the data lake, including:

[0030] In a case where the data update times are greater than a preset number threshold, the data compression component in the data management system controls the data lake management component to perform data compression on the data lake.

[0031] In some embodiments, before the data compression component controls the data lake management component to obtain a time point at which the data lake is lastly updated, the method further includes:

[0032] The data compression component controls the data lake management component to close a metadata configuration function of the data lake management component, so that the data lake management component is prohibited from obtaining metadata that has been statically cached in the data lake.

[0033] In some embodiments, the data compression component in the data management system controls the data lake management component to perform data compression on data in the data lake, including:

[0034] The data compression component controls the data lake management component to generate a data compression plan;

[0035] The data compression component controls the data lake management component to open the metadata configuration function, so that the data lake management component generates corresponding metadata based on the metadata configuration function for new data, where the new data is data in the data lake after the data compression;

[0036] In a case where the metadata configuration function of the data lake management component is opened, the data compression component controls the data lake management component to perform data compression according to the data compression plan

[0037] To achieve the above object, a second aspect of the embodiment of the present application provides an electronic device, which comprises a memory and a processor, the memory stores a computer program, and the processor implements the data lake entry method of the first aspect when executing the computer program.

[0038] To achieve the above object, a third aspect of the embodiment of the present application provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the data lake entry method of the first aspect.

[0039] In the embodiment of the present application, the lake-in execution component in the data management system executes the lake-in operation in response to the data lake-in instruction sent by the lake-in scheduling component every preset time interval. The lake-in execution component first acquires the start position parameter and the end position parameter corresponding to the lake-in operation, and acquires the target incremental data of the database from the message middleware according to the data position indicated by the start position parameter and the end position parameter, and imports the acquired target incremental data into the data lake through the data lake management component. In this way, the lake-in execution component realizes batch lake-in of incremental data, releases the resources in the device cluster of the data management system, so that the resources in the device cluster of the data management system can be reasonably and dynamically allocated, and the resource utilization rate of the device cluster of the data management system is improved. BRIEF DESCRIPTION OF DRAWINGS

[0040] Figure 1 is a flowchart of a data lake-in method provided by the embodiment of the present application;

[0041] Figure 2 is an interaction diagram of a data management system provided by the embodiment of the present application;

[0042] Figure 3 is a flowchart of a data compression process provided by the embodiment of the present application;

[0043] Figure 4 is a flowchart of a data compression process provided by the embodiment of the present application;

[0044] Figure 5 is a structural diagram of a data management system provided by the embodiment of the present application;

[0045] Figure 6 is a hardware structure diagram of an electronic device provided by the embodiment of the present application. DETAILED DESCRIPTION

[0046] In order to make the purpose, technical scheme and advantages of the present application clearer, the present application will be further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application, and are not used to limit the present application.

[0047] It should be noted that although the functional modules are divided in the device schematic diagram, and the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a different order from the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and above-described drawings are used to distinguish similar objects, and do not necessarily describe a specific order or sequence.

[0048] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to be limiting of this application.

[0049] First, let's analyze some of the terms involved in this application:

[0050] Data Ingestion: Data Ingestion refers to the process of importing data from various data sources into a data lake. A data lake is a repository for storing large amounts of raw data, including structured, semi-structured, and unstructured data. The Ingestion process typically involves the following steps:

[0051] Data Collection - Collecting data from various data sources such as databases, log files, sensors, social media, etc.

[0052] Data Cleaning - Before importing data into the data lake, it is often necessary to clean and preprocess the data. This includes removing duplicate data, filling in missing values, correcting errors, etc.

[0053] Data Transformation - Converting data formats according to requirements. For example, converting CSV (Comma-Separated Values) format data to Parquet or Avro format to improve storage and query efficiency.

[0054] Data Transfer - Transferring data from source systems to the data lake. This can be done using data lake management platforms / components such as Apache Hudi or other data transfer tools such as Apache NiFi.

[0055] Data Storage - Storing processed data in the data lake. The data lake usually uses a distributed storage system such as Hadoop or cloud storage (such as Amazon S3, Azure Data Lake Storage).

[0056] Metadata Management - Maintaining metadata information of the data lake, including data source, structure, quality, etc., which helps to organize and retrieve data in the data lake.

[0057] Currently, data Ingestion is often implemented using platforms or components that manage the data lake. The main form of incremental data Ingestion generated by the database is stream Ingestion. Stream Ingestion requires the data Ingestion execution component in the data management system to start a resident process stream task to continuously consume incremental data delivered by the database through the message middleware component.

[0058] However, the streaming task occupies a large number of resources in the device cluster of the data management system as a resident process, so that the occupied resources cannot be reasonably dynamically allocated, and the resource utilization rate of the device cluster is reduced.

[0059] Based on this, the embodiment of the application provides a data import method and device, electronic equipment and medium, aiming to realize batch import of database incremental data.

[0060] The data import method, device, electronic equipment and medium provided by the embodiment of the application are specifically explained by the following embodiment, first, the data import method in the embodiment of the application is described.

[0061] The embodiment of the application can acquire and process related data based on artificial intelligence technology. Among them, artificial intelligence (Artificial Intelligence, AI) is to use digital computers or digital computer controlled machines to simulate, extend and expand human intelligence, perceive environment, acquire knowledge and use knowledge to obtain the best results.

[0062] The basic technology of artificial intelligence generally includes technologies such as sensors, special artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction system, mechatronics, etc. The artificial intelligence software technology mainly includes computer vision technology, robot technology, biometric technology, speech processing technology, natural language processing technology and machine learning / deep learning, etc.

[0063] The data import method provided by the embodiment of the application relates to the field of data management. The data import method provided by the embodiment of the application can be applied in a terminal, can also be applied in a server end, and can also be software running in a terminal or a server end. In some embodiments, the terminal can be a smart phone, a tablet computer, a notebook computer, a desktop computer, etc.; the server end can be configured as an independent physical server, can also be configured as a server cluster or a distributed system composed of multiple physical servers, can also be configured as a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN and basic cloud computing services such as big data and artificial intelligence platforms; the software can be an application for realizing the data import method, etc., but is not limited to the above forms.

[0064] The application is operable in a multitude of generic or specific computer system environments or configurations. Examples include personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments that include any of the above systems or devices, and the like. The application can be described in the general context of computer-executable instructions, such as program modules, being executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, and the like, that perform particular tasks or implement particular abstract data types. The application can also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules can be located in local and remote computer storage media including memory storage devices.

[0065] It should be noted that in each specific embodiment of the present application, when it is necessary to process relevant data related to the identity or characteristics of the user according to user information, user behavior data, user history data, and user location information, the user's permission or consent will be obtained first, and the collection, use, and processing of these data will comply with relevant laws, regulations, and standards. In addition, when the embodiments of the present application need to obtain sensitive personal information of the user, the separate permission or separate consent of the user will be obtained through a pop-up window or a jump to a confirmation page, and after obtaining the separate permission or separate consent of the user, the necessary user-related data for enabling the embodiments of the present application to normally operate will be obtained.

[0066] Figure 1 is a flowchart of a data lake entry method provided by the embodiments of the present application. The data lake entry method provided by the embodiments of the present application is applied to a data management system, and can include but is not limited to steps S110 to S140.

[0067] Step S110: The lake entry execution component in the data management system receives the data lake entry instruction sent by the lake entry task scheduling component in the data management system every interval of a preset time period. The data lake entry instruction is used to instruct the lake entry execution component to perform the Nth lake entry operation, where N is a positive integer.

[0068] Figure 2 is an interaction diagram of a data management system provided by the embodiments of the present application. Please refer to Figures 1 to 2 , the data management system is a system for managing a database and data lake data. The data management system can be a system applied to different business scenarios such as financial business, insurance business, or medical business. In the embodiments of the present application, the data management system includes a database, a data lake, a lake entry execution component, a lake entry task scheduling component, a message middleware, and a data lake management component.

[0069] The message middleware serves as a data relay, and is configured to obtain the incremental data in the database. The incremental data in the database can be transmitted from the database to the message middleware through a transmission component. The incremental data is consumed and obtained from the message middleware by the lake execution component, and is imported into the data lake by using the data lake management component.

[0070] It can be understood by those skilled in the art that the data management system provided by the embodiments of the present application can be a distributed data management system, that is, each device in the data management system device cluster has the function of all components in the data management system, and the data lake entry method provided by the embodiments of the present application can be implemented.

[0071] Exemplarily, the database can be Oracle, which can use tables to store data and use structured query language (SQL) for data query and management. The database can also be MySQL, which can run on various operating systems.

[0072] The lake entry execution component can be Apache Flink, which is an open-source data processing engine for large-scale data processing and real-time data stream analysis. It can provide high-throughput, low-latency, fault-tolerance and accurate distributed data processing functions, and is widely used in real-time data analysis, data pipeline, event-driven applications and other scenarios.

[0073] The lake entry task scheduling component can be LinkDo task scheduling system, which is an asynchronous task scheduling system mainly used in big data platform projects, BI projects, data mining projects and other scenarios. It is committed to solving the scheduling problem of enterprise-level asynchronous tasks and supports relationship control and scheduling execution of various data processing tasks.

[0074] The message middleware can be Apache Kafka, which is an open-source stream processing platform for building real-time data pipelines and stream applications. It can process and transmit high-throughput real-time data streams and has high scalability and fault tolerance.

[0075] The data lake management component can be Apache Hudi, which is a widely used data lake platform that can directly introduce core warehouse and database functions into the data lake, providing tables, transactions, advanced indexing, stream ingestion services, data cluster / compression optimization, and concurrency while maintaining open source file formats for data; the delivery component can directly use the data transfer cloud service (DTS).

[0076] The default data consumption logic of the lake entry execution component such as Apache Flink is stream consumption, that is, a stream task is started to complete the data entry operation. In order to complete the batch entry of the database incremental data and prevent the stream task started by the lake entry execution component from occupying the cluster resources, the lake entry execution component is configured to be silent in the normal state and only responds to the data entry instruction issued by the lake entry task scheduling component to perform the entry operation. The lake entry task scheduling component sends a data entry instruction to the lake entry execution component every preset time, instructing the lake entry execution component to perform the entry operation.

[0077] Illustratively, the database is the database of an e-commerce platform, and a large amount of order data is stored in the database, and new order data is constantly entering the database. The database has a log tool such as Binlog, which records the changes of the order data, and the log data representing these changes and the newly added order data in the database can all be used as incremental data of the database, which needs to be imported into the data lake for use by downstream tasks (such as order volume growth analysis tasks). The lake entry task scheduling component sends a data entry instruction to the lake entry execution component every hour, and the lake entry execution component responds to the data entry instruction sent by the lake entry task scheduling component to perform the data entry operation, that is, the lake entry execution component performs the data entry operation every hour, rather than using an uninterrupted resident stream task to perform the data entry operation. When the lake entry execution component performs the Nth entry operation, it is in response to the Nth data entry instruction issued by the lake entry task scheduling component.

[0078] Step S120: The lake entry execution component responds to the data entry instruction and obtains a start position parameter and an end position parameter corresponding to the Nth entry operation. The start position parameter and the end position parameter are used to indicate different data positions in the message middleware in the data management system, and the message middleware is used to obtain incremental data from the database in the data management system.

[0079] The lake entry execution component performs the lake entry operation in response to the data lake entry instruction. The lake entry execution component first acquires the start position parameter and the end position parameter corresponding to the lake entry operation, which are parameters configured in the lake entry execution component and are set in the message middleware to indicate different data positions in the message middleware, so that the lake entry execution component acquires the incremental data in the message middleware according to the different data positions (start position of data acquisition and end position of data acquisition) indicated by the start position parameter and the end position parameter.

[0080] The default working mode of the lake entry execution component can be the working mode of the streaming lake entry. When the start position parameter and the end position parameter are acquired, it is determined that the working mode is changed to the working mode of the batch lake entry.

[0081] Exemplarily, the start position parameter and the end position parameter corresponding to each lake entry operation can be pre-configured in the lake entry execution component. The start position parameter and the end position parameter corresponding to each lake entry operation can be the same parameter or different parameters according to the type of the parameter. The start position parameter and the end position parameter can be in the form of a timestamp, which indicates the position of the incremental data acquired at a certain time point in the message middleware. The timestamps corresponding to each lake entry operation are different.

[0082] Step S130: The lake entry execution component acquires the target incremental data from the message middleware according to the start position parameter and the end position parameter corresponding to the Nth lake entry operation.

[0083] After the lake entry execution component acquires the start position parameter and the end position parameter corresponding to the operation, the lake entry execution component acquires the incremental data from the message middleware according to the acquired start position parameter and end position parameter. The acquired incremental data is used as the target incremental data for subsequent data lake entry operations. That is, the lake entry execution component is intervalized, and the start position and the end position of the lake entry execution component in the message middleware are demarcated by the start position parameter and the end position parameter, so that the batch acquisition of the incremental data is completed.

[0084] Step S140: The lake entry execution component imports the target incremental data into the data lake in the data management system through the data lake management component in the data management system.

[0085] After the lake entry execution component acquires the target incremental data in the message middleware, the lake entry execution component imports the target incremental data into the data lake through the data lake management component, completes the batch processing of the data lake entry, ends the lake entry operation, and waits for the next data lake entry instruction.

[0086] Exemplarily, in a credit management system (i.e., a data management system) of a financial institution, the system needs to continuously monitor and analyze customer loan data to assess risks and make decisions. In order to improve the efficiency and accuracy of data analysis, the financial institution decides to regularly import incremental data (such as newly applied loans, repayment records, etc.) in the core loan database into a data lake for subsequent analysis tasks.

[0087] The lake entry task scheduling component in the credit management system is configured to send a data lake entry instruction to the lake entry execution component every hour, which instructs the lake entry execution component to start a new data lake entry operation (such as the Nth operation). The system ensures that incremental data is imported every hour, avoiding the problem of continuous occupation of system resources by real-time streaming tasks, while ensuring timely updating of data.

[0088] After receiving the Nth data lake entry instruction, the lake entry execution component first obtains the start position parameter and the end position parameter corresponding to this lake entry operation. According to the obtained start position and end position parameters, the lake entry execution component batch obtains loan data newly added since the last lake entry operation from the message middleware, such as newly submitted loan applications and recent repayment records. These incremental data are designated as target incremental data of this operation.

[0089] The lake entry execution component imports the target incremental data into the data lake using the data lake management component. In the data lake, these incremental data will be integrated into existing data sets for use by downstream tasks such as risk analysis models and credit scoring models. After the lake entry is completed, the lake entry execution component waits for the next data lake entry instruction.

[0090] Through the above steps S110-S140, the lake entry execution component in the data management system executes the lake entry operation in response to the data lake entry instruction sent by the lake scheduling component every preset time interval. The lake entry execution component first obtains the start position parameter and the end position parameter corresponding to this lake entry operation, and according to the data position indicated by the start position parameter and the end position parameter, obtains the target incremental data of the database from the message middleware, and imports the obtained target incremental data into the data lake through the data lake management component. In this way, the lake entry execution component realizes batch lake entry of incremental data, releases resources in the data management system device cluster, so that the resources in the data management system cluster device can be reasonably and dynamically allocated, improving the resource utilization of the data management system device cluster.

[0091] In some embodiments, the lake entry execution component, in response to the data lake entry instruction, obtains the start position parameter and the end position parameter corresponding to the Nth lake entry operation, comprising:

[0092] The lake entry execution component obtains, in response to the data lake entry instruction, a first parameter and an end position parameter corresponding to the Nth lake entry operation, the first parameter being used to indicate a position of incremental data obtained by the message middleware from the database at a preset starting time point in the message middleware; or

[0093] The lake entry execution component obtains, in response to the data lake entry instruction, a second parameter and an end position parameter corresponding to the Nth lake entry operation, the second parameter being used to indicate a preset starting position of the message middleware; or

[0094] The lake entry execution component obtains, in response to the data lake entry instruction, a third parameter and an end position parameter corresponding to the Nth lake entry operation, the third parameter being used to indicate a position of the incremental data last obtained by the lake entry execution component in the incremental data obtained by the lake entry execution component.

[0095] The starting position parameter can be one of a plurality of types, and the plurality of types can include the first parameter, the second parameter, and the third parameter. In this way, the lake entry execution component can start data acquisition according to the starting position indicated by the starting position parameter.

[0096] The first parameter is a timestamp parameter, which indicates a position of incremental data obtained by the message middleware from the database at a preset starting time point in the message middleware, and the position is the starting position of the lake entry execution component for this data acquisition.

[0097] For example, the first parameter indicates that the starting position of the lake entry execution component for data acquisition in this lake entry operation is the position of incremental data obtained by the message middleware at 9:00 a.m. in the message middleware. If the message middleware is Kafka (partition and offset are used to mark the data position), and the incremental data obtained by the message middleware at 9:00 a.m. is at the position of "partition C offset 20" in the message middleware, then "partition C offset 20" is the starting position of the lake entry execution component for this data acquisition.

[0098] The second parameter is a fixed position parameter, which indicates a specific preset starting position of the message middleware as the starting position of the lake entry execution component for this lake entry operation data acquisition.

[0099] For example, if the message middleware is Kafka, and the preset starting position of this lake entry operation is the position of "partition C offset 20", then "partition C offset 20" is the starting position of the lake entry execution component for this data acquisition.

[0100] The third parameter is a non-fixed position parameter, indicating a position of the last incremental data acquired by the lake entry execution component in the incremental data acquired by the lake entry execution component, i.e., a position of the last acquired data in the message middleware by the lake entry execution component in the last lake entry operation, and the third parameter indicates that the lake entry execution component starts data acquisition from the position of the last data acquisition in the last lake entry operation. In the case that the starting position parameter is the third parameter, the starting position parameter corresponding to each lake entry operation can be the same parameter.

[0101] Exemplarily, if the message middleware is Kafka, the lake entry execution component acquires data from the message middleware in the execution of the N-1th lake entry operation, and acquires the data up to the position of "partition C offset 20" of the message middleware. In the case that the starting position parameter is the third parameter, it is indicated that the starting position of data acquisition in the Nth lake entry operation is "partition C offset 20".

[0102] In some embodiments, the lake entry execution component acquires the first parameter and the ending position parameter corresponding to the Nth lake entry operation in response to the data lake entry instruction, including:

[0103] The lake entry execution component acquires the first parameter and the fourth parameter corresponding to the Nth lake entry operation in response to the data lake entry instruction, and the fourth parameter is used to indicate a position of the incremental data acquired by the message middleware at a preset ending time point in the message middleware.

[0104] The lake entry execution component acquires the second parameter and the ending position parameter corresponding to the Nth lake entry operation in response to the data lake entry instruction, including:

[0105] The lake entry execution component acquires the second parameter and the fifth parameter corresponding to the Nth lake entry operation in response to the data lake entry instruction, and the fifth parameter is used to indicate a preset ending position of the message middleware, or a position of the latest incremental data acquired by the message middleware from the database in the message middleware.

[0106] The lake entry execution component acquires the third parameter and the ending position parameter corresponding to the Nth lake entry operation in response to the data lake entry instruction, including:

[0107] The lake entry execution component acquires the third parameter and the fifth parameter corresponding to the Nth lake entry operation in response to the data lake entry instruction.

[0108] The ending position parameter corresponds to the starting position parameter, so that the lake entry execution component can end data acquisition according to the ending position indicated by the ending position parameter, and complete batch acquisition of the incremental data.

[0109] When the start position parameter is the first parameter, i.e., the timestamp parameter, the end position parameter can only be the fourth parameter, i.e., the timestamp parameter, indicating that the incremental data obtained by the message middleware at the preset end time point is the position of the message middleware.

[0110] For example, the fourth parameter indicates that the end position of the data acquisition performed by the lake execution component in this lake operation is the position of the incremental data obtained by the message middleware at 9:00 p.m. in the message middleware. If the message middleware is Kafka (partition and offset are used to mark the data position), and the incremental data obtained by the message middleware at 9:00 p.m. is at the position of "partition C offset 20" in the message middleware, then "partition C offset 20" is the end position of the data acquisition performed by the lake execution component.

[0111] When the start position parameter is the second position parameter or the third position parameter, the end position parameter can be the fifth parameter. The fifth parameter can indicate a preset end position of the message middleware, i.e., indicate that a specific preset end position of the message middleware is the end position of the data acquisition performed by the lake execution component in this lake operation, or indicate the position of the latest incremental data obtained by the message middleware from the database in the message middleware, i.e., indicate that the lake execution component performs the data acquisition until the latest incremental data in the message middleware is obtained.

[0112] For example, if the message middleware is Kafka, and the fifth parameter indicates that the preset end position of this lake operation is the position of "partition C offset 20", then "partition C offset 20" is the end position of the data acquisition performed by the lake execution component. If the message middleware is Kafka, and the fifth parameter indicates that the end position is the position of the latest incremental data obtained by the message middleware from the database in the message middleware, and the position of the latest incremental data obtained by the message middleware from the database in the message middleware is "partition C offset 20", then "partition C offset 20" is the end position of the data acquisition performed by the lake execution component.

[0113] In some embodiments, the method can further include:

[0114] The data compression component in the data management system controls the data lake management component to perform data compression on the data in the data lake.

[0115] The data newly entered into the lake is in a scattered and compressed state, which is not convenient for indexing and searching, so data compression needs to be performed on the data in the data lake. When the lake entry execution component performs lake entry operation in a streaming task, in the streaming lake entry mode, data is continuously and real-timely entered into the data lake through a data lake management component such as Hudi, which triggers the immediate data compression operation of the data lake management component to ensure the organization and optimization of the data in the data lake; but in the batch lake entry mode, the lake entry execution component imports data into the data lake in batches, and after each batch of data is imported into the data lake, the data lake management component ends the import task and does not immediately trigger the data compression operation. Therefore, in the batch lake entry mode, a data compression component needs to be used to control the data lake management component to perform data compression on the data in the data lake. The data compression component can be configured with a start condition parameter, such as an interval time reaching a preset threshold, to start the data compression operation and control the data lake management component to perform data compression on the data in the data lake. In this way, the data in the data lake can also be timely compressed in the batch lake entry mode, ensuring the rationality of the organization of the data in the data lake.

[0116] Exemplarily, the data compression component can be Apache Spark, which is a fast unified analysis engine for large-scale data processing. It has the characteristics of fast speed, easy use, and strong universality, and is widely used in data analysis, machine learning, and big data processing fields. The data compression component can be configured to start a data compression operation every hour, that is, to control the data lake management component to perform data compression on the data in the data lake every hour.

[0117] In some embodiments, before the data compression component in the data management system controls the data lake management component to perform data compression on the data in the data lake, the method can further include:

[0118] The data compression component controls the data lake management component to obtain the time point at which the data lake is last updated;

[0119] The data compression component in the data management system controls the data lake management component to perform data compression on the data in the data lake, including:

[0120] In the case where the time difference is greater than the preset time difference threshold, the data compression component in the data management system controls the data lake management component to perform data compression on the data in the data lake, wherein the time difference is the difference between the time point at which the data lake is last updated and the time point at which the data lake is last compressed.

[0121] The data compression component first controls the data lake management component to obtain a time point at which the data lake last performed data updating, that is, a time point at which the data lake last generated incremental data, and then calculates a time difference between the time point and a time point at which the data lake last performed data compression. In a case where the time difference is greater than a preset time difference threshold, it is indicated that a duration for which the data lake has not performed data compression has reached a maximum limit value, and a large amount of data in the data lake needs to be compressed. At this time, the data compression component can control the data lake management component to compress the data in the data lake. In this way, the data in the data lake can be compressed in a timely manner, and the organization rationality of the data in the data lake is ensured.

[0122] In some embodiments, before the data compression component in the data management system controls the data lake management component to compress the data in the data lake, the method can further include:

[0123] The data compression component controls the data lake management component to obtain a data updating frequency of the data lake, where the data updating frequency is a frequency of data updating of the data lake after the data lake last performed data compression.

[0124] The data compression component in the data management system controls the data lake management component to compress the data in the data lake, including:

[0125] In a case where the data updating frequency is greater than a preset frequency threshold, the data compression component in the data management system controls the data lake management component to compress the data in the data lake.

[0126] The judgment standard of whether the data compression component controls the data lake management component to compress the data in the data lake can also be a data updating frequency of the data lake after the data lake last performed data compression, that is, a frequency of generating incremental data. If the data updating frequency is greater than a preset frequency threshold, it is indicated that a duration for which the data lake has not performed data compression has reached a maximum limit value, and a large amount of data in the data lake needs to be compressed. At this time, the data compression component can control the data lake management component to compress the data in the data lake. In this way, the data in the data lake can be compressed in a timely manner, and the organization rationality of the data in the data lake is ensured.

[0127] Figure 3 is a flow logic diagram of a data compression process provided by an embodiment of the present application, as shown in Figure 3As shown, the skilled person in the art can understand that the data compression component can control the data lake management component to determine whether to compress the data in the data lake according to the difference between the time point at which the data lake is last updated and the time point at which the data lake is last compressed, or the number of times of data update after the data lake is last compressed. In either case, if the difference (time difference) or the number of times meets the data compression condition (the difference is greater than the preset time difference threshold, or the number of times is greater than the preset number of times threshold), the data compression component can control the data lake management component to compress the data in the data lake.

[0128] In some embodiments, before the data compression component controls the data lake management component to obtain the time point at which the data lake is last updated, the method further comprises:

[0129] The data compression component controls the data lake management component to close the metadata configuration function of the data lake management component, so that the data lake management component prohibits obtaining the metadata in the data lake that has been statically cached.

[0130] Figure 4 is a flowchart of the data compression process provided by the embodiments of the present application, as Figure 4 As shown, the data lake management component has a metadata configuration function, which generates metadata for the data in the data lake, and processes the data in the data lake according to the metadata. Metadata is data about data, which describes the structure, content, context and characteristics of data. Metadata can help users or platforms understand and use data, and provide background and related information of data, so that data is easier to find, manage and analyze.

[0131] However, similar to the metadata generation of the data lake management component of Hud i, the metadata generation has a certain lag with respect to the data update in the data lake, that is, new data has entered the data lake, but the data management component does not generate the corresponding metadata for the new data in time and statically cache it. Therefore, if the data lake management component compresses the data in the data pool according to the metadata that has been generated and statically cached, the new data in the data lake cannot be scanned by the data lake management component, and the new data cannot be compressed.

[0132] Therefore, before the data compression component makes the judgment of whether the data lake management component compresses the data in the data lake, that is, before the data compression component controls the data lake management component to obtain the time point of the last data update of the data lake, or before the data compression component controls the data lake management component to obtain the number of data updates of the data lake, the data compression component controls the data lake management component to close the metadata configuration function of the data lake management component, so that the data lake management component is prohibited from obtaining the metadata that has been statically cached in the data lake, and the data lake management component can only compress the data in the data lake by scanning all the data in the data lake, to ensure that all the data in the data lake (including new data for which corresponding metadata has not been generated in time) can be compressed.

[0133] In some embodiments, the data compression component in the data management system controls the data lake management component to compress the data in the data lake, comprising:

[0134] The data compression component controls the data lake management component to generate a data compression plan;

[0135] The data compression component controls the data lake management component to open the metadata configuration function, so that the data lake management component generates corresponding metadata for new data based on the metadata configuration function, wherein the new data is data in the data lake after data compression;

[0136] In the case where the metadata configuration function of the data lake management component is opened, the data compression component controls the data lake management component to perform data compression according to the data compression plan.

[0137] The specific process of data compression can be divided into two steps. The first step is that the data compression component controls the data lake management component to generate a data compression plan, which can be a specific compression plan file. The second step is that the data compression component controls the data lake management component to perform data compression according to the data compression plan.

[0138] In the process of the first step, in the case where the metadata configuration function of the data lake management component has been closed, the data lake management component scans all the data in the data lake to generate a data compression plan.

[0139] Therefore, after the data compression plan is generated, the metadata configuration function of the data lake management component can be opened, because the data compression plan has been generated, and the opening of the metadata configuration function will not affect the second step of the above process. Therefore, after the data compression plan is generated, the metadata configuration function of the data lake management component is opened, so that the data lake management component can generate corresponding metadata for the new data after compression, ensure the completeness of the metadata in the data lake, and make the data in the data lake easier to be managed and searched.

[0140] Please refer toFigure 5 , Figure 5 is a structural schematic diagram of a data management system provided by an embodiment of the present application, and the present application further provides a data management system 200, which can implement the data lake method described above, and the system 200 comprises:

[0141] a data lake task scheduling component 210, configured to send a data lake instruction every preset time interval;

[0142] a database 220, configured to generate incremental data;

[0143] a message middleware 230, configured to obtain the incremental data from the database;

[0144] a data lake execution component 240, configured to perform the following steps:

[0145] receiving a data lake instruction sent by a data lake task scheduling component in a data management system every preset time interval, the data lake instruction being used to instruct a data lake execution component to perform an Nth data lake operation, N being a positive integer;

[0146] in response to the data lake instruction, obtaining a start position parameter and an end position parameter corresponding to the Nth data lake operation, the start position parameter and the end position parameter being used to indicate different data positions in a message middleware in the data management system;

[0147] obtaining target incremental data from the message middleware according to the start position parameter and the end position parameter corresponding to the Nth data lake operation;

[0148] importing the target incremental data into a data lake through a data lake management component;

[0149] a data lake management component 250, configured to import the target incremental data into a data lake;

[0150] a data lake 260, configured to store the target incremental data.

[0151] Those skilled in the art can understand that the data management system provided by the embodiments of the present application is a distributed data management system, that is, each device in the data management system device cluster has the function of all components in the data management system, and can implement the data lake method provided by the embodiments of the present application.

[0152] The specific implementation of the data management system 200 corresponds to the specific embodiments of the data lake method described above, and will not be repeated here.

[0153] The embodiment of the present application further provides an electronic device, which comprises a memory and a processor, the memory stores a computer program, and the processor executes the computer program to realize the data entering method. The electronic device can be any intelligent terminal, such as a tablet computer or a vehicle-mounted computer.

[0154] Please refer to Figure 6 , Figure 6 The hardware structure of the electronic device of another embodiment is illustrated, and the electronic device comprises:

[0155] The processor 901 can be implemented in the form of a general-purpose CPU (Central Processing Unit), a microprocessor, an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits, and is used to execute related programs to realize the technical solutions provided by the embodiments of the present application.

[0156] The memory 902 can be implemented in the form of a ROM (Read Only Memory), a static storage device, a dynamic storage device, or a RAM (Random Access Memory). The memory 902 can store an operating system and other application programs. When the technical solutions provided by the embodiments of the present application are implemented by software or firmware, the related program codes are stored in the memory 902 and are called and executed by the processor 901 to implement the data entering method of the embodiments of the present application.

[0157] The input / output interface 903 is used to realize information input and output.

[0158] The communication interface 904 is used to realize the communication interaction between the device and other devices, and can realize communication through a wired manner (for example, a USB, a network cable, etc.) or a wireless manner (for example, a mobile network, a WI-FI, Bluetooth, etc.).

[0159] The bus 905 is used to transmit information between various components (for example, the processor 901, the memory 902, the input / output interface 903, and the communication interface 904) of the device.

[0160] The processor 901, the memory 902, the input / output interface 903, and the communication interface 904 are connected to each other through the bus 905 to realize communication connection between them in the device.

[0161] The embodiment of the present application further provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the data lake entering method.

[0162] The memory, as a non-transitory computer readable storage medium, can be used to store non-transitory software programs and non-transitory computer executable programs. In addition, the memory can include a high-speed random access memory, and can also include a non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state memory device. In some embodiments, the memory can optionally include a memory remotely arranged relative to the processor, and these remote memories can be connected to the processor through a network. Examples of the above network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0163] The data lake entering method, the electronic device and the medium provided by the embodiment of the present application, the lake entering execution component in the data management system responds to the data lake entering instruction sent by the lake entering scheduling component every interval preset time period, and executes the lake entering operation. The lake entering execution component first acquires the starting position parameter and the ending position parameter corresponding to the lake entering operation, and acquires the target incremental data of the database from the message middleware according to the data position indicated by the starting position parameter and the ending position parameter, and imports the target incremental data acquired through the data lake management component into the data lake. In this way, the lake entering execution component realizes batch lake entering of incremental data, releases the resources in the data management system device cluster, so that the resources in the data management system cluster device can be reasonably and dynamically allocated, and the resource utilization rate of the data management system device cluster is improved.

[0164] The embodiments described in the embodiments of the present application are used to more clearly illustrate the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided by the embodiments of the present application. Those skilled in the art can know that, with the evolution of technology and the appearance of new application scenarios, the technical solutions provided by the embodiments of the present application are also applicable to similar technical problems.

[0165] Those skilled in the art can understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and can include more or fewer steps than the figures, or combine certain steps, or different steps.

[0166] The device embodiments described above are only schematic, and the units described as separate components can or can not be physically separated, that is, can be located in one place, or can be distributed on multiple network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the embodiment of the present application.

[0167] Those skilled in the art can understand that all or some of the steps in the method disclosed above, the function modules / units in the system and the device can be implemented as software, firmware, hardware or appropriate combination thereof.

[0168] The terms "first", "second", "third", "fourth" and the like in the description of the application and in the claims, if any, are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the use of these terms herein is to be construed to cover a general order and / or structure unless otherwise indicated. Furthermore, the terms "comprise", "comprising", "has", "having", "includes", "including", "contain", "containing" or any other similar forms are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises, has, includes, contains items or components does not include items or components not explicitly recited. The terms "a" or "an", as used herein in the detailed description and in the claims, mean "one or more" or "at least one", unless otherwise indicated.

[0169] It should be understood that, in the present application, "at least one" means one or more, and "multiple" means two or more. "And / or" is used to describe the relationship between associated objects, which means that there can be three relationships, for example, "A and / or B" can mean that there are three cases: only A, only B, and A and B at the same time, where A and B can be singular or plural. The character " / " generally represents an "or" relationship between the associated objects. "At least one of the following" or similar expressions means any combination of these items, including any combination of single or multiple items. For example, at least one of a, b or c can mean a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be singular or plural.

[0170] In several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of the above units is only a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the displayed or discussed objects can be indirect coupling or communication connection through some interfaces, devices or units, and can be electrical, mechanical or other forms.

[0171] The units described as separate components above can or can not be physically separate, and the components shown as units can or can not be physical units, that is, can be located in one place, or can be distributed to multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment scheme.

[0172] In addition, each functional unit in each embodiment of the present application can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0173] If the integrated unit is realized in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application, the essential part or the whole or part of the technical solutions that make contributions to the prior art can be embodied in the form of a software product. The computer software product is stored in a storage medium, including multiple instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method of each embodiment of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various program storage media.

[0174] The preferred embodiments of the embodiments of the present application are described above with reference to the accompanying drawings, but this does not limit the scope of the embodiments of the present application. Any modifications, equivalent replacements and improvements made by those skilled in the art without departing from the scope and essence of the embodiments of the present application shall be within the scope of the embodiments of the present application.

Claims

1. A method of data lake-ing, characterized by, The method is applied to a data management system, and comprises the following steps: An in-lake execution component in the data management system receives a data in-lake instruction sent by an in-lake task scheduling component in the data management system every interval of a preset time period, and the data in-lake instruction is used to instruct the in-lake execution component to perform an Nth in-lake operation, wherein N is a positive integer; The in-lake execution component acquires a start position parameter and an end position parameter corresponding to the Nth in-lake operation in response to the data in-lake instruction, and the start position parameter and the end position parameter are used to indicate different data positions in a message middleware in the data management system, and the message middleware is used to acquire incremental data from a database in the data management system; The in-lake execution component acquires target incremental data from the message middleware according to the start position parameter and the end position parameter corresponding to the Nth in-lake operation; The in-lake execution component imports the target incremental data into a data lake in the data management system through a data lake management component in the data management system; A data compression component in the data management system controls the data lake management component to close a metadata configuration function of the data lake management component, so that the data lake management component is prohibited from acquiring metadata that has been statically cached in the data lake; in the case that the metadata configuration function of the data lake management component is closed, the data compression component controls the data lake management component to acquire a time point at which the data lake is subjected to a last data update; In the case that a time difference is greater than a preset time difference threshold, the data compression component controls the data lake management component to generate a data compression plan; wherein the time difference is a difference between the time point at which the data lake is subjected to the last data update and a time point at which the data lake is subjected to a last data compression; the data compression component controls the data lake management component to open the metadata configuration function, so that the data lake management component generates corresponding metadata for new data based on the metadata configuration function, wherein the new data is data in the data lake after the data compression; in the case that the metadata configuration function of the data lake management component is opened, the data compression component controls the data lake management component to perform data compression according to the data compression plan.

2. The method of claim 1, wherein, The in-lake execution component acquires a start position parameter and an end position parameter corresponding to the Nth in-lake operation in response to the data in-lake instruction, which comprises the following steps: The in-lake execution component acquires a first parameter and an end position parameter corresponding to the Nth in-lake operation in response to the data in-lake instruction, and the first parameter is used to indicate a position of incremental data acquired by the message middleware from the database at a preset start time point; or The in-lake execution component acquires a second parameter and an end position parameter corresponding to the Nth in-lake operation in response to the data in-lake instruction, and the second parameter is used to indicate a preset start position of the message middleware; or The in-lake execution component acquires a second parameter and an end position parameter corresponding to the Nth in-lake operation in response to the data in-lake instruction, and the second parameter is used to indicate a preset start position of the message middleware; or The lake-entering execution component acquires, in response to the data lake-entering instruction, a third parameter and an end position parameter corresponding to the Nth lake-entering operation, the third parameter being used to indicate a position of last incremental data acquired by the lake-entering execution component in incremental data that has been acquired by the lake-entering execution component.

3. The method of claim 2, wherein, The lake-entering execution component acquires, in response to the data lake-entering instruction, a first parameter and an end position parameter corresponding to the Nth lake-entering operation, including: The lake-entering execution component acquires, in response to the data lake-entering instruction, a first parameter and a fourth parameter corresponding to the Nth lake-entering operation, the fourth parameter being used to indicate a position of incremental data acquired by the message middleware at a preset end time point in the message middleware. The lake-entering execution component acquires, in response to the data lake-entering instruction, a second parameter and an end position parameter corresponding to the Nth lake-entering operation, including: The lake-entering execution component acquires, in response to the data lake-entering instruction, a second parameter and a fifth parameter corresponding to the Nth lake-entering operation, the fifth parameter being used to indicate a preset end position of the message middleware.

4. A data lake method, characterized by, The method is applied to a data management system, and the method comprises: A lake-entering execution component in the data management system receives a data lake-entering instruction sent by a lake-entering task scheduling component in the data management system every interval of a preset time period, the data lake-entering instruction being used to instruct the lake-entering execution component to perform an Nth lake-entering operation, N being a positive integer; The lake-entering execution component acquires, in response to the data lake-entering instruction, a start position parameter and an end position parameter corresponding to the Nth lake-entering operation, the start position parameter and the end position parameter being used to indicate different data positions in a message middleware in the data management system, the message middleware being used to acquire incremental data from a database in the data management system; The lake-entering execution component acquires target incremental data from the message middleware according to the start position parameter and the end position parameter corresponding to the Nth lake-entering operation; The lake-entering execution component imports the target incremental data into a data lake in the data management system through a data lake management component in the data management system; A data compression component in the data management system controls the data lake management component to close a metadata configuration function of the data lake management component, so that the data lake management component is prohibited from acquiring metadata that has been statically cached in the data lake; in a case where the metadata configuration function of the data lake management component is closed, the data compression component controls the data lake management component to acquire a data update number of the data lake, wherein the data update number is a number of data updates performed by the data lake after a last data compression is performed. In a case where the data update times are greater than a preset number threshold, the data compression component controls the data lake management component to generate a data compression plan; the data compression component controls the data lake management component to start the metadata configuration function, so that the data lake management component generates corresponding metadata for new data based on the metadata configuration function, wherein the new data is data compressed in the data lake; in a case where the metadata configuration function of the data lake management component is started, the data compression component controls the data lake management component to perform data compression according to the data compression plan.

5. The method of claim 4, wherein, The lake entry execution component acquires, in response to the data lake entry instruction, a start position parameter and an end position parameter corresponding to the Nth lake entry operation, including: The lake entry execution component acquires, in response to the data lake entry instruction, a first parameter and an end position parameter corresponding to the Nth lake entry operation, the first parameter being used to indicate a position of incremental data obtained by the message middleware from the database at a preset start time point in the message middleware; or The lake entry execution component acquires, in response to the data lake entry instruction, a second parameter and an end position parameter corresponding to the Nth lake entry operation, the second parameter being used to indicate a preset start position of the message middleware; or The lake entry execution component acquires, in response to the data lake entry instruction, a third parameter and an end position parameter corresponding to the Nth lake entry operation, the third parameter being used to indicate a position of incremental data last obtained by the lake entry execution component in the message middleware in the incremental data obtained by the lake entry execution component.

6. The method of claim 5, wherein, The lake entry execution component acquires, in response to the data lake entry instruction, a first parameter and an end position parameter corresponding to the Nth lake entry operation, including: The lake entry execution component acquires, in response to the data lake entry instruction, a first parameter and a fourth parameter corresponding to the Nth lake entry operation, the fourth parameter being used to indicate a position of incremental data obtained by the message middleware at a preset end time point in the message middleware; The lake entry execution component acquires, in response to the data lake entry instruction, a second parameter and an end position parameter corresponding to the Nth lake entry operation, including: The lake entry execution component acquires, in response to the data lake entry instruction, a second parameter and a fifth parameter corresponding to the Nth lake entry operation, the fifth parameter being used to indicate a preset end position of the message middleware.

7. An electronic device, comprising: The electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the data lake entry method of any one of claims 1 to 6 when executing the computer program.

8. A computer-readable storage medium storing a computer program, the computer-readable storage medium comprising instructions that, when executed by a computer, cause the computer to perform the method of any one of claims 1 to 7. The computer program is executed by the processor to implement the data lake entry method of any one of claims 1 to 6.

Images