Data lake entering method, electronic equipment and medium
By introducing lake entry execution components and data compression components into the data management system, batch entry of database incremental data is realized, the problem of resource occupation by streaming lake entry is solved, and the resource utilization rate of the data management system is improved.
Patent Information
- Application Number
- CN202411178194.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-26
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2044-08-26
AI Technical Summary
The existing data entry method mainly adopts the flow entry method, which causes the resident process to occupy a large amount of resources, reduces the resource utilization rate of the data management system equipment cluster, and makes it difficult to achieve batch entry of database incremental data.
By introducing the lake-entry execution component in the data management system, receiving the data entry instructions sent by the lake-entry task scheduling component between intervals, obtaining the starting and ending position parameters, batch obtaining incremental data from the message middleware, and managing components through the data lake Import the data lake to realize batch entry of incremental data, and at the same time, data compression is carried out when necessary to optimize resource utilization.
It realizes batch entry of incremental data of databases, releases resources of the data management system equipment cluster, improves resource utilization, and avoids the continuous occupation of resources by streaming tasks.
Smart Images

Figure CN119065814A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of data management, and in particular to a method for entering data into a lake, an electronic device, and a medium. Background Art
[0002] As a cluster component that can store massive amounts of data of any type (including structured, semi-structured and unstructured data), data lakes are widely used in network architectures in various industries. For example, e-commerce platforms generate a large amount of order data every day, which is stored in the database. The change records of order data in the database need to be consumed by various downstream platform tasks, so the change records of order data need to be processed as incremental data of the database into the lake for consumption by various downstream platform tasks.
[0003] At present, data entry into the lake is often implemented using a platform or component that manages the data lake. The main form of entry of incremental data generated by the database that needs to be entered into the lake is streaming. Streaming entry into the lake requires the data entry execution component in the data management system to start a streaming task of a resident process to continuously consume the incremental data transmitted by the database through the message intermediate component.
[0004] However, as a resident process, streaming tasks occupy a large amount of resources in the distributed data management system device cluster, making it impossible to dynamically allocate the occupied resources reasonably, reducing the resource utilization of the device cluster. Therefore, how to realize the batch entry of incremental database data into the lake is an issue that technical personnel need to consider at this stage. Summary of the invention
[0005] The main purpose of the embodiments of the present application is to propose a method, electronic device and medium for entering data into a lake, aiming to realize batch entry of incremental data of a database into the lake.
[0006] To achieve the above-mentioned purpose, a first aspect of an embodiment of the present application proposes a method for entering data into a lake, which is applied to a data management system. The method includes:
[0007] The lake entry execution component in the data management system receives a data lake entry instruction sent by the lake entry task scheduling component in the data management system at intervals of a preset time period, and the data lake entry instruction is used to instruct the lake entry execution component to perform an Nth lake entry operation, where N is a positive integer;
[0008] The lake entry execution component responds to the data lake entry instruction, obtains a starting position parameter and an ending position parameter corresponding to the Nth lake entry operation, wherein the starting position parameter and the ending position parameter are used to indicate different data positions in the message middleware in the data management system, and the message middleware is used to obtain incremental data from the database in the data management system;
[0009] The lake entry execution component obtains target incremental data from the message middleware according to the starting position parameter and the ending position parameter corresponding to the Nth lake entry operation;
[0010] The lake entry execution component imports the target incremental data into the data lake in the data management system through the data lake management component in the data management system.
[0011] In some implementations, the lake entry execution component obtains a starting position parameter and an ending position parameter corresponding to the Nth lake entry operation in response to the data lake entry instruction, including:
[0012] The entry-into-the-lake execution component responds to the data entry-into-the-lake instruction to obtain a first parameter and an end position parameter corresponding to the Nth entry-into-the-lake operation, wherein the first parameter is used to indicate the position of the incremental data obtained from the database by the message middleware at a preset starting time point in the message middleware; or
[0013] The lake entry execution component responds to the data lake entry instruction to obtain a second parameter and an end position parameter corresponding to the Nth lake entry operation, wherein the second parameter is used to indicate a preset starting position of the message middleware; or
[0014] The lake entry execution component responds to the data lake entry instruction to obtain a third parameter and an end position parameter corresponding to the Nth lake entry operation, wherein the third parameter is used to indicate the position of the incremental data last obtained by the lake entry execution component among the incremental data obtained by the lake entry execution component in the message middleware.
[0015] In some implementations, the lake entry execution component obtains a first parameter and an end position parameter corresponding to the Nth lake entry operation in response to the data lake entry instruction, including:
[0016] The entry-into-lake execution component obtains, in response to the data entry-into-lake instruction, a first parameter and a fourth parameter corresponding to the Nth entry-into-lake operation, wherein the fourth parameter is used to indicate the position of the incremental data obtained by the message middleware at a preset end time point in the message middleware;
[0017] The lake entry execution component obtains the second parameter and the end position parameter corresponding to the Nth lake entry operation in response to the data lake entry instruction, including:
[0018] The entry-into-the-lake execution component obtains, in response to the data entry-into-the-lake instruction, a second parameter and a fifth parameter corresponding to the Nth entry-into-the-lake operation, wherein the fifth parameter is used to indicate a preset end position of the message middleware, or to indicate a position in the message middleware of the latest incremental data obtained by the message middleware from the database;
[0019] The lake entry execution component obtains a third parameter and an end position parameter corresponding to the Nth lake entry operation in response to the data lake entry instruction, including:
[0020] The lake entry execution component obtains the third parameter and the fifth parameter corresponding to the Nth lake entry operation in response to the data lake entry instruction.
[0021] In some embodiments, the method further comprises:
[0022] The data compression component in the data management system controls the data lake management component to perform data compression on the data in the data lake.
[0023] In some implementations, before the data compression component in the data management system controls the data lake management component to perform data compression on the data in the data lake, the method further includes:
[0024] The data compression component controls the data lake management component to obtain the time point when the data lake was last updated;
[0025] The data compression component in the data management system controls the data lake management component to perform data compression on the data in the data lake, including:
[0026] When the time difference is greater than a preset time difference threshold, the data compression component in the data management system controls the data lake management component to perform data compression on the data in the data lake, wherein the time difference is the difference between the time point when the data lake was last updated and the time point when the data lake was last compressed.
[0027] In some implementations, before the data compression component in the data management system controls the data lake management component to perform data compression on the data in the data lake, the method further includes:
[0028] The data compression component controls the data lake management component to obtain the number of data updates of the data lake, wherein the number of data updates is the number of data updates performed after the data lake is last compressed;
[0029] The data compression component in the data management system controls the data lake management component to perform data compression on the data in the data lake, including:
[0030] When the number of data updates is greater than a preset number threshold, the data compression component in the data management system controls the data lake management component to perform data compression on the data lake.
[0031] In some implementations, before the data compression component controls the data lake management component to obtain the time point at which the data lake was last updated, the method further includes:
[0032] The data compression component controls the data lake management component to close a metadata configuration function of the data lake management component, so that the data lake management component is prohibited from obtaining metadata that has been statically cached in the data lake.
[0033] In some implementations, the data compression component in the data management system controls the data lake management component to perform data compression on the data in the data lake, including:
[0034] The data compression component controls the data lake management component to generate a data compression plan;
[0035] The data compression component controls the data lake management component to enable the metadata configuration function, so that the data lake management component generates corresponding metadata for new data based on the metadata configuration function, wherein the new data is the data in the data lake after the data compression;
[0036] When the metadata configuration function of the data lake management component is turned on, the data compression component controls the data lake management component to perform data compression according to the data compression plan.
[0037] To achieve the above-mentioned purpose, the second aspect of an embodiment of the present application proposes an electronic device, which includes a memory and a processor, the memory stores a computer program, and the processor implements the data entry method described in the first aspect when executing the computer program.
[0038] To achieve the above-mentioned purpose, the third aspect of an embodiment of the present application proposes a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the data entry method described in the first aspect is implemented.
[0039] In an embodiment of the present application, the lake entry execution component in the data management system responds to the data lake entry instruction sent by the lake entry scheduling component at intervals of a preset time period to perform the lake entry operation. The lake entry execution component first obtains the starting position parameter and the ending position parameter corresponding to the lake entry operation, and obtains the target incremental data of the database from the message middleware according to the data position indicated by the starting position parameter and the ending position parameter, and imports the acquired target incremental data into the data lake through the data lake management component. In this way, the lake entry execution component realizes the batch entry of incremental data into the lake, releases the resources in the data management system device cluster, enables the resources in the data management system cluster devices to be reasonably and dynamically allocated, and improves the resource utilization of the data management system device cluster. BRIEF DESCRIPTION OF THE DRAWINGS
[0040] Figure 1 It is a flowchart of the data entry method provided in the embodiment of the present application;
[0041] Figure 2 is an interactive schematic diagram of a data management system provided in an embodiment of the present application;
[0042] Figure 3 It is a flow logic diagram of the data compression process provided by the embodiment of the present application;
[0043] Figure 4 is a flowchart of a data compression process provided by an embodiment of the present application;
[0044] Figure 5 It is a structural diagram of a data management system provided in an embodiment of the present application;
[0045] Figure 6 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION
[0046] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.
[0047] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.
[0048] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.
[0049] First, some nouns involved in this application are analyzed:
[0050] Data entry: Data entry refers to the process of importing data from various data sources into a data lake. A data lake is a repository for storing large amounts of raw data, which can accommodate structured, semi-structured, and unstructured data. The entry process usually involves the following steps:
[0051] Data collection – Collecting data from various data sources (e.g., databases, log files, sensors, social media, etc.);
[0052] Data cleaning - Before importing data into the data lake, it is usually necessary to clean and pre-process the data. This includes removing duplicate data, filling missing values, correcting errors, etc.
[0053] Data conversion: convert the data format according to the requirements. For example, convert the data in CSV (Comma-Separated Values) format to Parquet or Avro format to improve storage and query efficiency;
[0054] Data transfer – transfer data from the source system into the data lake. This can be done using a data lake management platform / component such as Apache Hudi or other data transfer tools such as Apache NiFi.
[0055] Data storage – Store the processed data in a data lake. Data lakes typically use distributed storage systems such as Hadoop or cloud storage (e.g. Amazon S3, Azure Data Lake Storage).
[0056] Metadata management - Maintaining metadata information of the data lake, including the source, structure, quality, etc. of the data, which helps in the organization and retrieval of data in the data lake.
[0057] At present, data entry into the lake is often implemented using a platform or component that manages the data lake. The main form of entry of incremental data generated by the database that needs to be entered into the lake is streaming. Streaming entry into the lake requires the data entry execution component in the data management system to start a streaming task of a resident process to continuously consume the incremental data transmitted by the database through the message intermediate component.
[0058] However, as a resident process, the streaming task occupies a large amount of resources in the data management system device cluster, making it impossible to reasonably and dynamically allocate the occupied resources, thereby reducing the resource utilization of the device cluster.
[0059] Based on this, the embodiments of the present application provide a method, device, electronic device and medium for entering data into a lake, aiming to realize batch entry of incremental data of a database into a lake.
[0060] The data entry method, device, electronic device and medium provided in the embodiments of the present application are specifically described through the following embodiments. First, the data entry method in the embodiments of the present application is described.
[0061] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.
[0062] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.
[0063] The data entry method provided in the embodiment of the present application relates to the field of data management. The data entry method provided in the embodiment of the present application can be applied to the terminal, can also be applied to the server side, and can also be software running in the terminal or the server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or a server cluster or distributed system composed of multiple physical servers, and can also be configured as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements the data entry method, etc., but is not limited to the above forms.
[0064] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.
[0065] It should be noted that in each specific implementation of the present application, when it comes to the need to perform relevant processing based on data related to user identity or characteristics such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first, and the collection, use, and processing of these data will comply with relevant laws, regulations, and standards. In addition, when the embodiment of the present application needs to obtain the user's sensitive personal information, the user's separate permission or consent will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the user's separate permission or consent, the necessary user-related data for the normal operation of the embodiment of the present application will be obtained.
[0066] Figure 1 It is a flowchart of the data entry method provided in an embodiment of the present application. The data entry method provided in an embodiment of the present application is applied to a data management system, and may include but is not limited to steps S110 to S140.
[0067] Step S110: the lake entry execution component in the data management system receives the data lake entry instruction sent by the lake entry task scheduling component in the data management system at intervals of a preset time period. The data lake entry instruction is used to instruct the lake entry execution component to perform the Nth lake entry operation, where N is a positive integer.
[0068] Figure 2 This is an interactive diagram of the data management system provided in the embodiment of the present application. Please refer to Figure 1 to Figure 2 The data management system is a system for managing database and data lake data. The data management system can be a system applied to different business scenarios such as financial business, insurance business or medical business. In the embodiment of the present application, the data management system includes a database, a data lake, a lake entry execution component, a lake entry task scheduling component, a message middleware and a data lake management component.
[0069] Among them, the message middleware plays the role of data transfer and is used to obtain incremental data in the database. The incremental data in the database can be transmitted from the database to the message middleware through a transmission component. The lake execution component then consumes and obtains the incremental data from the message middleware, and uses the data lake management component to import the incremental data into the data lake.
[0070] Those skilled in the art will appreciate that the data management system provided in the embodiment of the present application may be a distributed data management system, that is, each device in the data management system device cluster has the functions of all components in the data management system, and can implement the data entry method provided in the embodiment of the present application.
[0071] Exemplarily, the database may be Oracle, which may use tables to store data and perform data query and management through Structured Query Language (SQL). The database may also be MySQL, which may run on various operating systems.
[0072] The execution component entering the lake can be Apache Flink, which is an open source data processing engine used for large-scale data processing and real-time data stream analysis. It can provide distributed data processing functions with high throughput, low latency, fault tolerance and accuracy, and is widely used in real-time data analysis, data pipelines, event-driven applications and other scenarios.
[0073] The task scheduling component in the lake can be the LinkDo task scheduling system. The LinkDo task scheduling system is an asynchronous task scheduling system, which is mainly used in scenarios such as big data platform projects, BI projects, and data mining projects. It is committed to solving the scheduling problems of enterprise-level asynchronous tasks and supports the relationship control and scheduling execution of various data processing tasks.
[0074] The message middleware may be Apache Kafka. Apache Kafka is an open source stream processing platform for building real-time data pipelines and stream applications. It can process and transmit high-throughput real-time data streams and has high scalability and fault tolerance.
[0075] The data lake management component can be Apache Hud i, which is a widely used data lake platform that can directly introduce core warehouse and database functions into the data lake, provide tables, transactions, advanced indexes, stream ingestion services, data clustering / compression optimization and concurrency, and maintain the open source file format of the data; the transmission component can directly adopt the data transmission cloud service (Data Transfer Service, DTS).
[0076] For example, the default data consumption logic of the Apache Flink lake entry execution component is streaming consumption, that is, a streaming task is started to complete the data entry operation. In order to complete the batch entry of database incremental data into the lake and prevent the streaming tasks started by the lake entry execution component from occupying cluster resources, the lake entry execution component is configured to be silent in the normal state and only respond to the data entry instruction issued by the lake entry task scheduling component to perform the lake entry operation. The lake entry task scheduling component sends a data entry instruction to the lake entry execution component at preset time ends, instructing the lake entry execution component to perform the lake entry operation.
[0077] Exemplarily, the database is a database of an e-commerce platform, which stores a large amount of order data, and new order data is constantly entering the database. The database has a log tool such as Binlog, which records the changes in order data. The log data that characterizes these changes and the newly added order data in the database can be used as incremental data of the database. The incremental data of the database needs to be imported into the data lake for use by downstream tasks (such as order volume growth analysis tasks). The lake entry task scheduling component sends a data entry instruction to the lake entry execution component every hour. The lake entry execution component responds to the data entry instruction sent by the lake entry task scheduling component and performs a data entry operation. That is, the lake entry execution component performs a data entry operation every hour, rather than using an uninterrupted resident streaming task to perform the data entry operation. When the lake entry execution component performs the Nth lake entry operation, it responds to the Nth data entry instruction issued by the lake entry task scheduling component.
[0078] Step S120: The entry-lake execution component responds to the data entry-lake instruction and obtains the starting position parameter and the ending position parameter corresponding to the Nth entry-lake operation. The starting position parameter and the ending position parameter are used to indicate different data positions in the message middleware in the data management system. The message middleware is used to obtain incremental data from the database in the data management system.
[0079] The lake entry execution component responds to the data entry instruction and executes the lake entry operation. The lake entry execution component first obtains the start position parameter and end position parameter that have been configured for itself and correspond to this lake entry operation. The start position parameter and the end position parameter are parameters set for the message middleware and are used to indicate different data positions in the message middleware, so that the lake entry execution component batch obtains incremental data in the message middleware according to the different data positions (the start position of data acquisition, the end position of data acquisition) indicated by the start position parameter and the end position parameter.
[0080] The default working mode of the lake entry execution component can be a streaming lake entry working mode. When the start position parameter and the end position parameter are obtained, it is determined that the working mode is changed to a batch lake entry working mode.
[0081] Exemplarily, the starting position parameters and the ending position parameters corresponding to each entry operation of the entry-lake execution component can be pre-configured in the entry-lake execution component. The starting position parameters and the ending position parameters corresponding to each entry operation can be the same parameter or different parameters depending on the parameter type. The starting position parameters and the ending position parameters can be in the form of timestamps, that is, indicating the position of the incremental data obtained at a certain point in time in the message middleware. The timestamp corresponding to each entry operation is different.
[0082] Step S130: the entry-to-lake execution component obtains target incremental data from the message middleware according to the starting position parameter and the ending position parameter corresponding to the Nth entry-to-lake operation.
[0083] After obtaining the starting position parameter and the ending position parameter corresponding to this operation, the entry lake execution component obtains incremental data in batches from the message middleware according to the obtained starting position parameter and the ending position parameter, and the obtained incremental data is used as the target incremental data for subsequent data entry operations. That is, the embodiment of the present application completes the batch acquisition of incremental data by making the entry lake execution component work in intervals, and calibrating the starting position and the ending position of the data obtained by the entry lake execution component in the message middleware through the starting position parameter and the ending position parameter.
[0084] Step S140: The lake entry execution component imports the target incremental data into the data lake in the data management system through the data lake management component in the data management system.
[0085] After the lake entry execution component obtains the target incremental data in batches in the message middleware, it uses the data lake management component to import the target incremental data into the data lake, completes the batch processing of data entering the lake, ends this data entry operation, and waits for the next data entry instruction.
[0086] For example, in a financial institution's credit management system (i.e., data management system), the system needs to continuously monitor and analyze customer loan data to assess risks and make decisions. In order to improve the efficiency and accuracy of data analysis, the financial institution decides to regularly import incremental data in its core loan database (such as newly applied loans, repayment records, etc.) into the data lake for subsequent analysis tasks.
[0087] The data entry task scheduling component in the credit management system is configured to send a data entry instruction to the data entry execution component every hour. This instruction is used to instruct the data entry execution component to start a new data entry operation (such as the Nth operation). The system ensures that incremental data is imported once an hour, avoiding the problem of real-time streaming tasks continuously occupying system resources, while ensuring timely data updates.
[0088] After receiving the Nth data entry instruction, the entry execution component first obtains the starting position parameters and ending position parameters corresponding to this entry operation. Based on the obtained starting position and ending position parameters, the entry execution component batch obtains the newly added loan data since the last entry operation from the message middleware, such as newly submitted loan applications, recent repayment records, etc. These incremental data are marked as the target incremental data for this operation.
[0089] The lake entry execution component uses the data lake management component to import the target incremental data into the data lake. In the data lake, these incremental data will be integrated into the existing data set for use by downstream tasks such as risk analysis models and credit scoring models. After the lake entry is completed, the lake entry execution component waits for the next data entry instruction.
[0090] Through the above steps S110-S140, the lake entry execution component in the data management system responds to the data lake entry instruction sent by the lake entry scheduling component at intervals of a preset time period, and executes the lake entry operation. The lake entry execution component first obtains the starting position parameter and the ending position parameter corresponding to the lake entry operation, and obtains the target incremental data of the database from the message middleware according to the data position indicated by the starting position parameter and the ending position parameter, and imports the acquired target incremental data into the data lake through the data lake management component. In this way, the lake entry execution component realizes the batch entry of incremental data into the lake, releases the resources in the data management system device cluster, enables the resources in the data management system cluster devices to be reasonably and dynamically allocated, and improves the resource utilization of the data management system device cluster.
[0091] In some implementations, the lake entry execution component obtains the starting position parameter and the ending position parameter corresponding to the Nth lake entry operation in response to the data lake entry instruction, including:
[0092] The entry-into-lake execution component responds to the data entry-into-lake instruction and obtains the first parameter and the end position parameter corresponding to the Nth entry-into-lake operation, where the first parameter is used to indicate the position of the incremental data obtained from the database in the message middleware at a preset starting time point; or
[0093] The entry-into-lake execution component responds to the data entry-into-lake instruction and obtains a second parameter and an end position parameter corresponding to the Nth entry-into-lake operation, where the second parameter is used to indicate a preset starting position of the message middleware; or
[0094] The lake entry execution component responds to the data entry instruction and obtains the third parameter and the end position parameter corresponding to the Nth lake entry operation. The third parameter is used to indicate the position of the incremental data that has been obtained by the lake entry execution component and the last incremental data obtained by the lake entry execution component in the message middleware.
[0095] The starting position parameter may be one of multiple types, and the multiple types may include a first parameter, a second parameter, and a third parameter. In this way, the entry execution component can start data acquisition according to the starting position indicated by the starting position parameter.
[0096] The first parameter is a timestamp parameter, which indicates the location of the incremental data in the message middleware obtained from the database at a preset starting time point. This location is the starting position for the lake entry execution component to obtain this data.
[0097] Exemplarily, the first parameter indicates that the starting position for the lake entry execution component to acquire data in this lake entry operation is the position of the incremental data acquired by the message middleware at 9:00 a.m. in the message middleware. If the message middleware is Kafka (data position is calibrated by partition and offset), and the incremental data acquired by the message middleware at 9:00 a.m. is at the position of "Partition C Offset 20" in the message middleware, then "Partition C Offset 20" is the starting position for the lake entry execution component to acquire data this time.
[0098] The second parameter is a fixed position parameter, indicating that a specific preset starting position of the message middleware is the starting position for the entry-lake execution component to obtain data for this entry-lake operation.
[0099] Exemplarily, if the message middleware is Kafka, and the preset starting position of this entry into the lake operation is the position of "partition C offset 20", then "partition C offset 20" is the starting position for the entry into the lake execution component to obtain this data.
[0100] The third parameter is a non-fixed position parameter, indicating the position of the last incremental data acquired by the lake entry execution component in the message middleware, that is, the position of the last data acquired by the lake entry execution component in the message middleware in the last lake entry operation. The third parameter indicates that the lake entry execution component starts to acquire new data from the position where the last data acquisition was completed. In the case where the starting position parameter is the third parameter, the starting position parameter corresponding to each lake entry operation can be the same parameter.
[0101] Exemplarily, if the message middleware is Kafka, when the entry-to-lake execution component performs the N-1th entry-to-lake operation, it obtains data from the message middleware until the position of "Partition C offset 20" of the message middleware. When the starting position parameter is the third parameter, it indicates that in the Nth entry-to-lake operation, the starting position of data acquisition is "Partition C offset 20".
[0102] In some implementations, the lake entry execution component obtains the first parameter and the end position parameter corresponding to the Nth lake entry operation in response to the data lake entry instruction, including:
[0103] The entry lake execution component responds to the data entry lake instruction and obtains the first parameter and the fourth parameter corresponding to the Nth entry lake operation, where the fourth parameter is used to indicate the location of the incremental data obtained by the message middleware at the preset end time point in the message middleware;
[0104] The lake entry execution component responds to the data lake entry instruction and obtains the second parameter and the end position parameter corresponding to the Nth lake entry operation, including:
[0105] The entry lake execution component responds to the data entry lake instruction and obtains the second parameter and the fifth parameter corresponding to the Nth entry lake operation, where the fifth parameter is used to indicate the preset end position of the message middleware, or the position of the latest incremental data obtained by the message middleware from the database in the message middleware;
[0106] The lake entry execution component responds to the data lake entry instruction and obtains the third parameter and the end position parameter corresponding to the Nth lake entry operation, including:
[0107] The lake entry execution component responds to the data lake entry instruction and obtains the third parameter and the fifth parameter corresponding to the Nth lake entry operation.
[0108] The end position parameter corresponds to the start position parameter, so that the entry execution component can end data acquisition according to the end position indicated by the end position parameter to complete the batch acquisition of incremental data.
[0109] When the starting position parameter is the first parameter, that is, the timestamp parameter, the ending position parameter can only be the timestamp parameter, that is, the fourth parameter, indicating the position of the incremental data obtained by the message middleware at the preset end time point in the message middleware.
[0110] Exemplarily, the fourth parameter indicates that the end position of data acquisition performed by the entry-lake execution component in this entry-lake operation is the position of the incremental data obtained by the message middleware at 9:00 p.m. in the message middleware. If the message middleware is Kafka (data position is calibrated by partition and offset), and the incremental data obtained by the message middleware at 9:00 p.m. is at the position of "Partition C Offset 20" in the message middleware, then "Partition C Offset 20" is the end position of the data acquisition performed by the entry-lake execution component.
[0111] When the starting position parameter is the second position parameter or the third position parameter, the ending position parameter can be the fifth parameter. The fifth parameter can indicate the preset ending position of the message middleware, that is, indicate that a specific preset ending position of the message middleware is the ending position of the entry-lake execution component for acquiring data for this entry-lake operation. It can also indicate the position of the latest incremental data acquired by the message middleware from the database in the message middleware, that is, indicate that the entry-lake execution component should acquire the latest incremental data in the message middleware during this data acquisition.
[0112] Exemplarily, if the message middleware is Kafka, and the fifth parameter indicates that the preset end position of this entry-lake operation is the position of "Partition C offset 20", then "Partition C offset 20" is the end position of the data acquisition performed by the entry-lake execution component; if the message middleware is Kafka, and the fifth parameter indicates that the end position is the position of the latest incremental data obtained by the message middleware from the database in the message middleware, if the position of the latest incremental data obtained by the message middleware from the database in the message middleware is "Partition C offset 20", then "Partition C offset 20" is the end position of the data acquisition performed by the entry-lake execution component.
[0113] In some embodiments, the method may further include:
[0114] The data compression component in the data management system controls the data lake management component to compress the data in the data lake.
[0115] The newly entered data is in a fragmented state without compression, which is not convenient for indexing and searching. Therefore, it is necessary to compress the data in the data lake. When the lake entry execution component performs the lake entry operation in a streaming task, in the streaming lake entry mode, the data enters the data lake in real time through the data lake management component such as Hudi, which triggers the immediate data compression operation of the data lake management component to ensure the organization and optimization of the data in the data lake; but in the lake entry execution component performs batch entry mode, the data is imported into the data lake through the data lake management component in batches. After each batch of data is imported into the data lake, the data lake management component will end the import task and will not immediately trigger the data compression operation. Therefore, in the batch entry mode, a data compression component is needed to control the data lake management component to compress the data in the data lake. The data compression component can be configured with start condition parameters, such as starting the data compression operation when the interval time reaches the preset threshold, and controlling the data lake management component to compress the data in the data lake. In this way, in the batch data entry mode, the data in the data lake can be compressed in a timely manner to ensure the rationality of the organization of the data in the data lake.
[0116] Exemplarily, the data compression component may be Apache Spark, which is a fast unified analysis engine for large-scale data processing. It has the characteristics of high speed, ease of use, and strong versatility, and is widely used in fields such as data analysis, machine learning, and big data processing. The data compression component may be configured to start a data compression operation every hour, that is, to control the data lake management component to compress the data in the data lake every hour.
[0117] In some implementations, before the data compression component in the data management system controls the data lake management component to perform data compression on the data in the data lake, the method may further include:
[0118] The data compression component controls the data lake management component to obtain the time point when the data lake was last updated;
[0119] The data compression component in the data management system controls the data lake management component to compress the data in the data lake, including:
[0120] When the time difference is greater than a preset time difference threshold, the data compression component in the data management system controls the data lake management component to compress the data in the data lake, where the time difference is the difference between the time point when the data lake was last updated and the time point when the data lake was last compressed.
[0121] The data compression component first controls the data lake management component to obtain the time point when the data lake last updated data, that is, the time point when the data lake last generated incremental data, and then calculates the time difference between this time point and the time point when the data lake last compressed data. If the time difference is greater than the preset time difference threshold, it means that the length of time that the data lake has not been compressed has reached the maximum limit, and there is a large amount of data in the data lake that needs to be compressed. At this time, the data compression component can control the data lake management component to compress the data in the data lake. In this way, the data in the data lake can be compressed in a timely manner to ensure the rationality of the organization of the data in the data lake.
[0122] In some implementations, before the data compression component in the data management system controls the data lake management component to perform data compression on the data in the data lake, the method may further include:
[0123] The data compression component controls the data lake management component to obtain the number of data updates in the data lake, where the number of data updates is the number of data updates performed after the data lake performs the last data compression;
[0124] The data compression component in the data management system controls the data lake management component to compress the data in the data lake, including:
[0125] When the number of data updates is greater than a preset number threshold, the data compression component in the data management system controls the data lake management component to compress the data in the data lake.
[0126] The criterion for determining whether the data compression component controls the data lake management component to compress the data in the data lake can also be the number of data updates after the last data compression of the data lake, that is, the number of times incremental data is generated. If the number of data updates is greater than the preset threshold, it means that the length of time that the data lake has not been compressed has reached the maximum limit, and there is a large amount of data in the data lake that needs to be compressed. At this time, the data compression component can control the data lake management component to compress the data in the data lake. In this way, the data in the data lake can be compressed in a timely manner to ensure the rationality of the organization of the data in the data lake.
[0127] Figure 3 is a flow logic diagram of the data compression process provided by the embodiment of the present application, such as Figure 3As shown, those skilled in the art can understand that the judgment criterion for whether the data compression component controls the data lake management component to compress the data in the data lake can be the difference between the time point when the data lake was last updated and the time point when the data lake was last compressed, or the number of data updates after the data lake was last compressed. When any one of the above difference (time difference) and the number of times meets the data compression condition (the difference is greater than the preset time difference threshold, or the number of times is greater than the preset number threshold), the data compression component can control the data lake management component to compress the data in the data lake.
[0128] In some implementations, before the data compression component controls the data lake management component to obtain the time point at which the data lake is last updated, the method further includes:
[0129] The data compression component controls the data lake management component to close the metadata configuration function of the data lake management component, so that the data lake management component is prohibited from obtaining metadata that has been statically cached in the data lake.
[0130] Figure 4 is a flow chart of the data compression process provided by the embodiment of the present application, such as Figure 4 As shown in the figure, the data lake management component has the metadata configuration function, which generates metadata for the data in the data lake and processes the data in the data lake based on the metadata. Metadata is data about data, describing the structure, content, context and characteristics of the data. Metadata can help users or platforms understand and use data, and provide background and related information for the data, making the data easier to find, manage and analyze.
[0131] However, metadata generation by data lake management components like Hud i often lags behind data updates in the data lake, i.e., new data has entered the data lake, but the data management component has not generated corresponding metadata for the new data and cached it statically in a timely manner. Therefore, if in a stock pool that is performing data compression, the data lake management component compresses the data in the data lake based on the metadata that has been generated and statically cached, then the new data in the data lake cannot be scanned by the data lake management component, and the new data cannot be compressed.
[0132] Therefore, before the data compression component determines whether to control the data lake management component to compress the data in the data lake, that is, before the data compression component controls the data lake management component to obtain the time point of the last data update of the data lake, or before the data compression component controls the data lake management component to obtain the number of data updates in the data lake, the data compression component controls the data lake management component to close the metadata configuration function of the data lake management component, so that the data lake management component is prohibited from obtaining the metadata that has been statically cached in the data lake. The data lake management component can only perform data compression by scanning all data in the data lake to ensure that all data in the data lake (including new data for which corresponding metadata has not been generated in time) can be compressed.
[0133] In some implementations, the data compression component in the data management system controls the data lake management component to perform data compression on the data in the data lake, including:
[0134] The data compression component controls the data lake management component to generate a data compression plan;
[0135] The data compression component controls the data lake management component to enable the metadata configuration function, so that the data lake management component generates corresponding metadata for new data based on the metadata configuration function, wherein the new data is data in the data lake that has been compressed;
[0136] When the metadata configuration function of the data lake management component is enabled, the data compression component controls the data lake management component to perform data compression according to the data compression plan.
[0137] The specific process of data compression can be divided into two steps. The first step is that the data compression component controls the data lake management component to generate a data compression plan. The data compression plan can be a specific compression plan file. The second step is that the data compression component controls the data lake management component to compress data according to the data compression plan.
[0138] During the first step of the above process, when the metadata configuration function of the data lake management component has been turned off, the data lake management component generates a data compression plan by scanning all the data in the data lake.
[0139] Therefore, after the data compression plan is generated, the metadata configuration function of the data lake management component can be enabled. Because the data compression plan has been generated, the metadata configuration function will not affect the second step in the above process. Therefore, after the data compression plan is generated, the metadata configuration function of the data lake management component is enabled, so that the data lake management component can generate corresponding metadata for the compressed new data, ensure the integrity of the metadata in the data lake, and make the data in the data lake easier to manage and find.
[0140] See also Figure 5 , Figure 5 is a structural diagram of a data management system provided in an embodiment of the present application. The embodiment of the present application also provides a data management system 200, which can implement the above-mentioned data entry method. The system 200 includes:
[0141] The lake entry task scheduling component 210 is used to send the data lake entry instruction at every preset time interval;
[0142] Database 220, used to generate incremental data;
[0143] The message middleware 230 is used to obtain incremental data from the database;
[0144] The lake entry execution component 240 is used to perform the following steps:
[0145] Receive the data entry instruction sent by the entry task scheduling component in the data management system at intervals of a preset time period, the data entry instruction is used to instruct the entry execution component to perform the Nth entry operation, where N is a positive integer;
[0146] In response to the data entry instruction, obtain the starting position parameter and the ending position parameter corresponding to the Nth entry operation, where the starting position parameter and the ending position parameter are used to indicate different data positions in the message middleware in the data management system;
[0147] Obtain target incremental data from the message middleware according to the start position parameter and end position parameter corresponding to the Nth entry operation;
[0148] Import the target incremental data into the data lake through the data lake management component;
[0149] The data lake management component 250 is used to import the target incremental data into the data lake;
[0150] Data lake 260 is used to store target incremental data.
[0151] Those skilled in the art can understand that the data management system provided in the embodiment of the present application is a distributed data management system, that is, each device in the data management system device cluster has the functions of all components in the data management system, and can implement the data entry method provided in the embodiment of the present application.
[0152] The specific implementation of the data management system 200 corresponds to the specific implementation example of the above-mentioned data entry method, which will not be repeated here.
[0153] The embodiment of the present application also provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above data entry method when executing the computer program. The electronic device can be any smart terminal including a tablet computer, a car computer, etc.
[0154] See also Figure 6 , Figure 6 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:
[0155] The processor 901 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application.
[0156] The memory 902 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 902 can store an operating system and other applications. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 902, and the processor 901 calls and executes the data entry method of the embodiment of this application;
[0157] Input / output interface 903, used to implement information input and output;
[0158] Communication interface 904, used to realize communication interaction between the device and other devices, which can be realized through wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WI FI, Bluetooth, etc.);
[0159] A bus 905 that transmits information between various components of the device (e.g., the processor 901, the memory 902, the input / output interface 903, and the communication interface 904);
[0160] The processor 901 , the memory 902 , the input / output interface 903 and the communication interface 904 are connected to each other in communication within the device via a bus 905 .
[0161] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned data entry method is implemented.
[0162] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.
[0163] The data entry method, electronic device and medium provided by the embodiments of the present application, the entry-into-lake execution component in the data management system responds to the data entry-into-lake instruction sent by the entry-into-lake scheduling component at intervals of a preset time period, and executes the entry-into-lake operation. The entry-into-lake execution component first obtains the starting position parameter and the ending position parameter corresponding to the entry-into-lake operation, and obtains the target incremental data of the database from the message middleware according to the data position indicated by the starting position parameter and the ending position parameter, and imports the acquired target incremental data into the data lake through the data lake management component. In this way, the entry-into-lake execution component realizes the batch entry of incremental data into the lake, releases the resources in the data management system device cluster, enables the resources in the data management system cluster devices to be reasonably and dynamically allocated, and improves the resource utilization of the data management system device cluster.
[0164] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.
[0165] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.
[0166] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0167] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.
[0168] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.
[0169] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.
[0170] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.
[0171] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0172] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.
[0173] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.
[0174] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.
Claims
1. A method for entering data into a lake, characterized in that: Applied to a data management system, the method comprises: The lake entry execution component in the data management system receives a data lake entry instruction sent by the lake entry task scheduling component in the data management system at intervals of a preset time period, and the data lake entry instruction is used to instruct the lake entry execution component to perform an Nth lake entry operation, where N is a positive integer; The lake entry execution component responds to the data lake entry instruction, obtains a starting position parameter and an ending position parameter corresponding to the Nth lake entry operation, wherein the starting position parameter and the ending position parameter are used to indicate different data positions in the message middleware in the data management system, and the message middleware is used to obtain incremental data from the database in the data management system; The lake entry execution component obtains target incremental data from the message middleware according to the starting position parameter and the ending position parameter corresponding to the Nth lake entry operation; The lake entry execution component imports the target incremental data into the data lake in the data management system through the data lake management component in the data management system.
2. The method according to claim 1, characterized in that The lake entry execution component obtains the starting position parameter and the ending position parameter corresponding to the Nth lake entry operation in response to the data lake entry instruction, including: The entry-into-the-lake execution component responds to the data entry-into-the-lake instruction to obtain a first parameter and an end position parameter corresponding to the Nth entry-into-the-lake operation, wherein the first parameter is used to indicate the position of the incremental data obtained from the database by the message middleware at a preset starting time point in the message middleware; or The lake entry execution component responds to the data lake entry instruction to obtain a second parameter and an end position parameter corresponding to the Nth lake entry operation, wherein the second parameter is used to indicate a preset starting position of the message middleware; or The lake entry execution component responds to the data lake entry instruction to obtain a third parameter and an end position parameter corresponding to the Nth lake entry operation, wherein the third parameter is used to indicate the position of the incremental data last obtained by the lake entry execution component among the incremental data obtained by the lake entry execution component in the message middleware.
3. The method according to claim 2, characterized in that The lake entry execution component obtains the first parameter and the end position parameter corresponding to the Nth lake entry operation in response to the data lake entry instruction, including: The entry-into-lake execution component obtains, in response to the data entry-into-lake instruction, a first parameter and a fourth parameter corresponding to the Nth entry-into-lake operation, wherein the fourth parameter is used to indicate the position of the incremental data obtained by the message middleware at a preset end time point in the message middleware; The lake entry execution component obtains the second parameter and the end position parameter corresponding to the Nth lake entry operation in response to the data lake entry instruction, including: The entry-into-the-lake execution component obtains, in response to the data entry-into-the-lake instruction, a second parameter and a fifth parameter corresponding to the Nth entry-into-the-lake operation, wherein the fifth parameter is used to indicate a preset end position of the message middleware, or to indicate a position in the message middleware of the latest incremental data obtained by the message middleware from the database; The lake entry execution component obtains a third parameter and an end position parameter corresponding to the Nth lake entry operation in response to the data lake entry instruction, including: The lake entry execution component obtains the third parameter and the fifth parameter corresponding to the Nth lake entry operation in response to the data lake entry instruction.
4. The method according to claim 1, characterized in that: The method further comprises: The data compression component in the data management system controls the data lake management component to perform data compression on the data in the data lake.
5. The method according to claim 4, characterized in that Before the data compression component in the data management system controls the data lake management component to perform data compression on the data in the data lake, the method further includes: The data compression component controls the data lake management component to obtain the time point when the data lake was last updated; The data compression component in the data management system controls the data lake management component to perform data compression on the data in the data lake, including: When the time difference is greater than a preset time difference threshold, the data compression component in the data management system controls the data lake management component to perform data compression on the data in the data lake, wherein the time difference is the difference between the time point when the data lake was last updated and the time point when the data lake was last compressed.
6. The method according to claim 4, characterized in that Before the data compression component in the data management system controls the data lake management component to perform data compression on the data in the data lake, the method further includes: The data compression component controls the data lake management component to obtain the number of data updates of the data lake, wherein the number of data updates is the number of data updates performed after the data lake is last compressed; The data compression component in the data management system controls the data lake management component to perform data compression on the data in the data lake, including: When the number of data updates is greater than a preset number threshold, the data compression component in the data management system controls the data lake management component to perform data compression on the data lake.
7. The method according to claim 5, characterized in that Before the data compression component controls the data lake management component to obtain the time point when the data lake was last updated, the method further includes: The data compression component controls the data lake management component to close a metadata configuration function of the data lake management component, so that the data lake management component is prohibited from obtaining metadata that has been statically cached in the data lake.
8. The method according to claim 7, characterized in that The data compression component in the data management system controls the data lake management component to perform data compression on the data in the data lake, including: The data compression component controls the data lake management component to generate a data compression plan; The data compression component controls the data lake management component to enable the metadata configuration function, so that the data lake management component generates corresponding metadata for new data based on the metadata configuration function, wherein the new data is the data in the data lake after the data compression; When the metadata configuration function of the data lake management component is turned on, the data compression component controls the data lake management component to perform data compression according to the data compression plan.
9. An electronic device, characterized in that: The electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the data entry method according to any one of claims 1 to 8 when executing the computer program.
10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the data entry method according to any one of claims 1 to 8 is implemented.
Citation Information
Patent Citations
System for importing data into a data repository
CN109997125A
Data processing method and device based on big data, computer equipment and medium
CN114356938A
Data processing method and device, storage medium and electronic equipment
CN114528127A
Hudi asynchronous compression method and system based on hotspot prediction
CN117370288A
Data extraction and management
US20240134859A1
