Big data entry method and its system, device, electronic device and storage medium

Abstract

The present invention discloses a method for entering big data into a lake, and its system, device, electronic device and storage medium, which relate to the field of big data or other related fields. The method comprises: reading the business tables and daily cutting table operation records of N data sources, extracting the actual daily cutting time from the daily cutting table operation records, checking whether the actual daily cutting time of all N data sources has arrived, and if so, connecting the lake-entering consumption process, which checks the consumption process, and if the business table operation records have all been consumed before the actual daily cutting time, sending a T-day partition closure notification to the lake-entering consumption process, closing the business table and daily cutting table operation records to the T-day partition, and loading them into the data lake. When it is determined that the business table and daily cutting table operation records have been closed to the T-day partition, writing the delayed data to the T+1 day partition. The present invention solves the technical problem of poor fault tolerance when entering big data into a lake in the related art.

Description

Technical Field

[0001] The present invention relates to the field of big data or other related technical fields, and in particular to a method for entering big data into a lake and its system, device, electronic device and storage medium. Background Art

[0002] With the rapid development of big data technology, real-time data warehouse streaming into the data lake has been widely adopted, enabling the collection, processing, storage, and analysis of real-time data. Related technologies typically employ strong data consistency checks to ensure the quality of the incoming data (i.e., data consistency), but this also reduces the timeliness of rapid big data ingestion. While weakening strong data consistency checks improves the timeliness of rapid data ingestion into the data lake, it cannot guarantee the quality of the incoming data.

[0003] At the same time, related technologies for data ingestion into the lake suffer from deficiencies in fault tolerance, verification, and recovery. This not only increases the pressure on data quality assurance and fails to guarantee data consistency for massive data ingestion, but also significantly reduces the timeliness of massive data ingestion. These deficiencies are primarily manifested in the following aspects.

[0004] 1. Poor fault tolerance in abnormal scenarios.

[0005] The entire link of replication into the lake includes multiple important links such as source business applications, data replication platform, message middleware, big data service platform, data lake, etc. In the multi-database mode, the replication and data consumption speeds of multiple source databases are often different, and the fault-tolerant processing logic is relatively complex. If there is a serious stagnation or delay in the link, it will have a serious impact on the accuracy and timeliness of the data entering the lake.

[0006] 2. There are defects in the data verification implementation mechanism.

[0007] Data consistency risks are typically addressed by establishing an end-to-end upstream and downstream consistency verification mechanism. However, if risk controls for some access applications are lax, this end-to-end verification mechanism will not function effectively, significantly reducing consistency verification capabilities within the data flow process. Once a risk occurs, it will be impossible to quickly locate the point where the data consistency risk occurred.

[0008] 3. The coordination capability for emergency response is not sound.

[0009] When data consistency risks occur, coordinated emergency response measures must be implemented across upstream and downstream applications and platforms related to the data entry link. At the tool level, the recovery capabilities of each link are not yet fully developed, with some links' retention periods failing to meet recovery requirements. At the implementation level, there is currently no standardized process for such emergency response. Once a data consistency risk arises, emergency recovery communication and implementation progress become uncontrollable.

[0010] To address the above-mentioned problems, no effective solutions have been proposed so far. Summary of the Invention

[0011] The embodiments of the present invention provide a method for storing big data in a lake and its system, device, electronic device and storage medium, so as to at least solve the technical problem of poor fault tolerance when storing big data in a lake in related technologies.

[0012] According to one aspect of an embodiment of the present invention, a method for entering a big data lake is provided, comprising: reading business table operation records and daily cut table operation records of N data sources, extracting the actual daily cut time from the daily cut table operation records, wherein N is a positive integer greater than or equal to 1; checking whether the actual daily cut time of all the N data sources is in sync, and if the actual daily cut time of all the N data sources is in sync, connecting the lake entry consumption process, and having the lake entry consumption process check the consumption process, wherein the consumption process refers to the process of converting the business table operation record into a data record; in the business table operation of each data source, When all records have been consumed before the corresponding actual daily cut time, a T-day partition closure notification is sent to the consumption process entering the lake, wherein the consumption process entering the lake responds to the T-day partition closure notification, closes the business table operation records and the daily cut table operation records to the T-day partition, and loads them into the data lake; when it is determined that the business table operation records and the daily cut table operation records have been closed to the T-day partition, the T-day entry batch processing is completed, and the delayed data is written to the T+1 day partition in the daily partition group, wherein the delayed data is the T-day data that arrives after the communication is restored to normal after the T-day partition is closed.

[0013] Optionally, the steps of reading the business table operation records and the daily cutting table operation records of N data sources and extracting the actual daily cutting time from the daily cutting table operation records include: reading the log files of the daily cutting table and the business table in each of the data sources in the business application cluster to obtain the daily cutting table operation records and the business table operation records, wherein the business application cluster includes the N data sources, each of the data sources is used to store the business data of the business application and the basic unit of daily cutting data, each of the data sources includes at least one daily cutting table and the business table, the daily cutting table stores the batch date, actual daily cutting time and daily cutting backup time of each data source, the daily cutting backup time is later than the actual daily cutting time, and the business table stores business data; the business table operation records and the daily cutting table operation records are respectively spliced ​​into messages of a specified format to obtain an operation record message, and the operation record message is sent to the storage unit of the message middleware.

[0014] Optionally, the steps of reading the business table operation records and the day-cutting table operation records of N data sources and extracting the actual day-cutting time from the day-cutting table operation records include: obtaining the day-cutting table operation record message from the storage unit, and extracting the actual day-cutting time from the day-cutting table operation record message; and obtaining the day-cutting backup time from the day-cutting table of the data source.

[0015] Optionally, after sending the operation record message to the storage unit of the message middleware, it also includes: connecting to the lake entry consumption process, and consuming the business table operation record from the storage unit of the message middleware in real time, wherein the real-time consumption refers to writing the business date in the business table operation record into the T-day partition of the formal table in the data lake in real time. The formal table has been pre-sharded by date to obtain the day partition corresponding to each date, and all day partitions are combined into the day partition group. The data lake stores and processes the business table operation record according to the day partition.

[0016] Optionally, after checking whether the actual daily cut-off time of all the N data sources has arrived, it also includes: if the actual daily cut-off time of any of the data sources has not arrived, judging whether it is necessary to send a T-day batch processing request to the operation and maintenance end; if there is no need to send a T-day batch processing request to the operation and maintenance end, connecting the lake entry consumption process, and the lake entry consumption process checks the consumption process of the data source; judging whether the business table operation records before the daily cut-off time have been consumed; if the business table operation records of each of the data sources have been consumed before the corresponding daily cut-off time, sending a T-day partition closure notification to the lake entry consumption process.

[0017] Optionally, the step of checking the consumption process by the consumption process entering the lake includes: for each of the data sources, the consumption process entering the lake determines whether the business table operation records in the distributed processing platform have been consumed before the actual daily cut time.

[0018] Optionally, the step of determining whether the business table operation record in the distributed processing platform has been consumed before the actual day cut time includes: if the business table operation record exists in the distributed processing platform, extracting the processing time corresponding to the business data in the business table operation record; if the processing time exceeds the predetermined duration of the actual day cut time, and the T-day partition has not been updated for more than M minutes, determining that the business data of the T-day has been consumed, where M is a positive integer greater than or equal to 1; if the business table operation record does not exist in the distributed processing platform, the current system time exceeds the predetermined duration of the actual day cut time, and the T-day partition has not been updated for more than M minutes, determining that the business data of the T-day has been consumed.

[0019] Optionally, after the consumption process is checked by the consumption process entering the lake, it also includes: when the business table operation record of any of the data sources has not been consumed before the corresponding actual daily cut time, the consumption process is continuously checked by the consumption process entering the lake.

[0020] According to another aspect of an embodiment of the present invention, a big data entry system is also provided, including: a business application cluster, including N data sources, each of which is used to store business data and daily cutting data of business applications, each of which includes at least one daily cutting table and a business table, the daily cutting table stores the batch date, actual daily cutting time and daily cutting backup time of each data source, the daily cutting backup time is later than the actual daily cutting time, the business table stores business data, and N is a positive integer greater than or equal to 1; a data replication tool is used to read the business table operation records and daily cutting table operation records of the N data sources, extract the actual daily cutting time from the daily cutting table operation records, and send the business table operation records and the daily cutting table operation records to a message middleware; the message middleware includes N daily cutting table storage units and N business table storage units, the daily cutting table storage unit is used to store the operation record message of the daily cutting table, the business table storage unit is used to store the operation record message of the business table, and the message middleware is mainly responsible for the storage and transmission of the operation record message; a big data service platform The platform is configured to connect to the lake entry consumption process when the business table operation records and the daily cut table operation records pass the integrity check and the actual daily cut times of all the N data sources are coincident, and the lake entry consumption process checks the consumption process, wherein the consumption process refers to the process of converting the business table operation records into data records; when the business table operation records of each data source have been consumed before the corresponding actual daily cut time, send a T-day partition closure notification to the lake entry consumption process, wherein the lake entry consumption process responds to the T-day partition closure notification, closes the business table operation records and the daily cut table operation records to the T-day partition, and loads them into the data lake; when it is determined that the business table operation records and the daily cut table operation records have been closed to the T-day partition, complete the lake entry batch processing for T day, and write the delayed data to the T+1 day partition in the daily partition group, wherein the delayed data is the T-day data that arrives after the communication is restored to normal after the T-day partition is closed; the data lake is configured to store and process the business table operation records according to the daily partition.

[0021] According to another aspect of an embodiment of the present invention, a big data entry device is provided, comprising: a record reading unit, for reading the business table operation records and the daily cut table operation records of N data sources, and extracting the actual daily cut time from the daily cut table operation records, wherein N is a positive integer greater than or equal to 1; a consumption checking unit, for checking whether the actual daily cut time of all the N data sources is in sync, and when the actual daily cut time of all the N data sources is in sync, connecting the entry consumption process, and having the entry consumption process check the consumption process, wherein the consumption process refers to the process of converting the business table operation record into a data record; a closed notification unit, for checking at each of the When the business table operation records of the data source have been consumed before the corresponding actual daily cutting time, a T-day partition closure notification is sent to the lake entry consumption process, wherein the lake entry consumption process responds to the T-day partition closure notification, closes the business table operation records and the daily cutting table operation records to the T-day partition, and loads them into the data lake; the closure processing unit is used to complete the T-day lake entry batch processing when it is determined that the business table operation records and the daily cutting table operation records have been closed to the T-day partition, and write the delayed data into the T+1 day partition in the daily partition group, wherein the delayed data is the T-day data that arrives after the communication is restored to normal after the T-day partition is closed.

[0022] Optionally, the record reading unit includes: a first reading module, used to read the log files of the daily cutting table and the business table in each of the data sources in the business application cluster, and obtain the daily cutting table operation records and the business table operation records, wherein the business application cluster includes the N data sources, each of the data sources is used to store the business data of the business application and the basic unit of daily cutting data, and each of the data sources includes at least one daily cutting table and one business table, and the daily cutting table stores the batch date, actual daily cutting time and daily cutting backup time of each data source, and the daily cutting backup time is later than the actual daily cutting time, and the business table stores business data; a splicing module, used to splice the business table operation records and the daily cutting table operation records into messages of a specified format, obtain an operation record message, and send the operation record message to the storage unit of the message middleware.

[0023] Optionally, the record reading unit also includes: a second reading module, used to obtain the day-cutting table operation record message from the storage unit, and extract the actual day-cutting time from the day-cutting table operation record message; an acquisition module, which obtains the day-cutting backup time from the day-cutting table of the data source.

[0024] Optionally, the record reading unit further includes: a consumption module, used to connect to the lake entry consumption process, and consume the business table operation record in real time from the storage unit of the message middleware, wherein the real-time consumption refers to writing the business date in the business table operation record into the T-day partition of the formal table in the data lake in real time. The formal table has been pre-sharded by date to obtain the day partition corresponding to each date, and all day partitions are combined into the day partition group. The data lake stores and processes the business table operation record according to the day partition.

[0025] Optionally, the consumption inspection unit also includes: a request module, used to determine whether it is necessary to send a T-day batch processing request to the operation and maintenance end when the actual daily cutting time of any of the data sources has not arrived; an inspection module, used to connect the lake entry consumption process without sending a T-day batch processing request to the operation and maintenance end, and the lake entry consumption process checks the consumption process of the data source; a first judgment module, used to determine whether the business table operation records before the daily cutting backup time have been consumed; a sending module, which sends a T-day partition closure notification to the lake entry consumption process when the business table operation records of each of the data sources have been consumed before the corresponding daily cutting backup time.

[0026] Optionally, the inspection unit further includes: a second judgment module, configured to determine, for each of the data sources, by the consumption process entering the lake, whether the business table operation records in the distributed processing platform have been consumed before the actual daily cut time.

[0027] Optionally, the second judgment unit also includes: a first judgment module, used to extract the processing time corresponding to the business data in the business table operation record if the business table operation record exists in the distributed processing platform, and when the processing time exceeds the predetermined duration of the actual daily cut time, and the T-day partition has not been updated for more than M minutes, it is determined that the business data of day T has been consumed, where M is a positive integer greater than or equal to 1; a second judgment module, used to determine that the business data of day T has been consumed if the business table operation record does not exist in the distributed processing platform, the current time of the system exceeds the predetermined duration of the actual daily cut time, and the T-day partition has not been updated for more than M minutes.

[0028] Optionally, the consumption inspection unit also includes: after the consumption process is inspected by the consumption process entering the lake, if the business table operation record of any of the data sources has not been consumed before the corresponding actual daily cut time, the consumption process entering the lake continues to inspect the consumption process.

[0029] According to another aspect of an embodiment of the present invention, a computer-readable storage medium is further provided, wherein the computer-readable storage medium includes a stored computer program, wherein when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute any of the above-mentioned methods for entering big data into a lake.

[0030] According to another aspect of an embodiment of the present invention, an electronic device is also provided, comprising one or more processors and a memory, wherein the memory is used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement any one of the above-mentioned methods for entering big data into a lake.

[0031] According to another aspect of an embodiment of the present invention, a computer program product is provided, including a computer program, which, when executed by a processor, implements the steps of any one of the above-mentioned methods for entering big data into a lake.

[0032] In the present disclosure, by reading the business table operation records and the daily cutting table operation records of N data sources, the actual daily cutting time in the daily cutting table operation records is obtained. Then, when it is determined that the actual daily cutting time of all N data sources has arrived, the consumption process entering the lake is connected, and the consumption process entering the lake is checked. Subsequently, when the business table operation records of each data source have been consumed before the corresponding actual daily cutting time, a T-day partition closure notification is sent to the consumption process entering the lake, and the business table operation records and the daily cutting table operation records are closed to the T-day partition and loaded into the data lake. Finally, the batch processing of entering the lake on T day is completed, and the delayed data is written to the T+1 day partition in the daily partition group. In the present disclosure, by reading the actual daily cutting time in the daily cutting table and judging the actual daily cutting time, the consumption process will be checked through the lake entry consumption process only when the actual daily cutting time of all N data sources is reached. When the business table operation records of each data source have been consumed before the corresponding actual daily cutting time, the T-day lake entry batch processing is completed, thereby comprehensively improving the fault tolerance capability and greatly improving the data quality, thereby solving the technical problem of poor fault tolerance when entering big data into the lake in related technologies.

[0033] This disclosure can significantly improve the recoverability of data entering the lake by improving data storage and disaster recovery cycles, and significantly reduce data error correction time, further improving timeliness and data consistency. This application can comprehensively improve the fault tolerance, verification and recovery capabilities of massive big data replication into the lake, and combined with real-time data warehouse technology, ensure the rapid entry of massive data into the lake while ensuring data consistency. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0035] Figure 1 This is a flowchart of an optional method for entering big data into a lake according to an embodiment of the present invention;

[0036] Figure 2 is a schematic diagram of an optional big data entry system according to an embodiment of the present invention;

[0037] Figure 3 is a schematic diagram of an optional process for entering big data into a lake according to an embodiment of the present invention;

[0038] Figure 4 is a schematic diagram of an optional big data entry device according to an embodiment of the present invention;

[0039] Figure 5 This is a hardware structure block diagram of an electronic device (or mobile device) according to an optional method for entering big data into a lake according to an embodiment of the present invention. DETAILED DESCRIPTION

[0040] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0041] It should be noted that the terms "first", "second", etc. in the description and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0042] To facilitate those skilled in the art to understand the present invention, some of the terms or nouns involved in the embodiments of the present invention are explained below:

[0043] A data lake is a big data storage and processing architecture designed to store and process large amounts of diverse data, including structured, semi-structured, and unstructured data.

[0044] Message middleware is a subsystem in a distributed system that focuses on sending and receiving data, and integrates other subsystems in the distributed system using an efficient and reliable asynchronous messaging mechanism.

[0045] Delayed data refers to data on day T that should have arrived normally before the data partition on day T is closed, but due to communication abnormalities, it did not arrive normally before the data partition on day T is closed. After the partition is closed and communication is restored to normal, the data on day T arrives with a delay.

[0046] T-day partition closure notification: Before the system is switched, the T-day data partition needs to be closed. The last data record entering the T-day data partition will serve as the closing point of the T-day data partition. After the closing point, the T-day data arriving due to communication abnormalities and delays will not be able to enter the T-day data partition, but can enter the T+1 day data partition.

[0047] It should be noted that the method for entering big data into the lake and its system and device in the present disclosure can be used in the field of big data technology while ensuring the timeliness of rapid entry of massive data into the lake and the quality of the big data entering the lake. It can also be used in any field other than the field of big data technology while ensuring the timeliness of rapid entry of massive data into the lake and the quality of the big data entering the lake. The present disclosure does not limit the application field of the method for entering big data into the lake and its system and device.

[0048] It should be noted that the information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) collected by this disclosure are information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of relevant data comply with the relevant laws, regulations and standards of the relevant regions, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation portals for users to choose to authorize or refuse. For example, an interface is set up between this system and relevant users or institutions. Before obtaining relevant information, it is necessary to send an acquisition request to the aforementioned user or institution through the interface, and obtain relevant information after receiving the consent information fed back by the aforementioned user or institution.

[0049] It should be noted that in this disclosure, when collecting and analyzing customer information, the corresponding operation entrance is provided for users to choose to agree or reject the automated decision-making results; if the user chooses to reject, the expert decision-making process will be entered.

[0050] The following embodiments of the present invention can be applied to various systems / applications / devices for entering the lake for big data. The present invention obtains the actual daily cut time by reading the business table operation records and the daily cut table operation records in the data source, and checks the consumption process by connecting to the consumption process entering the lake. When the inspection result indicates that the business table operation records of all data sources are consumed before the corresponding actual daily cut time, the partition of day T is closed by sending a partition closure notification on day T, and the batch processing of entering the lake on day T and the writing of delayed data of day T to the partition of day T+1 are completed, thereby comprehensively improving the fault tolerance, verification and recovery capabilities of copying massive big data into the lake, ensuring the rapid entry of massive data into the lake while ensuring the consistency of data.

[0051] The present invention will be described in detail below with reference to various embodiments.

[0052] Example 1

[0053] According to an embodiment of the present invention, an embodiment of a method for entering big data into a lake is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0054] Figure 1 This is a flow chart of an optional method for entering big data into a lake according to an embodiment of the present invention. Figure 1 As shown, the method includes the following steps:

[0055] Step S101 , reading the business table operation records and the day-cutting table operation records of N data sources, and extracting the actual day-cutting time from the day-cutting table operation records, wherein N is a positive integer greater than or equal to 1.

[0056] Step S102, check whether the actual daily cutting time of all N data sources is in sync, and when the actual daily cutting time of all N data sources is in sync, connect the consumption process into the lake, and let the consumption process into the lake check the consumption process, where the consumption process refers to the process of converting business table operation records into data records.

[0057] In step S103, when the business table operation records of each data source have been consumed before the corresponding actual daily cutting time, a T-day partition closure notification is sent to the consumption process entering the lake. The consumption process entering the lake responds to the T-day partition closure notification, closes the business table operation records and the daily cutting table operation records to the T-day partition, and loads them into the data lake.

[0058] Step S104, when it is determined that the business table operation records and the daily table operation records have been closed to the T-day partition, complete the T-day entry batch processing and write the delayed data into the T+1 day partition in the daily partition group, where the delayed data is the T-day data that arrives after the communication is restored to normal after the T-day partition is closed.

[0059] The steps provided by the above-mentioned big data entry method can obtain the actual day-cut time in the day-cut table operation record by reading the business table operation records and day-cut table operation records of N data sources. Then, after confirming that the actual day-cut time of all N data sources has arrived, the entry-to-lake consumption process is connected, and the entry-to-lake consumption process checks the consumption process. Subsequently, when the business table operation records of each data source have been consumed before the corresponding actual day-cut time, a T-day partition closure notification is sent to the entry-to-lake consumption process, and the business table operation records and day-cut table operation records are closed to the T-day partition and loaded into the data lake. Finally, the T-day entry-to-lake batch processing is completed, and the delayed data is written to the T+1 day partition in the day partition group. In this application, the actual daily cutting time in the daily cutting table is read to judge the actual daily cutting time. When the actual daily cutting time of all N data sources is reached, the consumption process will be checked through the lake entry consumption process. When the business table operation records of each data source have been consumed before the corresponding actual daily cutting time, the T-day lake entry batch processing is completed, thereby comprehensively improving the fault tolerance capability and greatly improving the data quality, thereby solving the technical problem of poor fault tolerance when entering big data into the lake in related technologies.

[0060] The embodiments of the present invention can be applied to various data update systems and big data entry systems of financial institutions, and can ensure data consistency while ensuring that massive data is quickly entered into the lake.

[0061] It should be noted that the "day-switch" mentioned in this example refers to the process by which the financial system switches from the current business day to the next, typically occurring during the nighttime trading downturn. This process involves updating the system date, creating and activating a new transaction log, closing the current transaction log, and sending a day-switch response message to the exchange center. After the day-switch begins, the exchange center will enter a dual-log operation state until the network day-switch is complete.

[0062] The embodiment of the present invention is described in detail below in conjunction with the above steps.

[0063] Step S101 , reading the business table operation records and the day-cutting table operation records of N data sources, and extracting the actual day-cutting time from the day-cutting table operation records, wherein N is a positive integer greater than or equal to 1.

[0064] The data source mentioned in this embodiment may refer to a data source that stores the basic unit of business data and daily data for various business applications of a financial institution. Each business application may correspond to a data source, and the types of business applications include, but are not limited to, financial management applications, credit applications, deposit applications, insurance applications, and credit card applications. Each business application generates a large amount of business data and daily data every day, and the data source is used to store the basic unit of business data and daily data of the business application.

[0065] It should be noted that the cluster formed by combining all business applications of each financial institution is a business application cluster, and the business applications in the business application cluster can be continuously updated, deleted, and added.

[0066] Optionally, step S101 in this embodiment includes: reading the log files of the daily cutting table and the business table in each data source in the business application cluster to obtain the daily cutting table operation records and the business table operation records, wherein the business application cluster contains N data sources, each data source is used to store the business data of the business application and the basic unit of daily cutting data, and each data source contains at least one daily cutting table and a business table, the daily cutting table stores the batch date, actual daily cutting time and daily cutting backup time of each data source, the daily cutting backup time is later than the actual daily cutting time, and the business table stores the business data; the business table operation records and the daily cutting table operation records are respectively spliced ​​into messages of a specified format to obtain an operation record message, and the operation record message is sent to the storage unit of the message middleware.

[0067] It should be noted that each data source mentioned in this embodiment includes at least one daily cut table and a business table. The daily cut table is used to store the batch date, actual daily cut time, and daily cut backup time for each data source. Business applications need to add a daily cut table to each data source in the business application cluster and update the batch date and daily cut time of the current data source when a daily cut occurs. The business table is used to store the business data of each business application.

[0068] Each daily table and business table may be modified by data generated by business applications at any time point every day, resulting in new business table operation records and new daily table operation records.

[0069] In an embodiment of the present invention, business applications need to add a daily cut table to each data source in the business application cluster. The actual daily cut time is the daily cut time updated by the current data source when a daily cut occurs. This is used when closing daily partitions in normal scenarios, for example, 10 p.m. or 11 p.m.; the daily cut fallback time is the latest daily cut time for each data source. This is used when closing daily partitions in abnormal scenarios. That is, when the actual daily cut time corresponding to any of the N data sources cannot be obtained, the daily cut fallback time is used to close the daily partition, for example, 11:30 p.m. or 11:50 p.m. Each data source has one and only one actual daily cut time and one daily cut fallback time. The daily cut fallback time is generally later than the actual daily cut time.

[0070] In this embodiment, the business table operation records and the daily table operation records are respectively spliced ​​into messages of a specified format to obtain an operation record message, and the operation record message is sent to the storage unit of the message middleware. This may refer to first reading and parsing the log file of the daily table in the data source to obtain the daily table operation record, splicing the daily table operation record into a message recognizable by the message middleware (i.e., splicing into a message of a specified format, which can be selected as needed, such as the JSON format), and sending it to the storage unit in the message middleware, in which the corresponding storage daily table can be deployed. Secondly, reading and parsing the log file of the business table in the data source to obtain the business table operation record, splicing the operation record into a message recognizable by the message middleware, and sending it to the storage unit in the message middleware, in which the corresponding storage business table can be deployed.

[0071] Optionally, the steps of reading the business table operation records and the day-cutting table operation records of N data sources and extracting the actual day-cutting time from the day-cutting table operation records also include: obtaining the day-cutting table operation record message from the storage unit, and extracting the actual day-cutting time from the day-cutting table operation record message; and obtaining the day-cutting backup time from the day-cutting table of the data source.

[0072] By obtaining the actual day-cut time from the storage unit in the message middleware and the day-cut backup time from the day-cut table of the data source, preparation is made for subsequent day-cut table operation record inspection and day-cut time inspection to verify data consistency and time consistency.

[0073] The method for importing big data into the data lake provided in this embodiment, after sending the operation record message to the storage unit of the message middleware, further includes: connecting to the consumption process for importing into the lake to consume the business table operation record from the storage unit of the message middleware in real time. The real-time consumption refers to writing the business date in the business table operation record into the T-day partition of the formal table in the data lake in real time. The formal table has been pre-sharded by date to obtain a day partition corresponding to each date. All day partitions are combined into a day partition group. The data lake stores and processes the business table operation record according to the day partition.

[0074] It's important to note that after consuming business data from the business table's operation records in real time, it must be written to the daily partition of the official data lake table in real time based on the business date. Normally, data from day T is written to the day T partition. If delayed data is still received after receiving a notification that the day T partition has closed, it is automatically written to the day T+1 partition. This ensures eventual consistency of data entering the lake and prevents downstream data loss.

[0075] It should also be noted that in this embodiment, a table with a large storage space is pre-configured in the data lake and defined as a formal table. Sharding is then performed on the formal table in the data lake. Sharding is performed by date, with each day corresponding to a daily partition. All daily partitions are combined into a daily partition group. All daily table slicing and business table operation records generated each day are stored in the daily partition corresponding to the date, and then the daily partition is closed. Here, the daily partition is the basic unit of data storage in the data lake.

[0076] After obtaining the actual day-cut time from the day-cut table operation record of each data source, it is determined whether the actual day-cut times of all data sources are all reached.

[0077] Step S102, check whether the actual daily cutting time of all N data sources is in sync, and when the actual daily cutting time of all N data sources is in sync, connect the consumption process into the lake, and let the consumption process into the lake check the consumption process, where the consumption process refers to the process of converting business table operation records into data records.

[0078] In this embodiment, after checking whether the actual daily cutting time of all N data sources has arrived, it also includes: if the actual daily cutting time of any data source has not arrived, judging whether it is necessary to send a T-day batch processing request to the operation and maintenance end; if there is no need to send a T-day batch processing request to the operation and maintenance end, connecting to the lake entry consumption process, and the lake entry consumption process checks the consumption process of the data source; judging whether the business table operation records before the daily cutting bottom-line time have been consumed; if the business table operation records of each data source have been consumed before the corresponding daily cutting bottom-line time, sending a T-day partition closure notification to the lake entry consumption process.

[0079] In an embodiment of the present invention, each data source has one and only one actual daily cut-off time. When checking the daily cut-off table operation records, the actual daily cut-off time of the data source may not be obtained on time. In this case, the actual daily cut-off time of the N data sources will be missing one or more, that is, the actual daily cut-off time of the N data sources has not been met. In the case that the actual daily cut-off time of the N data sources has not been met, the embodiment of the present invention will determine whether it is necessary to send a T-day batch processing request to the operation and maintenance end. If necessary, the processing end sends a T-day batch processing request to the operation and maintenance end. The operation and maintenance end responds to the request and starts emergency T-day batch processing. The processing end no longer performs automatic daily cut-off. If it is determined that it is not necessary, the processing end will not send a T-day batch processing request to the operation and maintenance end. The processing end connects to the lake-entering consumption process, and the lake-entering consumption process checks the consumption process of the data source. At this time, it mainly checks whether the business table operation records of each data source have been consumed before the corresponding daily cut-off time. It is not necessary to continue to use the actual daily cut-off time to make a judgment. If the business table operation records of each data source have been consumed before the corresponding daily cut-off time, a T-day partition closure notification can be sent to the lake-entering consumption process.

[0080] The above-mentioned step of checking the consumption process by the consumption process entering the lake includes: for each data source, the consumption process entering the lake determines whether the business table operation records in the distributed processing platform have been consumed before the actual daily cut time.

[0081] In addition, after the consumption process is checked by the consumption process entering the lake, if the business table operation record of any data source is not consumed before the corresponding actual daily cut time, the consumption process will be continuously checked by the consumption process entering the lake.

[0082] In this embodiment, the business table operation records read from the data source are consumed in real time by the in-lake consumption process module. This process converts the business table operation records into data records. The in-lake consumption process consumes the business table operation records in real time and checks whether the business table operation records of each data source have been consumed before the actual daily cut time corresponding to each data source. If so, a T-day partition closure notification is sent. If not, real-time consumption continues until consumption is complete.

[0083] Optionally, the step of determining whether the business table operation records in the distributed processing platform have been consumed before the actual day cut time includes: if there are business table operation records in the distributed processing platform, extracting the processing time corresponding to the business data in the business table operation record, and if the processing time exceeds the predetermined duration of the actual day cut time, and the partition of day T has not been updated for more than M minutes, then determining that the business data of day T has been consumed, where M is a positive integer greater than or equal to 1; if there are no business table operation records in the distributed processing platform, the current time of the system exceeds the predetermined duration of the actual day cut time, and the partition of day T has not been updated for more than M minutes, then determining that the business data of day T has been consumed.

[0084] In this embodiment, there are two criteria for judging whether the business table operation records of each data source have been consumed before the actual daily cut-off time corresponding to each data source. First, if there are business table operation records in the distributed processing platform, extract the processing time corresponding to the business data in the business table operation record. If the processing time exceeds the predetermined duration of the actual daily cut-off time, and the partition of day T has not been updated for more than M minutes, it is determined that the business data of day T has been consumed; second, if there are no business table operation records in the distributed processing platform, the current time of the system exceeds the predetermined duration of the actual daily cut-off time, and the partition of day T has not been updated for more than M minutes, it is determined that the business data of day T has been consumed. Among them, the current time of the system is the real time of the environment in which the consumers entering the lake are located; the predetermined duration is "the actual daily cut-off time (or daily cut-off backup time) of each data source + N" minutes, M and N are positive integers greater than or equal to 1, M depends on the write efficiency of the consumers entering the lake, and N depends on the speed difference when reading the business table operation records from the data source. When the consumption process does not meet any of the above two points, it is determined that the business table operation records of N data sources have not been consumed before the corresponding actual daily cut time. At this time, the lake consumption process will continuously check the consumption process until the business table operation records of all data sources are consumed before the corresponding actual daily cut time.

[0085] Step S103: When the business table operation records of each data source have been consumed before the corresponding actual daily cut time, a T-day partition closing notification is sent to the consumption process entering the lake.

[0086] Among them, the consumption process entering the lake responds to the T-day partition closure notification, closes the business table operation records and the daily table operation records to the T-day partition, and loads them into the data lake.

[0087] In this embodiment, a data lake is a big data storage and processing architecture designed to store and process large amounts of diverse data. A storage space is allocated within the data lake for formal tables or for direct partitioning and grouping. These formal tables or partition groups are divided into multiple daily partitions, one for each day, each storing the data for that day. Consumed business table operation records are stored in the partition on day T of the data lake. Once a partition on day T is closed, no further modifications, adjustments, deletions, or additions can be made to it.

[0088] Step S104, when it is determined that the business table operation records and the daily table operation records have been closed to the T-day partition, complete the T-day entry batch processing and write the delayed data into the T+1 day partition in the daily partition group, where the delayed data is the T-day data that arrives after the communication is restored to normal after the T-day partition is closed.

[0089] In an embodiment of the present invention, the data on day T that should have arrived normally before the data partition on day T is closed does not arrive normally before the data partition on day T is closed due to communication anomalies. After the partition is closed and communication is restored to normal, the data on day T that arrives late is the delayed data. Since the delayed data on day T belongs to the data on day T, under normal circumstances, the delayed data should be included in the partition on day T. However, due to abnormal circumstances, the delayed data cannot be enclosed in the partition on day T before the actual day cut time. In order to ensure the timeliness and consistency of the data, the delayed data needs to be enclosed in the partition on the next day. Subsequent daily business data and day cut data are operated in the above manner.

[0090] Through the above embodiments, the fault tolerance, verification and recovery capabilities of massive big data replication into the lake can be comprehensively improved, and combined with real-time data warehouse technology, it can ensure that massive data is quickly entered into the lake while ensuring data consistency.

[0091] Figure 2 is a schematic diagram of another optional big data entry system according to an embodiment of the present invention, such as Figure 2 As shown, the system includes the following modules:

[0092] The business application cluster, corresponding to the daily data switching and business data storage stages, is responsible for checking data integrity. The business application cluster contains N data sources, each of which contains at least one daily switching table and one business table. The daily switching table stores the batch date, actual daily switching time, and daily switching backup time for each data source, while the business table stores business data.

[0093] The data replication tool, corresponding to the data replication phase, is used to ensure the consistency of the data that has been successfully read and the data that has been successfully sent to the message middleware. The data replication tool includes a daily table replication module and a business table replication module, wherein the daily table replication module is used to read and parse the log file of the daily table in the data source, obtain the operation record of the daily table, and splice the daily table operation record into a message that can be recognized by the message middleware (that is, spliced ​​into a message in a specified format, and the specified format can be selected as needed, such as selecting the JSON format), and send it to the daily table storage unit in the message middleware; the business table replication module is used to read and parse the log file of the business table in the data source, obtain the operation record of the business table, and splice the business table operation record into a message that can be recognized by the message middleware, and send it to the business table storage unit in the message middleware.

[0094] The message middleware, corresponding to the data transmission phase, is responsible for checking data integrity. It contains N daily table storage units and N business table storage units. The daily table storage units are used to store operation log messages related to daily table operations, while the business table storage units are used to store operation log messages related to business table operations. The message middleware is primarily responsible for the storage and transmission of operation log messages.

[0095] The big data service platform, corresponding to the data consumption phase, is responsible for verifying the number of records consumed by the message middleware and the number of data records written to the data lake. The big data service platform includes a daily cut judgment service module and a lake entry consumption process module. The daily cut judgment service module is used to provide daily cut judgment services, including: obtaining the daily cut backup time from the data source, obtaining the daily cut table operation record message from the daily cut table storage unit and extracting the actual daily cut time; checking the obtained daily cut table operation record and waiting for the actual daily cut time records of all data sources to be recorded; connecting to the lake entry consumption process module to check the consumption process and confirm that the business data of all data sources has been consumed before the corresponding actual daily cut time point; sending a T-day partition closure notification, closing the T-day partition, and completing the T-day lake entry batch processing. In abnormal scenarios, the actual daily cut time of the data source cannot be reached, and the daily partition is closed according to the obtained daily cut backup time. The lake entry consumption process module consumes business data in real time and writes the daily partition in the official table of the data lake in real time according to the business date. Under normal circumstances, data on day T is written to the partition on day T. When a partition closure notification is received on day T, if delayed data is still received, it is automatically written to the partition on day T+1 to ensure the eventual consistency of data entering the lake and that downstream data processing is not lost.

[0096] There are two ways to check the consumption process and confirm that the business data of all data sources has been consumed before the corresponding actual daily cut-off time point:

[0097] In the first method, when the distributed processing platform has data, when it consumes data of "daily cutoff time + N minutes" and the partition of day T has not been updated for more than M minutes, it is determined that the consumption of day T is completed.

[0098] The second method is that when there is no data on the distributed processing platform, the system date exceeds the "daily cutoff time + N minutes", and the partition of day T has not been updated for more than M minutes, the consumption of day T is considered to be completed.

[0099] Among them, the above M and N are both positive integers greater than or equal to 1. M depends on the write efficiency of consumers entering the lake, and N depends on the difference in replication speed of the business table replication module for different data sources.

[0100] In addition, the big data entry system mentioned in this embodiment may also include a data lake (or a big data cluster) for storing and processing business table operation records according to daily partitions.

[0101] The data lake mentioned in this embodiment can store and process business data by daily partition. A daily partition group includes multiple daily partitions, one for each day. A daily partition is the unit in which the data lake stores business data.

[0102] This embodiment integrates consumption and lake entry processes through real-time lake entry technology. A daily cut judgment service module, a lake entry consumption process module, a daily cut table replication module, and a business table replication module are set up. The daily cut table is deployed in the source system library through business applications, and the daily cut information is sent to the daily cut judgment service module by using the daily cut table replication method, thereby comprehensively improving the fault tolerance and greatly improving the data quality. In addition, in this embodiment, the traditional but time-consuming and inefficient daily cut judgment step of checking the heartbeat of the data replication tool and the daily cut offset of the application configuration will be abandoned, so as to achieve the goal of shortening time consumption and improving the timeliness of data entry into the lake. At the same time, this embodiment uses the "full-link segmented verification" method to greatly improve the consistency verification capability of the data flow link, effectively ensure the integrity of the data, and greatly improve the data quality of massive big data entering the lake. In addition, this embodiment greatly improves the recoverability of the data entering the lake by improving the data storage and disaster recovery cycle, and significantly reduces the data error correction time, further improving the timeliness and data consistency.

[0103] The above embodiments optimize the logic of big data entry into the lake, comprehensively improving the fault tolerance, verification, and recovery capabilities of massive big data replication into the lake. This not only ensures that massive data can be quickly entered into the lake, but also ensures data consistency.

[0104] It should be noted that this embodiment provides a process for entering a big data lake to ensure both timeliness and quality of massive big data entering the lake and improve fault tolerance. Figure 3 is a schematic diagram of another optional process of entering the big data lake according to an embodiment of the present invention, such as Figure 3As shown, the method includes the following steps:

[0105] Step S301: The business application stores business data and daily data in a data source.

[0106] Step S302: The daily table replication module reads the daily table operation record from the data source and sends it to the daily table storage unit of the message middleware.

[0107] Step S303 : the daily cut judgment service module obtains the daily cut backup time from the data source, obtains the daily cut table operation record from the daily cut storage unit, and extracts the actual daily cut time.

[0108] Step S304: the day-cut determination service module checks the acquired day-cut table operation records and waits for the actual day-cut time records of all data sources to be collected.

[0109] Step S305: The operation and maintenance end performs emergency T-day batch processing.

[0110] Step S306: The daily cut judgment service module connects to the consumption process module to check the consumption process.

[0111] Step S307: the daily cut judgment service module connects to the consumption process module to check the consumption process.

[0112] Step S308: The daily cut judgment service module sends a T-day partition closing notification.

[0113] Step S309: The business table replication module reads the business table operation record from the data source and sends it to the business table storage unit of the message middleware.

[0114] Step S310: The consumption process module consumes business data from the business storage unit of the message middleware in real time.

[0115] Step S311: the consumption process module receives a notification of closure of the T-day partition in the day partition group.

[0116] Step S312: The T-day partition in the day partition group is closed, batch processing for T-day is completed, and delayed data is written to the T+1-day partition in the day partition group.

[0117] The steps provided by the above-mentioned big data entry method are to read the business table operation records and daily cutting table operation records of N data sources, obtain the actual daily cutting time in the daily cutting table operation records, and then connect to the lake entry consumption process after confirming that the actual daily cutting time of all N data sources has arrived. The lake entry consumption process checks the consumption process. Subsequently, when the business table operation records of each data source have been consumed before the corresponding actual daily cutting time, a T-day partition closure notification is sent to the lake entry consumption process, and the business table operation records and daily cutting table operation records are closed to the T-day partition and loaded into the data lake. Finally, the T-day lake entry batch processing is completed, and the delayed data is written to the T+1 day partition in the daily partition group. In this embodiment, the actual daily cut time is read from the daily cut table to judge the actual daily cut time. When the actual daily cut time of all N data sources is reached, the consumption process will be checked through the consumption process entering the lake. When the business table operation records of each data source have been consumed before the corresponding actual daily cut time, the batch processing of entering the lake on day T is completed, thereby comprehensively improving the fault tolerance and greatly improving the data quality, thereby solving the technical problem of poor fault tolerance when entering big data into the lake in related technologies. In addition, in this embodiment, the data verification capability can also be improved, and the "full-link segmented verification" method is used to greatly improve the consistency verification capability of the data flow link, effectively ensure data integrity, and greatly improve the data quality of massive big data entering the lake. At the same time, by improving the data storage and disaster recovery cycle, the recoverability of the data entering the lake is greatly improved, and the data error correction time is significantly reduced, further improving the timeliness and data consistency.

[0118] The following describes it in detail with reference to another embodiment.

[0119] Example 2

[0120] A big data entry device provided in this embodiment includes multiple implementation units, each implementation unit corresponding to each implementation step in the above-mentioned embodiment one.

[0121] Figure 4 is a schematic diagram of an optional big data entry device according to an embodiment of the present invention, such as Figure 4 As shown, the big data entry device may include: a record reading unit 40, a consumption checking unit 41, a closure notification unit 42, and a closure processing unit 43.

[0122] The record reading unit 40 is used to read the business table operation records and the day-cutting table operation records of N data sources, and extract the actual day-cutting time from the day-cutting table operation records, where N is a positive integer greater than or equal to 1.

[0123] The consumption check unit 41 is used to check whether the actual daily cutting time of all N data sources has arrived. When the actual daily cutting time of all N data sources has arrived, the consumption process entering the lake is connected, and the consumption process entering the lake is used to check the consumption process, where the consumption process refers to the process of converting business table operation records into data records.

[0124] The closure notification unit 42 is used to send a T-day partition closure notification to the lake-entering consumption process when the business table operation records of each data source have been consumed before the corresponding actual daily cutting time. The lake-entering consumption process responds to the T-day partition closure notification, closes the business table operation records and the daily cutting table operation records to the T-day partition, and loads them into the data lake.

[0125] The closed processing unit 43 is used to complete the batch processing of the T-day entry into the lake when it is determined that the business table operation records and the daily table operation records have been closed to the T-day partition, and write the delayed data into the T+1 day partition in the daily partition group, wherein the delayed data is the T-day data that arrives after the communication is restored to normal after the T-day partition is closed.

[0126] The above-mentioned big data entry device can read the business table operation records and daily cutting table operation records of N data sources through the record reading unit 40, obtain the actual daily cutting time in the daily cutting table operation record, and then check whether the actual daily cutting time of all N data sources is reached through the consumption checking unit 41. When it is determined that the actual daily cutting time of all N data sources is reached, the entry-to-lake consumption process is connected, and the entry-to-lake consumption process checks the consumption process. Subsequently, when the business table operation records of each data source have been consumed before the corresponding actual daily cutting time, the closing notification unit 42 sends a T-day partition closing notification to the entry-to-lake consumption process, closes the business table operation records and the daily cutting table operation records to the T-day partition, and loads them into the data lake. Finally, when it is determined that the business table operation records and the daily cutting table operation records have been closed to the T-day partition, the closing processing unit 43 completes the T-day entry batch processing and writes the delayed data into the T+1 day partition in the day partition group. In this application, the actual daily cutting time in the daily cutting table is read to judge the actual daily cutting time. When the actual daily cutting time of all N data sources is reached, the consumption process will be checked through the lake entry consumption process. When the business table operation records of each data source have been consumed before the corresponding actual daily cutting time, the T-day lake entry batch processing is completed, thereby comprehensively improving the fault tolerance capability and greatly improving the data quality, thereby solving the technical problem of poor fault tolerance when entering big data into the lake in related technologies.

[0127] Optionally, the record reading unit includes: a first reading module, used to read the log files of the daily cutting table and the business table in each data source in the business application cluster, and obtain the daily cutting table operation records and the business table operation records, wherein the business application cluster contains N data sources, each data source is used to store the business data of the business application and the basic unit of daily cutting data, and each data source contains at least one daily cutting table and a business table, the daily cutting table stores the batch date, actual daily cutting time and daily cutting backup time of each data source, the daily cutting backup time is later than the actual daily cutting time, and the business table stores the business data; a splicing module, used to splice the business table operation records and the daily cutting table operation records into messages of a specified format, obtain operation record messages, and send the operation record messages to the storage unit of the message middleware.

[0128] Optionally, the record reading unit also includes: a second reading module, used to obtain the day cutting table operation record message from the storage unit, and extract the actual day cutting time from the day cutting table operation record message; an acquisition module, which obtains the day cutting backup time from the day cutting table of the data source.

[0129] Optionally, the record reading unit also includes: a consumption module, which is used to connect to the consumption process entering the lake and consume the business table operation records from the storage unit of the message middleware in real time. Here, real-time consumption means writing the business date in the business table operation record into the T-day partition of the formal table in the data lake in real time. The formal table has been pre-sharded by date to obtain the day partition corresponding to each date. All day partitions are combined into day partition groups. The data lake stores and processes the business table operation records according to the day partitions.

[0130] Optionally, the consumption inspection unit also includes: a request module, which is used to determine whether it is necessary to send a T-day batch processing request to the operation and maintenance end when the actual daily cutting time of any data source has not arrived; an inspection module, which is used to connect to the lake-entering consumption process without sending the T-day batch processing to the operation and maintenance end, and the lake-entering consumption process checks the consumption process of the data source; a first judgment module, which is used to determine whether the business table operation records before the daily cutting backup time have been consumed; a sending module, which sends a T-day partition closure notification to the lake-entering consumption process when the business table operation records of each data source have been consumed before the corresponding daily cutting backup time.

[0131] Optionally, the inspection unit further includes: a second judgment module, which is used to determine, for each data source, by the consumption process entering the lake, whether the business table operation records in the distributed processing platform have been consumed before the actual daily cut time.

[0132] Optionally, the second judgment unit also includes: a first judgment module, which is used to extract the processing time corresponding to the business data in the business table operation record if there is a business table operation record in the distributed processing platform, and if the processing time exceeds the predetermined duration of the actual daily cut time, and the T-day partition has not been updated for more than M minutes, it is determined that the business data of T day has been consumed, where M is a positive integer greater than or equal to 1; a second judgment module, which is used to determine that the business data of T day has been consumed if there is no business table operation record in the distributed processing platform, the current time of the system exceeds the predetermined duration of the actual daily cut time, and the T-day partition has not been updated for more than M minutes.

[0133] Optionally, the consumption inspection unit also includes: after the consumption process is inspected by the consumption process entering the lake, if the business table operation record of any data source is not consumed before the corresponding actual daily cut time, the consumption process entering the lake will continue to inspect the consumption process.

[0134] The above-mentioned big data entry device can also include a processor and a memory. The above-mentioned record reading unit 40, consumption inspection unit 41, closure notification unit 42, closure processing unit 43, etc. are all stored in the memory as program units, and the processor executes the above-mentioned program units stored in the memory to realize the corresponding functions.

[0135] The processor includes a kernel that retrieves the corresponding program unit from memory. One or more kernels can be configured. By adjusting kernel parameters, business table operation records and daily table switching operation records are confined to the T-day partition, completing batch processing of data entering the lake on T day and writing delayed data to the T+1 day partition within the daily partition group.

[0136] The above-mentioned memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0137] According to another aspect of an embodiment of the present invention, a computer-readable storage medium is further provided. The computer-readable storage medium includes a stored computer program, wherein when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute the method for entering big data into a lake according to any one of the above-mentioned embodiments.

[0138] According to another aspect of an embodiment of the present invention, an electronic device is also provided, including one or more processors and a memory, wherein the memory is used to store one or more programs, wherein when the one or more programs are executed by one or more processors, the one or more processors implement the method for entering big data into a lake of any one of the above-mentioned embodiments.

[0139] Figure 5 This is a hardware structure diagram of an electronic device (or mobile device) according to an optional method for entering big data into a lake according to an embodiment of the present invention. Figure 5 As shown, the electronic device may include one or more ( Figure 5 (As shown in the figure, 502a, 502b, ..., 502n are used) processor 502 (processor 502 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), memory 504 for storing data. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the I / O interface), a network interface, a keyboard, a power supply and / or a camera. It will be understood by those skilled in the art that Figure 5 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 5 More or fewer components than shown, or with Figure 5 Different configurations shown.

[0140] The present application also provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of the method for entering big data into a lake in each embodiment of the present application.

[0141] The present application also provides a computer program product, including a non-volatile computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the steps of the method for entering big data into a lake in each embodiment of the present application.

[0142] The serial numbers of the above embodiments of the present invention are for description only and do not represent the advantages or disadvantages of the embodiments.

[0143] In the above embodiments of the present invention, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0144] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only exemplary. For example, the division of units can be a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0145] Units described as separate components may or may not be physically separate, and components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple units. Some or all of the units may be selected to achieve the purpose of the present embodiment according to actual needs.

[0146] In addition, the functional units in various embodiments of the present invention may be integrated into a single consumption inspection unit, each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0147] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server or network device, etc.) to execute all or part of the steps of the various embodiments of the present invention. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk, and other media that can store program codes.

[0148] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A method for entering big data into a lake, characterized in that: include: Reading the business table operation records and the day-cutting table operation records of N data sources, and extracting the actual day-cutting time from the day-cutting table operation records, where N is a positive integer greater than or equal to 1; Check whether the actual daily cut-off times of all the N data sources are in sync. If the actual daily cut-off times of all the N data sources are in sync, connect the lake-entering consumption process, and have the lake-entering consumption process check the consumption process, where the consumption process refers to the process of converting the business table operation records into data records; If the business table operation records of each data source have been consumed before the corresponding actual daily cut time, a T-day partition closure notification is sent to the lake-entering consumption process. The lake-entering consumption process responds to the T-day partition closure notification, closes the business table operation records and the daily cut table operation records to the T-day partition, and loads them into the data lake; When it is determined that the business table operation records and the daily table operation records have been closed to the T-day partition, the T-day entry batch processing is completed, and the delayed data is written to the T+1 day partition in the daily partition group, wherein the delayed data is the T-day data that arrives after the communication is restored to normal after the T-day partition is closed.

2. The method for entering big data into a lake according to claim 1, characterized in that: The steps of reading the business table operation records and the day-cutting table operation records of N data sources and extracting the actual day-cutting time from the day-cutting table operation records include: Read the log files of the daily cutting table and the business table in each of the data sources in the business application cluster to obtain the daily cutting table operation records and the business table operation records, wherein the business application cluster includes the N data sources, each of the data sources is used to store the business data of the business application and the basic unit of daily cutting data, each of the data sources includes at least one daily cutting table and the business table, the daily cutting table stores the batch date, actual daily cutting time and daily cutting backup time of each data source, the daily cutting backup time is later than the actual daily cutting time, and the business table stores business data; The business table operation record and the daily table operation record are respectively spliced ​​into messages of a specified format to obtain an operation record message, and the operation record message is sent to a storage unit of a message middleware.

3. The method for entering big data into a lake according to claim 2, characterized in that: The steps of reading the business table operation records and the day-cutting table operation records of N data sources and extracting the actual day-cutting time from the day-cutting table operation records include: Acquire the day-cutting table operation record message from the storage unit, and extract the actual day-cutting time from the day-cutting table operation record message; The daily cut-off backup time is obtained from the daily cut-off table of the data source.

4. The method for entering big data into a lake according to claim 2, characterized in that: After sending the operation record message to the storage unit of the message middleware, the method further includes: Connect to the lake consumption process and consume the business table operation records from the storage unit of the message middleware in real time, wherein the real-time consumption means writing the business date in the business table operation record into the T-day partition of the formal table in the data lake in real time. The formal table has been pre-sharded by date to obtain the day partition corresponding to each date. All day partitions are combined into the day partition group. The data lake stores and processes the business table operation records according to the day partition.

5. The method for entering big data into a lake according to claim 1, characterized in that: After checking whether the actual daily switching times of all the N data sources have arrived, the following steps are further included: If the actual daily cut-off time of any of the data sources has not arrived, determine whether it is necessary to send a T-day batch processing request to the operation and maintenance end; Without sending a T-day batch processing request to the operation and maintenance end, connect to the lake-entering consumption process, and let the lake-entering consumption process check the consumption process of the data source; Determine whether the business table operation records before the daily cut-off time have been consumed; When the business table operation records of each data source have been consumed before the corresponding daily cut-off time, a T-day partition closure notification is sent to the lake-entering consumption process.

6. The method for entering big data into a lake according to claim 1, characterized in that: The step of checking the consumption process by the lake consumption process includes: For each of the data sources, the lake consumption process determines whether the business table operation records in the distributed processing platform have been consumed before the actual daily cut time.

7. The method for entering big data into a lake according to claim 6, characterized in that: The step of determining whether the business table operation records in the distributed processing platform have been consumed before the actual day cut time includes: If the business table operation record exists in the distributed processing platform, extract the processing time corresponding to the business data in the business table operation record. If the processing time exceeds the predetermined duration of the actual daily cut time, and the partition of day T has not been updated for more than M minutes, then it is determined that the business data of day T has been consumed, where M is a positive integer greater than or equal to 1; If the business table operation record does not exist in the distributed processing platform, the current system time exceeds the predetermined duration of the actual daily cut time, and the T-day partition has not been updated for more than M minutes, it is determined that the business data of T-day has been consumed.

8. The method for entering big data into a lake according to claim 1, characterized in that: After the consumption process is checked by the consumption process entering the lake, it also includes: In the case that the business table operation record of any of the data sources has not been consumed before the corresponding actual daily cut time, the consumption process is continuously checked by the consumption process entering the lake.

9. A big data lake entry system, characterized by: include: A business application cluster includes N data sources, each of which is used to store business data and daily cut data of business applications. Each data source includes at least one daily cut table and a business table. The daily cut table stores the batch date, actual daily cut time, and daily cut fallback time of each data source, where the daily cut fallback time is later than the actual daily cut time. The business table stores business data, where N is a positive integer greater than or equal to 1. A data replication tool is configured to read the business table operation records and the day-cutting table operation records of the N data sources, extract the actual day-cutting time from the day-cutting table operation records, and send the business table operation records and the day-cutting table operation records to a message middleware; The message middleware includes N daily table storage units and N business table storage units. The daily table storage units are used to store operation record messages of the daily table, and the business table storage units are used to store operation record messages of the business table. The message middleware is mainly responsible for the storage and transmission of the operation record messages. A big data service platform is used for connecting to a consumption process entering the lake when the business table operation records and the daily cutting table operation records pass the integrity check and the actual daily cutting times of all the N data sources are all arrived, and the consumption process is checked by the consumption process entering the lake, wherein the consumption process refers to the process of converting the business table operation records into data records; when the business table operation records of each data source have been consumed before the corresponding actual daily cutting time, sending a T-day partition closure notification to the consumption process entering the lake, wherein the consumption process entering the lake responds to the T-day partition closure notification, closes the business table operation records and the daily cutting table operation records to the T-day partition, and loads them into the data lake; when it is determined that the business table operation records and the daily cutting table operation records have been closed to the T-day partition, completing the batch processing of entering the lake on T day, and writing the delayed data to the T+1 day partition in the daily partition group, wherein the delayed data is the T-day data that arrives after the communication is restored to normal after the T-day partition is closed; The data lake is used to store and process the business table operation records according to the daily partitions.

10. A device for entering a lake into big data, characterized in that: include: a record reading unit, configured to read the business table operation records and the day-cutting table operation records of N data sources, and extract the actual day-cutting time from the day-cutting table operation records, where N is a positive integer greater than or equal to 1; a consumption checking unit, configured to check whether the actual daily cut-off times of all the N data sources are coincident, and, if the actual daily cut-off times of all the N data sources are coincident, connect the consumption process entering the lake, and have the consumption process entering the lake check the consumption process, wherein the consumption process refers to the process of converting the operation records of the business table into data records; A closure notification unit is configured to send a T-day partition closure notification to the lake-entering consumption process if all business table operation records of each data source have been consumed before the corresponding actual daily cut time. The lake-entering consumption process responds to the T-day partition closure notification, closes the business table operation records and the daily cut table operation records to the T-day partition, and loads them into the data lake. The closed processing unit is used to complete the batch processing of the T-day entry into the lake and write the delayed data into the T+1 day partition in the day partition group after determining that the business table operation record and the daily cut table operation record have been closed to the T-day partition, wherein the delayed data is the T-day data that arrives after the communication is restored to normal after the T-day partition is closed.

11. A computer-readable storage medium, characterized in that The computer-readable storage medium includes a stored computer program, wherein when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute the method for entering big data into the lake as described in any one of claims 1 to 8.

12. An electronic device, characterized in that: It includes one or more processors and a memory, wherein the memory is used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method for entering big data into a lake as described in any one of claims 1 to 8.

Images