Big data replication into lake counting method and device, electronic equipment, storage medium and program product

Abstract

The application provides a big data replication into a lake counting method and device, electronic equipment, a storage medium and a program product, and relates to the field of big data. The method comprises the following steps: according to a source database, a plurality of messages are written into corresponding topic partitions respectively, wherein for each message, the message comprises corresponding target data in the source database and time information; according to the time when each of the plurality of messages is written into the corresponding topic partition, the plurality of messages are written into each time partition in a back-end storage system respectively; the messages written into each time partition in the back-end storage system are mapped into each time partition in a back-end query system; according to the data size of the messages written into each topic partition and the data size queried from each time partition in the back-end query system, the plurality of messages are subjected to counting processing, and a counting result is obtained. The method provided in the application can verify the integrity and reliability of the data in the data lake.

Description

Technical Field

[0001] This application relates to the field of big data, and in particular to a core method, apparatus, electronic device, storage medium, and program product for replicating big data into a lake. Background Technology

[0002] Big data replication into a data lake refers to the process of copying large amounts of data from the source database and importing it into a data lake.

[0003] In related technologies, a data processing platform is used to replicate big data into a data lake. Specifically, data is written from the source database to the data processing platform, processed by the platform, and then written into the data lake.

[0004] However, the process of using a data processing platform to replicate big data into the data lake cannot guarantee the integrity and reliability of the data in the data lake. Summary of the Invention

[0005] This application provides a data replication method, apparatus, electronic device, storage medium, and program product for data lake replication, in order to solve the technical problem of being unable to ensure the integrity and reliability of data in a data lake.

[0006] Firstly, this application provides a core numbering method for replicating big data into a lake, the method comprising:

[0007] Based on the source database, multiple messages are written to their respective topic partitions. For each message, the message includes the target data in the source database and time information, which indicates the time when the message was written to the corresponding topic partition.

[0008] Based on the time when each message was written to its corresponding topic partition, the messages are written to their respective time partitions in the backend storage system.

[0009] The messages written to each time partition in the backend storage system are mapped to each time partition in the backend query system.

[0010] Based on the data volume corresponding to the messages written to each topic partition and the data volume queried from each time partition in the backend query system, the core count of multiple messages is processed to obtain the core count result.

[0011] Secondly, this application provides a core data replication device for big data into a lake, the device comprising:

[0012] The first processing module is used to write multiple messages into the corresponding topic partitions according to the source database. For each message, the message includes the target data in the source database and time information. The time information is used to indicate the time when the message is written into the corresponding topic partition.

[0013] The second processing module is used to write multiple messages into the time partitions of the backend storage system according to the time when each message is written into its corresponding topic partition.

[0014] The mapping module is used to map messages written to each time partition in the backend storage system to each time partition in the backend query system.

[0015] The core count module is used to perform core count processing on multiple messages based on the data volume corresponding to the messages written to each topic partition and the data volume queried from each time partition in the backend query system, and obtain the core count result.

[0016] Thirdly, this application provides an electronic device, including: a processor, and a memory communicatively connected to the processor;

[0017] The memory stores instructions that the computer executes;

[0018] The processor executes computer execution instructions stored in memory to implement the method described in the first aspect.

[0019] Fourthly, this application provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the aforementioned method for replicating large data into a lake.

[0020] Fifthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the aforementioned method for replicating large data into a lake.

[0021] The big data replication and indexing method provided in this application embodiment involves a server writing multiple messages into corresponding topic partitions based on a source database. Each message includes target data from the source database and time information, indicating the moment the message was written to the corresponding topic partition. By recording the time information, the amount of target data written to the topic partitions at different times can be determined. Then, based on the time each message was written to its corresponding topic partition, the server writes the multiple messages into different time partitions in the backend storage system. The messages written to each time partition in the backend storage system are mapped to different time partitions in the backend query system, thus determining the amount of data retrieved in each time partition in the backend query system. Finally, based on the amount of data written to each topic partition and the amount of data retrieved in each time partition in the backend query system, the multiple messages are indexed to obtain the indexing result. This embodiment of the application, by recording the time each message was written to its corresponding topic partition, can determine the amount of target data written to each topic partition and perform indexing with the amount of data retrieved in each time partition in the backend query system, thereby verifying the integrity and reliability of the data in the data lake. Attached Figure Description

[0022] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0023] Figure 1 This is a schematic diagram of an application scenario provided by an embodiment of this application;

[0024] Figure 2 A flowchart illustrating the core number method for big data replication into a lake provided in this application embodiment;

[0025] Figure 3 This is a schematic diagram illustrating the process of a big data replication method for importing data into a lake, as provided in an embodiment of this application.

[0026] Figure 4 This is a schematic diagram of the message writing time partition provided in an embodiment of this application;

[0027] Figure 5 This is a schematic flowchart of the core count processing provided in an embodiment of this application;

[0028] Figure 6 A schematic diagram of the core data processing device for big data replication into the lake provided in this application embodiment;

[0029] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0030] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0031] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0032] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, storage, use, processing, transmission, provision, disclosure, and application of the relevant data all comply with the relevant laws, regulations, and standards of the relevant countries and regions, have taken necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation access points for users to choose to authorize or refuse.

[0033] It should be noted that the big data replication method, apparatus, electronic device, storage medium and program product provided in this application can be used in the field of big data, or in any field other than big data. The application field of the big data replication method, apparatus, electronic device, storage medium and program product in this application is not limited.

[0034] First, combine Figure 1 This paper introduces one applicable application scenario for this application.

[0035] Figure 1 This is a schematic diagram illustrating an application scenario provided by an embodiment of this application. For example... Figure 1 As shown, it includes a source database 11, a data processing platform 12, and a data lake 13.

[0036] Source database 11 is a database that provides data; there can be one or more source databases 11. Data processing platform 12 is a platform that provides real-time data processing and analysis. Data processing platform 12 can be deployed on a server. The server can collect real-time data from source database 11 through data processing platform 12 and store it in a message queue, allowing the server to asynchronously read data from data processing platform 12. Data lake 13 is the destination for big data replication, and data lake 13 can provide data storage, query, and analysis functions. Data processing platform 12 collects data from source database 11 in real time and then writes the data into data lake 13.

[0037] In related technologies, data from the source database is copied to a data processing platform using a replication tool. The server reads the data from the data processing platform, processes it, and then writes the data to the corresponding time partition in the data lake according to the time it was written to the data processing platform. This allows the data lake to be used for data storage, data analysis, and data querying. However, the above-mentioned big data replication method for data lakes lacks effective means to detect whether the data is complete and reliable, and cannot effectively and promptly guarantee data reliability.

[0038] The big data replication method, apparatus, electronic device, storage medium, and program product provided in this application solve the above-mentioned technical problems of the prior art by writing data into the subject partition of the data processing platform, and then writing the data in the subject partition into the corresponding time partition of the back-end storage system of the data lake according to the time of writing into the subject partition. The data volume in the subject partition and the data volume in the time partition of the back-end query system are processed to solve the big data replication method, apparatus, electronic device, storage medium, and program product of the data lake.

[0039] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0040] It should be noted that the execution entity in each embodiment of this application can be a server, processor, microprocessor, etc. The specific execution entity in each embodiment of this application is not limited, and it can be selected and set according to actual needs. In the following embodiments, a server is used as an example of the execution entity, which does not constitute a limitation on the actual execution entity.

[0041] Figure 2 A flowchart of the big data replication method for importing data into a lake provided in this application embodiment is shown below. Figure 2 As shown, the method includes:

[0042] S21. Based on the source database, write multiple messages into the corresponding topic partitions respectively. For each message, the message includes the target data corresponding to the source database and time information. The time information is used to indicate the time when the message is written into the corresponding topic partition.

[0043] Source databases are databases that provide data sources. There can be one or more source databases, and the database types of source databases can be one or more. Topic partitions are thematic partitions of the data processing platform. The data processing platform can be deployed on a server and contains multiple topics. These topics are used to distinguish message categories or business domains. When writing messages to a topic, messages can be written to the corresponding topic based on the data category or business domain they correspond to, or messages can be written to topics specified in the producer configuration. Each topic contains multiple partitions. A partition is the smallest storage unit in the data processing platform, storing a subset of messages from a topic.

[0044] Data from the source database is captured using the Change Data Capture (CDC) tool. The target data is written as messages to the corresponding topic partitions. Messages are written to the topic partitions in the order they were written. The CDC tool captures the target data in the database in real time. The target data can be all data in the database or changed data. When the CDC tool writes data to a database for the first time, it writes all data to the corresponding topic partition; in this case, the target data is all data in that database. When the CDC tool is not writing data to a database for the first time, it writes changed data (such as data inserted, updated, or deleted data) to the corresponding topic partitions; in this case, the target data is the changed data in that database. The target data is written to the corresponding topic partitions as messages. Each message includes the target data from the source database and time information. The target data refers to the data in the source database, which can be all data or changed data. The time information records the moment the message was written to the corresponding topic partition. Optionally, the message may also include offset information to identify the message's position within the topic partition.

[0045] S22, based on the time when each message is written to its corresponding topic partition, write the multiple messages to their respective time partitions in the backend storage system.

[0046] A data lake can include backend storage systems and backend query systems. Backend storage systems provide data storage functionality, while backend query systems provide data query functionality. Both can be deployed on servers. When writing messages to the backend storage system, the storage addresses are divided according to time periods. Each time period corresponds to a time partition, and each time partition stores messages written to the topic partition at a time that belongs to that time period. For example, a time partition might correspond to a 15-minute period. Therefore, when storing data, the backend storage system divides the time into 15-minute partitions, and each time partition can only store messages written to the topic partition at a time that belongs to that time partition.

[0047] During the process of writing messages to various time partitions in the backend storage system, there are multiple topic partitions in the data processing platform. Each topic partition corresponds to a processing thread. Therefore, during the process of writing messages to the backend storage system, there are multiple processing threads. Each processing thread writes the messages in its corresponding topic partition to the corresponding time partition in the backend storage system.

[0048] Based on the time when each message was written to its corresponding topic partition, the messages are written to their respective time partitions in the backend storage system. Taking a single processing thread as an example, the thread reads the messages in the topic partitions sequentially according to the time they were written. For each message, it first determines the time partition to which it belongs in the backend storage system, and then writes the message to that time partition. For example, if a topic partition contains messages 1 and 2, with message 1 written at 1:01 and message 2 written at 1:16, and the backend storage system divides time into 15-minute intervals, then time partition 1 corresponds to the interval 1:00-1:15, and time partition 2 corresponds to the interval 1:16-1:30. Therefore, message 1 can be written to time partition 1 in the backend storage system, and message 2 can be written to time partition 2.

[0049] S23, maps the messages written to each time partition in the backend storage system to each time partition in the backend query system.

[0050] The backend query system establishes time partitions with the backend storage system, divided into the same time periods. It then maps messages from each time partition in the backend storage system to the same time partition in the backend query system, so that when querying data, the backend query system can be used to retrieve data from the corresponding time partition.

[0051] S24. Based on the data volume corresponding to the messages written to each topic partition and the data volume queried from each time partition in the backend query system, perform core counting on multiple messages to obtain the core count result.

[0052] The backend storage system contains multiple time partitions. Core counting is performed on each time partition in chronological order. During core counting, a specific time partition to be counted is determined from the multiple time partitions in the backend query system; this partition corresponds to a time partition in the backend storage system. This specific time partition contains multiple messages. By querying the backend query system, the data size of each message in the specific time partition can be determined, thus determining the sum of the data sizes of all messages in the specific time partition.

[0053] For the time partition of the pending core count, determine the messages whose times are written to each topic partition and belong to the time period corresponding to the pending core count time partition. Record the amount of data belonging to the pending core count time partition in each topic partition. Based on the amount of data belonging to the pending core count time partition in each topic partition, obtain the sum of the amount of data belonging to the pending core count time partition in each topic partition.

[0054] The core count result can be obtained by comparing the sum of the data volume of all messages in the time partition to be counted, obtained through the backend query system, with the data volume of messages belonging to the time partition to be counted, obtained through each topic partition.

[0055] The big data replication and indexing method provided in this application embodiment involves a server writing multiple messages into corresponding topic partitions based on a source database. Each message includes target data from the source database and time information, indicating the moment the message was written to the corresponding topic partition. By recording the time information, the amount of target data written to the topic partitions at different times can be determined. Then, based on the time each message was written to its corresponding topic partition, the server writes the multiple messages into different time partitions in the backend storage system. The messages written to each time partition in the backend storage system are mapped to different time partitions in the backend query system, thus determining the amount of data retrieved in each time partition in the backend query system. Finally, based on the amount of data written to each topic partition and the amount of data retrieved in each time partition in the backend query system, the multiple messages are indexed to obtain the indexing result. This embodiment of the application, by recording the time each message was written to its corresponding topic partition, can determine the amount of target data written to each topic partition and perform indexing with the amount of data retrieved in each time partition in the backend query system, thereby verifying the integrity and reliability of the data in the data lake.

[0056] Based on any of the above embodiments, the solutions of the embodiments of this application will be further described below with reference to the accompanying drawings.

[0057] Figure 3 This is a schematic diagram of the core numbering method for replicating big data into a lake, as provided in the embodiments of this application. Figure 3 As shown, it includes the source database, data processing platform, and data lake. The data processing platform contains multiple topic partitions, such as topic partition 1, topic partition 2, ..., topic partition N, where N is a positive integer greater than 1. The data lake includes a backend storage system and a backend query system.

[0058] The data processing platform writes the target data obtained from the source database into various topic partitions of the data processing platform in the form of messages. Then, the data processing platform writes the messages in each topic partition into the back-end storage system of the data lake. Each topic partition corresponds to a processing thread, such as processing thread 1, processing thread 2, ..., processing thread N. The data processing platform writes the messages in the topic partitions corresponding to each processing thread into the back-end storage system of the data lake through each processing thread.

[0059] During core count processing, the core count result is obtained by comparing the sum of the data volume of messages written to the corresponding time partition of the backend storage system in each topic partition of the data processing platform with the sum of the data volume of messages in the corresponding time partition obtained through the backend query system.

[0060] The following section, with reference to the accompanying diagram, describes how to write messages from any given topic partition to the backend storage system.

[0061] Figure 4 This is a flowchart illustrating the message writing to a time partition provided in an embodiment of this application. This method can be applied to a processor, such as... Figure 4 As shown, it includes:

[0062] S41, read at least one message in the topic partition sequentially through the processing thread corresponding to the topic partition.

[0063] The server reads at least one message from the topic partition sequentially through the processing thread corresponding to that topic partition. The at least one message is arranged in the order in which it was written into the topic partition.

[0064] The data processing platform contains multiple topic partitions, each with at least one message. Each topic partition corresponds to a processing thread, and each thread sequentially reads at least one message from its corresponding topic partition. Messages within each topic partition are arranged in a queue according to the order in which they were written. For example, if message 1 is written to topic partition 1 at the first moment, and message 2 is written to topic partition 1 at the second moment (later than the first moment), then message 2 is at the head of the queue, and message 1 is at the tail. Correspondingly, messages written to the partition after the second moment are placed at the head of the queue. Thus, messages from the head to the tail of the queue are arranged in ascending order of their write time.

[0065] Correspondingly, when reading messages from each topic partition, they are read in the order in which they were written to their respective topic partitions. Specifically, when reading messages from each topic partition, messages are first read from the tail of the message queue of each topic partition, that is, the earliest message written to that topic partition is read first. Then, other messages written to that topic partition are read sequentially from the tail of the message queue. In this way, messages from each topic partition are read in the order of their writing time from earliest to latest.

[0066] S42, determine the time partition to which each of the at least one message belongs in the backend storage system based on the time when each message was written to the topic partition.

[0067] The time partitions in the backend storage system are divided into time periods. For example, a time partition might correspond to a 15-minute time period, with each 15-minute period forming a separate time partition. Time partition 1 could correspond to 1:00-1:15, time partition 2 to 1:16-1:30, and so on. For a processing thread, the corresponding topic partition can include at least one message. For example, a topic partition can include M messages: message 1, message 2, ..., message M. The backend storage system's time partitions can include at least one time partition. For example, the backend storage system currently includes three time partitions: time partition 1, time partition 2, and time partition 3. At the current moment, only the newest time partition is allowed to be active; all other time partitions earlier than this one are closed. That is, time partition 3 is active, while time partitions 1 and 2 are closed. Active time partitions are allowed to write messages, while closed time partitions are not.

[0068] In each processing thread, it is necessary to determine whether each message belongs to the current time partition corresponding to the backend storage system. Specifically, for any message in a topic partition, the server determines whether the time when the i-th message was written into that topic partition belongs to the time period corresponding to the j-th time partition. Here, the i-th message is the message already written into that topic partition, and the number of time partitions currently included in the backend storage system is j, where j is a positive integer greater than or equal to 1. If the time when the i-th message was written into that topic partition belongs to the time period corresponding to the j-th time partition, the j-th time partition is determined as the time partition to which the i-th message belongs. If the time when the i-th message was written into that topic partition does not belong to the time period corresponding to the j-th time partition, the (j+1)-th time partition is generated, and the (j+1)-th time partition is determined as the time partition to which the i-th message belongs. Here, the time period corresponding to the j-th time partition is earlier than the time period corresponding to the (j+1)-th time partition, and the time when the i-th message was written into the topic partition belongs to the time period corresponding to the (j+1)-th time partition. Where i is 1, 2, ..., M, M is the number of at least one message included in the topic partition, and M is a positive integer.

[0069] For example, if the j-th time partition (i.e., the latest time partition currently included) of the backend storage system corresponds to the time period 1:00-1:15, and message 1 is written to this topic partition at 1:01, then message 1 is determined to belong to the j-th time partition; if message 2 is written to this topic partition at 1:16, then message 2 is determined not to belong to the j-th time partition.

[0070] If the current message belongs to the latest time partition in the backend storage system, proceed to step S43; if the current message does not belong to the latest time partition in the backend storage system, a new time partition (i.e., the (j+1)th time partition) needs to be generated. If the (j+1)th time partition is generated, the j-th time partition is closed.

[0071] For example, a topic partition contains messages 1 and 2. Message 1 was written to this topic partition at 1:01, and message 2 was written at 1:16. The time period corresponding to the time partition is 15 minutes. The backend storage system currently has one time partition, time partition 1, and the time period corresponding to time partition 1 is 1:00-1:15. Therefore, the time when message 1 was written to the topic partition belongs to the time period corresponding to time partition 1, while the time when message 2 was written to the topic partition does not belong to the time period corresponding to time partition 1. For message 2, a new time partition 2 is created, and time partition 1 is closed. The time period corresponding to time partition 2 is 1:16-1:30, and the time when message 2 was written to the topic partition belongs to the time period corresponding to time partition 2.

[0072] S43, for each message in at least one message, write the message to the time partition to which the message belongs.

[0073] After determining the time partition to which each message belongs in a topic partition, the server writes each message to its corresponding time partition. During the writing process, messages are written to their respective time partitions in the order they were written to the topic partition, ensuring that messages written to the topic partition earlier than the current time partition are written to the backend storage system after the new time partition is generated.

[0074] Through the above steps, for any given topic partition, each message in that topic partition is first checked against the time it was written to that topic partition to determine if it belongs to the time period corresponding to the current time partition. If the message belongs to the time period corresponding to the current time partition, it is written to that time partition. If the message does not belong to the time period corresponding to the current time partition, a new time partition is generated. This process continues until the message belongs to the current time partition, at which point it is written to that time partition. This allows all messages in the topic partition to be written to the backend storage system in the data lake.

[0075] After the messages are written to the backend storage system, the core count of multiple messages can be processed based on the data volume of the messages written to each topic partition and the data volume of each time partition queried in the backend query system. The core count process is described below with reference to the attached diagram.

[0076] Figure 5 This is a schematic diagram of the core count processing provided in the embodiments of this application, such as... Figure 5 As shown, it includes:

[0077] S51: Determine if the core thread pool is idle. If not, execute S51; if so, execute S52.

[0078] S51 is an optional step. The server can perform core counting through a core counting thread pool. The server can check if the thread pool is idle at regular intervals (e.g., 30 minutes, or other suitable intervals). If there is no idle time, it waits and then executes S51 again after a certain period. If there is idle time, it needs to query multiple time partitions in the backend system to determine the time partition to be counted. The time partition to be counted includes at least one first target data. This process can be seen in S52 to S54.

[0079] S52 retrieves configuration information for each of the multiple time partitions. The configuration information includes the core count instruction field, the core count frequency field, and the core count time field.

[0080] The server obtains configuration information for each of the multiple time partitions. The configuration information includes a core count instruction field, a core count frequency field, and a core count time field. The core count instruction field is used to indicate whether to turn core count on or off, the core count frequency field is used to indicate the core count frequency, and the core count time field is used to indicate the time when core count was last performed.

[0081] For example, the configuration information can be obtained from the defined core count configuration table (CHECK_CONFIG), as shown in Table 1 below. The configuration information may include a processing thread primary key field, a core count instruction field, a core count frequency field, a core count time field, a core count status field, and an update time field. The processing thread primary key field is used to uniquely identify each processing thread; the core count instruction field indicates whether core count is turned off or on; for example, 0 can represent turning off core count and 1 can represent turning on core count; the core count frequency field indicates the core count frequency, which is divided in days, such as 3 (once every 3 days), 1 / 24 (once every hour), etc.; the core count time field indicates the time of the last core count; the core count status field indicates the core count status of the corresponding time partition, including not in core count, in core count, and core count failed; for example, 0 can represent not in core count, 1 represents in core count, and 2 represents core count failed; the update time field records the current time. The field types corresponding to the thread primary key field, core count instruction field, core count frequency field, core count time field, core count status field, and update time field are, in order: String, Integer, String, String, Integer, and TIMESTAMP.

[0082]

[0083] S53, based on the core count instruction field of each of the multiple time partitions, determine at least one candidate time partition among the multiple time partitions, and the at least one candidate time partition is the time partition where the core count is enabled.

[0084] Candidate time partitions refer to a set of time partitions where cores are enabled. Whether cores are enabled is determined by the core count instruction field in the configuration information of each time partition. Specifically, in multiple time partitions, the core count instruction field in the configuration information of each time partition is checked. If the core count instruction field indicates that cores are enabled, then the time partition is determined as a candidate time partition; if the core count instruction field indicates that cores are disabled, then cores are not counted for that time partition, and that time partition is not a candidate time partition.

[0085] After determining at least one candidate time partition, the server determines the time partition to be counted from the at least one candidate time partition based on the core count frequency of each candidate time partition and the time of the last core count. This process can be referred to as S54 in the following embodiment.

[0086] S54, determine whether at least one candidate time partition meets the core count condition. If the core count condition is not met, execute S51. If the core count condition is met, execute S55.

[0087] Core counting conditions refer to the conditions that must be met to perform core counting on at least one candidate time partition. These conditions are determined by the core counting frequency and core counting time fields in the candidate time partition configuration information. Time partitions that meet these conditions are designated as the time partitions to be counted. The server first determines the core counting interval for each of the at least one candidate time partition. The core counting interval is the duration between the last time core counting was performed on the candidate time partition and the current time. Then, based on the core counting frequency and core counting interval for each of the at least one candidate time partition, the time partition to be counted is determined from among the at least one candidate time partition.

[0088] Specifically, the server examines the core count frequency field and core count time field of each time partition in the candidate time partition. Based on the core count time field, it determines the core count interval duration of the candidate time partition. The core count interval duration is the duration between the last core count in the candidate time partition and the current time. By comparing the core count interval duration of the candidate time partition with the duration corresponding to the core count frequency, it determines whether the candidate time partition meets the core count condition. The duration corresponding to the core count frequency is the duration interval that must be met between two adjacent core counts. When the core count interval duration of the candidate time partition is less than the duration corresponding to the core count frequency, the candidate time partition does not meet the core count condition, and S51 is executed; when the core count interval duration of the candidate time partition is greater than or equal to the duration corresponding to the core count frequency, the candidate time partition meets the core count condition, and the candidate time partition is determined as the time partition to be core counted, and S55 is executed.

[0089] For example, when multiple candidate time partitions meet the core count conditions and the core count interval is greater than 1 day, the candidate time partitions are split into days. Currently, only candidate time partitions within one day of the last core count are counted, and the core count time is set to the time corresponding to the latest time partition in the candidate time partitions currently being counted. For candidate time partitions with a core count interval greater than one day from the last core count, the above candidate time partitions are counted in the next core count.

[0090] S55, determine the sum of the first data based on the data size of each of the at least one first target data; determine the sum of the second data based on the data size of each of the at least one second target data.

[0091] If a candidate time partition meets the core counting criteria, it is designated as the time partition to be core counted, and the messages within that partition are then core counted. Optionally, the server updates the core counting status field of the time partition to be core counted to indicate that core counting is in progress. Then, the server determines the sum of the first data volumes based on the individual data volumes of at least one first target data set; and determines the sum of the second data volumes based on the individual data volumes of at least one second target data set.

[0092] The first target data refers to the data stored in the time partition to be counted. The data stored in the time partition to be counted includes at least one message written from the data processing platform. Based on the data size corresponding to each message, the sum of the data size of all the first target data in the time partition to be counted can be determined, that is, the sum of the first data size.

[0093] The second target data refers to the target data included in each target message. A target message is a message whose time of writing to the corresponding topic partition belongs to the time period corresponding to the time partition of the pending core count. As long as the message meets the condition that the time of writing to the corresponding topic partition belongs to the time period corresponding to the time partition of the pending core count, it can be used as target information. Therefore, the second target data also includes the target data included in at least one target message that meets the above conditions. The sum of the second data volume is determined according to the data volume of each second target data.

[0094] For example, the second sum of data volume can be obtained from the defined record read data volume table (RECORD_READ_COUNT). This table records the data volume of each message belonging to the corresponding topic partition in the current backend storage system when messages are read from various topic partitions in the data processing platform. The record read data volume table is shown in Table 2 below:

[0095]

[0096] The data volume record table includes the following fields: processing thread primary key, message number, topic partition name, data volume read, and update time. The data types for each field are, in order: String, Integer, String, Long (large integer), and TIMESTAMP. The processing thread primary key uniquely identifies each processing thread; the message number records the number of each message within that thread; the topic partition name records the name of the topic partition to which the current message belongs; the data volume read records the size of the current message; and the update time records the current time.

[0097] For each message belonging to the current time partition of the backend storage system, i.e. the second target data, its specific information is recorded in Table 3. Then, the data volume corresponding to all the second target data is added together to obtain the sum of the second data volume.

[0098] S56. Based on the sum of the first data volume and the sum of the second data volume, perform core counting processing on the messages in the time partition to be counted, and obtain the core counting result corresponding to the time partition to be counted.

[0099] By comparing the sum of the first data volume and the sum of the second data volume corresponding to the time partition to be counted, the core count result corresponding to the time partition to be counted is obtained. If the core count results are consistent, it means that the data in the time partition to be counted is complete. If the core count results are inconsistent, it means that the data in the time partition to be counted is incomplete.

[0100] For example, the core count result can be obtained from a defined core count record table (CHECK_RECORD). This table records specific information about the time partitions to be counted during core count processing, as well as the sum of the first and second data volumes obtained from the time partitions. The core count record table is shown in Table 3 below:

[0101]

[0102] The core counting record table includes a core counting thread primary key field, a core counting start time partition field, a core counting end time partition field, a first data volume sum field, and a second data volume sum field. The field types are String, String, String, Long, and Long, respectively. The core counting thread primary key field uniquely identifies each core counting thread; the core counting start time partition field indicates the start time partition of the current core counting thread; the core counting end time partition field indicates the end time partition of the current core counting thread; the first data volume sum indicates the sum of the message data in the time partition to be counted, determined in the backend query system; and the second data volume sum indicates the sum of the message data in each topic partition, corresponding to the time partition to be counted in the backend storage system. The core counting result is obtained by comparing the first data volume sum field and the second data volume sum field. Optionally, after completing this core counting, the process can return to step S51 for the next core counting.

[0103] Through the above steps, at least one candidate time partition is first determined from multiple time partitions. Then, it is determined whether the at least one candidate time partition meets the core count condition. The time partition that meets the core count condition is determined as the time partition to be core counted. The sum of the first data volume of the partition to be core counted is obtained from the backend query system, and the sum of the second data volume is obtained from the corresponding topic partition. The core count result is obtained by comparing the first data volume sum with the second data volume sum. The solution of this application embodiment can obtain the first data volume sum by recording the data volume of the target data written to each topic partition, and can obtain the second data volume sum by recording the data volume of the target data of each time partition in the backend query system. The core count is performed by comparing the first data volume sum with the second data volume sum, thereby realizing the verification of the integrity and reliability of the data in the data lake.

[0104] Figure 6 This is a schematic diagram of the core data processing device for big data replication into the lake provided in the embodiments of this application, as shown below. Figure 6 As shown, the core data replication device 60 for replicating big data into the lake includes:

[0105] The first processing module 61 is used to write multiple messages into corresponding topic partitions according to the source database. For each message, the message includes the target data corresponding to the source database and time information. The time information is used to indicate the time when the message is written into the corresponding topic partition.

[0106] The second processing module 62 is used to write multiple messages into the time partitions of the backend storage system according to the time when each message is written into its corresponding topic partition.

[0107] The mapping module 63 is used to map messages written to each time partition in the backend storage system to each time partition in the backend query system.

[0108] The core counting module 64 is used to perform core counting on multiple messages based on the data volume corresponding to the messages written to each topic partition and the data volume queried from each time partition in the backend query system, and to obtain the core counting result.

[0109] In one possible implementation, the second processing module 62 is specifically used for:

[0110] The processing thread corresponding to the topic partition reads at least one message from the topic partition in sequence, wherein the at least one message is arranged in the order in which it was written into the topic partition.

[0111] Based on the time when each message was written to the topic partition, determine the time partition to which each message belongs in the backend storage system.

[0112] For each message in at least one message, write the message to the time partition to which the message belongs.

[0113] In one possible implementation, the second processing module 62 is specifically used for:

[0114] Determine whether the time when the i-th message is written to the topic partition belongs to the time period corresponding to the j-th time partition; the number of time partitions currently included in the backend storage system is j, where j is a positive integer greater than or equal to 1;

[0115] If the time when the i-th message is written to the topic partition belongs to the time period corresponding to the j-th time partition, the j-th time partition is determined as the time partition to which the i-th message belongs.

[0116] If the time when the i-th message is written to the topic partition does not belong to the time period corresponding to the j-th time partition, generate the (j+1)-th time partition and determine the (j+1)-th time partition as the time partition to which the i-th message belongs; wherein, the time period corresponding to the j-th time partition is earlier than the time period corresponding to the (j+1)-th time partition, and the time when the i-th message is written to the topic partition belongs to the time period corresponding to the (j+1)-th time partition.

[0117] Where i is 1, 2, ..., M, M is the number of at least one message, and M is a positive integer.

[0118] In one possible implementation, the second processing module 62 is further configured to:

[0119] If the (j+1)th time partition is generated, the jth time partition is closed.

[0120] In one possible implementation, the core number module 64 is specifically used for:

[0121] In the backend query system, determine the time partition to be cored among multiple time partitions, and the time partition to be cored includes at least one first target data;

[0122] The sum of the first data quantities is determined based on the data quantity of each of the at least one first target data.

[0123] Based on the time period corresponding to the time partition of the pending core count, at least one second target data is determined in each topic partition. The at least one second target data is the target data included in the target message. The target message is the message written to the corresponding topic partition at a time that belongs to the time period corresponding to the time partition of the pending core count.

[0124] The sum of the second data volumes is determined based on the data volume of each of the at least one second target data.

[0125] Based on the sum of the first data volume and the sum of the second data volume, perform core counting on the messages in the time partition to be counted, and obtain the core counting result corresponding to the time partition to be counted.

[0126] In one possible implementation, the core number module 64 is specifically used for:

[0127] Obtain the configuration information for each of the multiple time partitions. The configuration information includes the core count instruction field, the core count frequency field, and the core count time field. The core count instruction field is used to indicate whether to turn the core count on or off, the core count frequency field is used to indicate the core count frequency, and the core count time field is used to indicate the time when the core count was last performed.

[0128] Based on the core count instruction field of each of the multiple time partitions, at least one candidate time partition is determined among the multiple time partitions, and the at least one candidate time partition is the time partition where the core count is enabled;

[0129] Based on the core counting frequency of each of the at least one candidate time partitions and the time of the last core counting, determine the time partition to be counted from the at least one candidate time partitions.

[0130] In one possible implementation, the core number module 64 is specifically used for:

[0131] Determine the core count interval for each of at least one candidate time partition. The core count interval is the duration between the last time core count was performed on the candidate time partition and the current time.

[0132] Based on the core count frequency and core count interval duration of each of the at least one candidate time partition, determine the time partition to be counted from the at least one candidate time partition.

[0133] In one possible implementation, the core number module 64 is also used for:

[0134] Update the core count status field of the time partition to be counted. The updated core count status field of the time partition to be counted indicates that the core count status is in progress.

[0135] The big data replication device 60 provided in this application embodiment can execute the technical solution in the above method embodiment. Its implementation principle and beneficial effects are similar, and will not be described again here.

[0136] Figure 7 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application, such as... Figure 7 As shown, the system includes a memory 71 and a processor 72, which are interconnected via a bus 73.

[0137] Memory 71 is used to store instructions executed by the computer;

[0138] The processor 72 is used to execute computer execution instructions stored in the memory to implement the methods in the above embodiments.

[0139] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily essential to this application.

[0140] It should be further noted that although the steps in the flowchart are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowchart may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0141] It should be understood that the above-described device embodiments are merely illustrative, and the device of this application can also be implemented in other ways. For example, the division of units / modules in the above embodiments is only a logical functional division, and there may be other division methods in actual implementation. For example, multiple units, modules, or components may be combined, or integrated into another system, or some features may be ignored or not executed.

[0142] Furthermore, unless otherwise specified, the functional units / modules in the various embodiments of this application can be integrated into one unit / module, or each unit / module can exist physically separately, or two or more units / modules can be integrated together. The integrated units / modules described above can be implemented in hardware or as software program modules.

[0143] When integrated units / modules are implemented in hardware, the hardware can be digital circuits, analog circuits, etc. The physical implementation of the hardware structure includes, but is not limited to, transistors, memristors, etc. Unless otherwise specified, the processor can be any suitable hardware processor, such as a CPU, GPU, FPGA, DSP, and ASIC, etc. Unless otherwise specified, the storage unit can be any suitable magnetic or magneto-optical storage medium, such as Resistive Random Access Memory (RRAM), Dynamic Random Access Memory (DRAM), Static Random Access Memory (SRAM), Enhanced Dynamic Random Access Memory (EDRAM), High-Bandwidth Memory (HBM), Hybrid Memory Cube (HMC), etc.

[0144] If the integrated unit / module is implemented as a software program module and sold or used as an independent product, it can be stored in a computer-readable storage device (CMD). Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a memory and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned memory includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0145] In the above embodiments, the descriptions of each embodiment have their own emphasis. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments. The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as the combination of these technical features does not contradict each other, it should be considered within the scope of this specification.

[0146] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the following claims.

[0147] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A core counting method for replicating large amounts of data into a lake, characterized in that, The method includes: Based on the source database, multiple messages are written into corresponding topic partitions. For each message, the message includes the target data corresponding to the source database and time information, and the time information is used to indicate the time when the message is written into the corresponding topic partition. Based on the time when each of the multiple messages was written to its corresponding topic partition, the multiple messages are written to their respective time partitions in the backend storage system. The messages written to each time partition in the backend storage system are mapped to each time partition in the backend query system; Based on the data volume corresponding to the messages written to each topic partition and the data volume queried from each time partition in the backend query system, the multiple messages are processed to obtain the core count result.

2. The method according to claim 1, characterized in that, For each topic partition, the step of writing the multiple messages into the respective time partitions of the backend storage system according to the time when each message was written into its corresponding topic partition includes: The processing thread corresponding to the topic partition sequentially reads at least one message from the topic partition, wherein the at least one message is arranged in the order in which it was written into the topic partition; Based on the time when each of the at least one message was written into the topic partition, the time partition to which each of the at least one message belongs in the backend storage system is determined. For each message in the at least one message, the message is written into the time partition to which the message belongs.

3. The method according to claim 2, characterized in that, Determining the time partition to which each of the at least one message belongs in the backend storage system based on the time when each message was written into the topic partition includes: Determine whether the time when the i-th message is written to the topic partition belongs to the time period corresponding to the j-th time partition; the number of time partitions currently included in the backend storage system is j, where j is a positive integer greater than or equal to 1; If the time when the i-th message is written to the topic partition belongs to the time period corresponding to the j-th time partition, the j-th time partition is determined as the time partition to which the i-th message belongs; If the time when the i-th message is written to the topic partition does not belong to the time period corresponding to the j-th time partition, a (j+1)-th time partition is generated, and the (j+1)-th time partition is determined as the time partition to which the i-th message belongs; wherein, the time period corresponding to the j-th time partition is earlier than the time period corresponding to the (j+1)-th time partition, and the time when the i-th message is written to the topic partition belongs to the time period corresponding to the (j+1)-th time partition; Where i is 1, 2, ..., M, M is the number of the at least one message, and M is a positive integer.

4. The method according to claim 3, characterized in that, The method further includes: In the case of generating the (j+1)th time partition, the jth time partition is closed.

5. The method according to any one of claims 1-4, characterized in that, The process of performing core counting on the multiple messages based on the data volume corresponding to the messages written to each topic partition and the data volume retrieved from each time partition in the backend query system to obtain core counting results includes: Among the multiple time partitions of the backend query system, a time partition for the number of cores to be counted is determined, and the time partition for the number of cores to be counted includes at least one first target data. The sum of the first data amounts is determined based on the data size of each of the at least one first target data. Based on the time period corresponding to the time partition of the pending core count, at least one second target data is determined in each topic partition. The at least one second target data is target data included in the target message. The target message is a message whose time of writing to the corresponding topic partition belongs to the time period corresponding to the time partition of the pending core count. The sum of the second data amounts is determined based on the data amount of each of the at least one second target data. Based on the sum of the first data volume and the sum of the second data volume, the messages in the time partition to be counted are processed to obtain the core count result corresponding to the time partition to be counted.

6. The method according to claim 5, characterized in that, The step of determining the time partition for the number of cores among multiple time partitions in the backend query system includes: Obtain the configuration information of each of the multiple time partitions. The configuration information includes a core count instruction field, a core count frequency field, and a core count time field. The core count instruction field is used to indicate whether to turn the core count off or on. The core count frequency field is used to indicate the core count frequency. The core count time field is used to indicate the time when the core count was last performed. Based on the core count instruction field of each of the plurality of time partitions, at least one candidate time partition is determined among the plurality of time partitions, and the at least one candidate time partition is the time partition in which the core count is enabled. The time partition to be counted is determined from the at least one candidate time partition based on the core count frequency of each of the at least one candidate time partition and the time of the last core count.

7. The method according to claim 6, characterized in that, The step of determining the time partition to be counted from the at least one candidate time partitions based on the core count frequency of each of the at least one candidate time partitions and the time of the last core count includes: Determine the core count interval for each of the at least one candidate time partition, wherein the core count interval is the duration between the last time core count was performed on the candidate time partition and the current time. The time partition to be counted is determined from the at least one candidate time partition based on the core count frequency and core count interval duration of each of the at least one candidate time partition.

8. The method according to claim 6, characterized in that, The configuration information also includes a core count status field, which indicates the core count status of the corresponding time partition. The method further includes: Update the core count status field of the time partition to be counted, and the updated core count status field of the time partition to be counted indicates that the core count status is in progress.

9. A core data replication device for importing big data into a lake, characterized in that, The device includes: The first processing module is used to write multiple messages into corresponding topic partitions according to the source database. For each message, the message includes target data and time information corresponding to the source database. The time information is used to indicate the time when the message is written into the corresponding topic partition. The second processing module is used to write the multiple messages into the respective time partitions of the backend storage system according to the time when each message was written into its corresponding topic partition. The mapping module is used to map messages written to each time partition in the backend storage system to each time partition in the backend query system. The core counting module is used to perform core counting on the multiple messages based on the data volume corresponding to the messages written to each topic partition and the data volume queried from each time partition in the backend query system, and to obtain the core counting result.

10. An electronic device, characterized in that, include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the method as described in any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1 to 8.

12. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method of any one of claims 1 to 8.

Images