Method and device for copying big data into lake kernel number, electronic equipment, storage medium and program product

Abstract

The invention provides a kernel number method and device for copying big data into a lake, electronic equipment, a storage medium and a program product, and relates to the field of big data. The method comprises the steps that according to a source database, a plurality of messages are written into corresponding theme partitions respectively, and for each message, the message comprises corresponding target data and time information in the source database; respectively writing the plurality of messages into each time partition in a back-end storage system according to the moments when the plurality of messages are respectively written into the corresponding theme partitions; mapping the messages written into the time partitions in the back-end storage system to the time partitions in the back-end query system; and according to the data volume corresponding to the messages written into each theme partition and the data volume queried by each time partition in the back-end query system, performing kernel number processing on the plurality of messages to obtain a kernel number result. According to the method, the integrity and reliability of the data in the data lake can be verified.

Description

Technical Field

[0001] The present application relates to the field of big data, and in particular to a method, device, electronic device, storage medium and program product for copying big data into a lake. Background Art

[0002] Big data replication into the lake refers to the process of copying large amounts of data from the source database and importing it into the data lake.

[0003] In the related technology, a data processing platform is used to implement big data replication into a lake. Specifically, data is written from a source database into a data processing platform, and after the data is processed by the data processing platform, the data is written into a data lake.

[0004] However, the above process of using the data processing platform to copy big data into the data lake cannot ensure the integrity and reliability of the data in the data lake. Summary of the invention

[0005] The present application provides a core counting method, device, electronic device, storage medium and program product for copying big data into a data lake, so as to solve the technical problem of being unable to ensure the integrity and reliability of data in a data lake.

[0006] In a first aspect, the present application provides a core counting method for copying big data into a lake, the method comprising:

[0007] According to the source database, multiple messages are written into corresponding topic partitions respectively, wherein for each message, the message includes the corresponding target data in the source database and time information, and the time information is used to indicate the time when the message is written into the corresponding topic partition;

[0008] According to the time when each of the multiple messages is written into the corresponding topic partition, the multiple messages are written into each time partition in the backend storage system respectively;

[0009] Map the messages written to each time partition in the backend storage system to each time partition in the backend query system;

[0010] According to the data volume corresponding to the messages written into each topic partition and the data volume queried in each time partition in the backend query system, multiple messages are processed to obtain the core count results.

[0011] In a second aspect, the present application provides a core counting device for copying big data into a lake, the device comprising:

[0012] A first processing module is used to write multiple messages into corresponding topic partitions according to a source database, wherein for each message, the message includes corresponding target data in the source database and time information, and the time information is used to indicate the time when the message is written into the corresponding topic partition;

[0013] The second processing module is used to write the multiple messages into each time partition in the backend storage system according to the time when the multiple messages are each written into the corresponding topic partition;

[0014] A mapping module, used to map the messages written into each time partition in the backend storage system to each time partition in the backend query system;

[0015] The core count module is used to perform core count processing on multiple messages to obtain core count results based on the data volume corresponding to the messages written into each topic partition and the data volume queried in each time partition in the backend query system.

[0016] In a third aspect, the present application provides an electronic device, comprising: a processor, and a memory communicatively connected to the processor;

[0017] Memory stores computer-executable instructions;

[0018] The processor executes the computer-executable instructions stored in the memory to implement the method described in the first aspect.

[0019] In a fourth aspect, the present application provides a computer-readable storage medium, in which computer execution instructions are stored. When the computer execution instructions are executed by a processor, they are used to implement the above-mentioned core number method for copying big data into a lake.

[0020] In a fifth aspect, the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the above-mentioned core number method for copying big data into a lake.

[0021] The embodiment of the present application provides a method for counting the number of big data copied into the lake. The server writes multiple messages into the corresponding subject partitions according to the source database, wherein for each message, the message includes the corresponding target data and time information in the source database, and the time information is used to indicate the time when the message is written into the corresponding subject partition. By recording the time information, the data volume of the target data written into the subject partition in different time periods can be determined; then, the server writes the multiple messages into each time partition in the backend storage system according to the time when the multiple messages are written into the corresponding subject partition; the messages written into each time partition in the backend storage system are mapped to each time partition in the backend query system, so that the data volume queried by each time partition in the backend query system can be determined, and finally, according to the data volume corresponding to the message written into each subject partition and the data volume queried by each time partition in the backend query system, the multiple messages are processed to obtain the result of the number of numbers. The scheme of the embodiment of the present application can determine the data volume of the target data written into each subject partition by recording the time when each message is written into the corresponding subject partition, and count it with the data volume queried by each time partition in the backend query system, so as to realize the verification of the integrity and reliability of the data in the data lake. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0023] Figure 1 A schematic diagram of an application scenario provided for an embodiment of the present application;

[0024] Figure 2 A flowchart of a core count method for copying big data into a lake provided in an embodiment of the present application;

[0025] Figure 3 A schematic diagram of the process of the core number method for copying big data into a lake provided in an embodiment of the present application;

[0026] Figure 4 A schematic diagram of the process of writing a message into time partitions provided in an embodiment of the present application;

[0027] Figure 5 A schematic diagram of the process flow of core number processing provided in an embodiment of the present application;

[0028] Figure 6 A schematic diagram of the structure of a core counting device for copying big data into a lake provided in an embodiment of the present application;

[0029] Figure 7 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0030] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0031] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0032] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of the relevant data comply with the relevant laws, regulations and standards of the relevant countries and regions, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0033] It should be noted that the core counting method, device, electronic device, storage medium and program product for copying big data into a lake provided in this application can be used in the field of big data, and can also be used in any field other than big data. The application field of the core counting method, device, electronic device, storage medium and program product for copying big data into a lake in this application is not limited.

[0034] First combine Figure 1 An applicable application scenario of the present application is introduced.

[0035] Figure 1 A schematic diagram of an application scenario provided by an embodiment of the present application. Figure 1 As shown, it includes a source database 11, a data processing platform 12 and a data lake 13.

[0036] The source database 11 is a database that provides data, and the number of source databases 11 can be one or more; the data processing platform 12 is a platform that provides real-time data processing and analysis, and the data processing platform 12 can be deployed on a server. The server can collect real-time data from the source database 11 through the data processing platform 12 and store it in a message queue, so that the server can asynchronously read data from the data processing platform 12; the data lake 13 is the destination of big data replication, and the data lake 13 can provide data storage, query, analysis and other functions. The data processing platform 12 collects data from the source database 11 in real time, and then writes the data into the data lake 13.

[0037] In the related technology, the data in the source database is written into the data processing platform through a replication tool. The server reads the data in the data processing platform, processes the data, and then writes the data into the corresponding time partition in the data lake according to the time when the data is written into the data processing platform, so that the data lake can be used for data storage, data analysis, data query and other operations. However, in the above-mentioned method of copying big data into the lake, there is no effective means to detect whether the data is complete and reliable, and it is impossible to effectively and timely guarantee the reliability of the data.

[0038] The present application provides a method, device, electronic device, storage medium and program product for counting big data copied into a lake, which aims to solve the above technical problems of the prior art by writing data into a subject partition of a data processing platform, and then writing the data in the subject partition into a corresponding time partition in the backend storage system of the data lake at the time when the data is written into the subject partition, and performing core counting on the corresponding data volume in the subject partition and the data volume in the time partition of the backend query system.

[0039] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0040] It should be noted that the execution subject of each embodiment of the present application can be a server, a processor, a microprocessor, etc. The specific execution subject of each embodiment of the present application is not limited, and it can be selected and set according to actual needs. In the following embodiments, the execution subject is introduced as a server as an example, which does not constitute a limitation on the actual execution subject.

[0041] Figure 2 A flowchart of a core count method for copying big data into a lake provided in an embodiment of the present application, such as Figure 2 As shown, the method includes:

[0042] S21, according to the source database, write multiple messages into corresponding topic partitions respectively, wherein for each message, the message includes the corresponding target data in the source database and time information, and the time information is used to indicate the moment when the message is written into the corresponding topic partition.

[0043] The source database is the database that provides the data source. The number of source databases can be one or more, and the database type of the source database can be one or more. The topic partition is the topic partition of the data processing platform. The data processing platform can be deployed in the server. The data processing platform contains multiple topics, and multiple topics are used to distinguish message categories or business areas. When writing a message to a topic, it can be written to the corresponding topic according to the category or business field of the data corresponding to the message, or it can be written to the topic specified by the configuration of the producer. Each topic contains multiple partitions. The partition is the smallest storage unit in the data processing platform, which stores a subset of the messages in a topic.

[0044] The data in the source database is captured by the Change Data Capture (CDC) tool, and the target data is written into the corresponding topic partition in the form of messages. The messages written into the topic partition are arranged in the order of the time of writing into the topic partition. Among them, the CDC tool captures the target data in the database in real time. The target data can be all the data in the database or the changed data in the database. When the CDC tool writes data in a database for the first time, all the data in the database is written into the corresponding topic partition. In this case, the target data is all the data in the database; when the CDC tool does not write data in a database for the first time, the changed data in the database is written into the corresponding topic partition. The changed data is the data involved in operations such as data insertion, data update, and data deletion in the database. In this case, the target data is the changed data in the database. The target data is written into the corresponding topic partition in the form of messages. The messages written into the topic partition include the corresponding target data in the source database and time information. The target data refers to the data in the source database. The target data can be all the data or the changed data in the source database. The time information is used to record the time when the message is written into the corresponding topic partition. Optionally, the message can also include offset information, which is used to identify the position of the message in the topic partition.

[0045] S22, writing the multiple messages into each time partition in the backend storage system according to the time when the multiple messages are each written into the corresponding topic partition.

[0046] The data lake may include a backend storage system and a backend query system. The backend storage system refers to a system that provides data storage functions in the data lake and is used to store data; the backend query system refers to a system that provides data query functions in the data lake and is used to query data. The backend storage system and the backend query system may also be deployed on a server. When writing messages in the backend storage system, the storage address of the backend storage system is divided according to time periods. Each time period corresponds to a time partition, and each time partition stores messages that belong to the time period when written to the subject partition. For example, the time period corresponding to a time partition may be 15 minutes. When the backend storage system performs data storage, it divides a time partition into 15 minutes, and each time partition can only store messages that belong to the time partition when written to the subject partition.

[0047] In the process of writing messages to each time partition in the backend storage system, there are multiple topic partitions in the data processing platform, and each topic partition corresponds to a processing thread. Therefore, in the process of writing messages to the backend storage system, there are multiple processing threads, and each processing thread writes the messages in the corresponding topic partitions to the corresponding time partitions in the backend storage system.

[0048] According to the time when the multiple messages are written into the corresponding topic partitions, the multiple messages are written into each time partition in the backend storage system. Taking a processing thread as an example, the processing thread reads the messages in the topic partition in sequence according to the time when the messages are written into the topic partition. For each message, the time partition to which the message belongs in the backend storage system is first determined, and then the message is written into the time partition to which it belongs. For example, there are message 1 and message 2 in a topic partition. The time when message 1 is written into the topic partition is 1:01, and the time when message 2 is written into the topic partition is 1:16. The time partitions in the backend storage system are divided into 15-minute time partitions. The time period corresponding to time partition 1 is 1:00-1:15, and the time period corresponding to time partition 2 is 1:16-1:30. Then, message 1 can be written into time partition 1 in the backend storage system, and message 2 can be written into time partition 2 in the backend storage system.

[0049] S23, mapping the messages written into each time partition in the backend storage system to each time partition in the backend query system.

[0050] The back-end query system establishes time partitions divided by the same time period as the back-end storage system, and maps the messages of each time partition in the back-end storage system to the same time partition of the back-end query system one by one, so that when performing data queries, the data of the corresponding time partition can be queried through the back-end query system.

[0051] S24, according to the data volume corresponding to the messages written into each topic partition and the data volume queried in each time partition in the backend query system, multiple messages are processed to obtain the core count result.

[0052] The backend storage system contains multiple time partitions. For each time partition, the core count processing is performed in chronological order. When performing the core count processing, a time partition to be counted is determined from multiple time partitions in the backend query system. This time partition corresponds to the time partition of the backend storage system. The time partition to be counted includes multiple messages. Through the backend query system, the query operation can be performed to determine the data volume of each message in the time partition to be counted, and then determine the sum of the data volume of all messages in the time partition to be counted.

[0053] For the time partition to be approved, determine in each subject partition the messages that belong to the time period corresponding to the time partition to be approved at the moment of writing into each subject partition, record the data volume of the messages belonging to the time partition to be approved in each subject partition, and based on the data volume of the messages belonging to the time partition to be approved in each subject partition, obtain the sum of the data volume of the messages belonging to the time partition to be approved in each subject partition.

[0054] The number of verification results can be obtained by comparing the sum of the data volume of all messages in the time partition to be verified obtained through the backend query system with the data volume of the messages belonging to the time partition to be verified obtained through each topic partition.

[0055] The embodiment of the present application provides a method for counting the number of big data copied into the lake. The server writes multiple messages into the corresponding subject partitions according to the source database, wherein for each message, the message includes the corresponding target data and time information in the source database, and the time information is used to indicate the time when the message is written into the corresponding subject partition. By recording the time information, the data volume of the target data written into the subject partition in different time periods can be determined; then, the server writes the multiple messages into each time partition in the backend storage system according to the time when the multiple messages are written into the corresponding subject partition; the messages written into each time partition in the backend storage system are mapped to each time partition in the backend query system, so that the data volume queried by each time partition in the backend query system can be determined, and finally, according to the data volume corresponding to the message written into each subject partition and the data volume queried by each time partition in the backend query system, the multiple messages are processed to obtain the result of the number of numbers. The scheme of the embodiment of the present application can determine the data volume of the target data written into each subject partition by recording the time when each message is written into the corresponding subject partition, and count it with the data volume queried by each time partition in the backend query system, so as to realize the verification of the integrity and reliability of the data in the data lake.

[0056] Based on any of the above embodiments, the scheme of the embodiments of the present application is further introduced below in conjunction with the accompanying drawings.

[0057] Figure 3 A schematic diagram of the core number method process for copying big data into a lake provided in an embodiment of the present application is shown in FIG. Figure 3 As shown, it includes a source database, a data processing platform, and a data lake. The data processing platform contains multiple topic partitions, such as topic partition 1, topic partition 2, ..., topic partition N, where N is a positive integer greater than 1, and the data lake includes a backend storage system and a backend query system.

[0058] The data processing platform writes the target data obtained from the source database into each topic partition of the data processing platform in the form of messages, and then writes the messages in each topic partition into the backend storage system in the data lake, where each topic partition corresponds to a processing thread, such as processing thread 1, processing thread 2, ..., processing thread N. The data processing platform writes the messages in the topic partition corresponding to each processing thread into the backend storage system of the data lake through each processing thread.

[0059] When performing core counting, the core counting result is obtained by comparing the sum of the data volume of messages in the time partition corresponding to the backend storage system in each topic partition in the data processing platform with the sum of the data volume of messages in the corresponding time partition obtained through the backend query system.

[0060] For any topic partition, the following describes the process of how to write messages in the topic partition into the backend storage system in conjunction with the attached figure.

[0061] Figure 4 The flowchart of the message writing time partition provided in the embodiment of the present application is as follows. The method can be applied to a processor such as Figure 4 As shown, including:

[0062] S41, reading at least one message in the topic partition in sequence through the processing thread corresponding to the topic partition.

[0063] The server reads at least one message in the topic partition in sequence through the processing thread corresponding to the topic partition, wherein the at least one message is arranged in the order of the time when it is written into the topic partition.

[0064] There are multiple topic partitions in the data processing platform, and there is at least one message in each topic partition. Each topic partition corresponds to a processing thread, and each processing thread reads at least one message in the corresponding topic partition in turn. The messages in each topic partition are arranged in the form of a queue according to the order of the time when they are written into the corresponding topic partition. Taking a certain topic partition as an example, if message 1 is written into topic partition 1 at the first moment, and message 2 is written into topic partition 1 at the second moment, and the second moment is later than the first moment, then message 2 is located at the head of the queue and message 1 is located at the tail of the queue. Correspondingly, the messages written into the partition after the second moment are located at the head of the queue in turn. In this way, the messages from the head of the queue to the tail of the queue are arranged in order from late to early by the writing time.

[0065] Correspondingly, when reading messages in each topic partition, they are also read in the order of the time when the messages are written into the corresponding topic partitions. Specifically, when reading messages in each topic partition, the messages are first read from the tail of the message queue of each topic partition, that is, the message written to the topic partition earliest is read first, and then other messages written to the topic partition are read from the tail of the message queue in sequence. In this way, when reading messages in each topic partition, they are read in the order of the time when the messages are written from early to late.

[0066] S42, determining the time partition to which each of the at least one message belongs in the backend storage system according to the time when each of the at least one message is written into the topic partition.

[0067] The time partitions in the backend storage system are divided into time periods. For example, the time period corresponding to the time partition may be 15 minutes, and each 15 minutes corresponds to a time partition. The time period corresponding to time partition 1 may be 1:00-1:15, and the time period corresponding to time partition 2 may be 1:16-1:30, etc. For a processing thread, the subject partition corresponding to the processing thread may include at least one message. For example, the subject partition may include M messages, namely: message 1, message 2, ..., message M; the time partitions of the backend storage system may include at least one time partition. For example, the number of time partitions currently included in the backend storage system is 3, namely: time partition 1, time partition 2, and time partition 3. At the current moment, only the latest time partition is allowed to be in the open state, and the other time partitions earlier than the time partition are in the closed state, that is, time partition 3 is in the open state, and time partition 1 and time partition 2 are in the closed state. The time partition in the open state allows writing messages, and the time partition in the closed state does not allow writing messages.

[0068] In each processing thread, it is necessary to determine whether each message belongs to the current time partition corresponding to the backend storage system. Specifically, for any message in a topic partition, the server determines whether the moment when the i-th message is written into the topic partition belongs to the time period corresponding to the j-th time partition, wherein the i-th message is a message that has been written into the topic partition, and the number of time partitions currently included in the backend storage system is j, and j is a positive integer greater than or equal to 1. In the case that the moment when the i-th message is written into the topic partition belongs to the time period corresponding to the j-th time partition, the j-th time partition is determined as the time partition to which the i-th message belongs; in the case that the moment when the i-th message is written into the topic partition does not belong to the time period corresponding to the j-th time partition, the j+1-th time partition is generated, and the j+1-th time is determined as the time partition to which the i-th message belongs; wherein the time period corresponding to the j-th time partition is earlier than the time period corresponding to the j+1-th time partition, and the moment when the i-th message is written into the topic partition belongs to the time period corresponding to the j+1-th time partition. Here, i is 1, 2, ..., M, M is the number of at least one message included in the topic partition, and M is a positive integer.

[0069] For example, the jth time partition (that is, the latest time partition currently included) included in the backend storage system corresponds to the time period of 1:00-1:15, and the time when message 1 is written into the topic partition is 1:01, then it is determined that message 1 belongs to the jth time partition; the time when message 2 is written into the topic partition is 1:16, then it is determined that message 2 does not belong to the jth time partition.

[0070] If the current message belongs to the latest time partition in the backend storage system, execute step S43; if the current message does not belong to the latest time partition in the backend storage system, a new time partition (i.e., the j+1th time partition) needs to be generated. When the j+1th time partition is generated, close the jth time partition.

[0071] For example, a topic partition includes message 1 and message 2. The time when message 1 is written into the topic partition is 1:01, and the time when message 2 is written into the topic partition is 1:16. The time period corresponding to the time partition is 15 minutes. The backend storage system currently includes one time partition, which is time partition 1. The time period corresponding to time partition 1 is 1:00-1:15. The time when message 1 is written into the topic partition belongs to the time period corresponding to time partition 1, and the time when message 2 is written into the topic partition does not belong to the time period corresponding to time partition 1. For message 2, a new time partition 2 is generated, and time partition 1 is closed. The time period corresponding to time partition 2 is 1:16-1:30, and the time when message 2 is written into the topic partition belongs to the time period corresponding to time partition 2.

[0072] S43: For each message in the at least one message, write the message into the time partition to which the message belongs.

[0073] After determining the time partition to which each message in the topic partition belongs, the server writes each message into the time partition to which the message belongs. During the writing process, the messages are written into the time partition in the order of the time when they are written into the topic partition, so as to ensure that after the new time partition is generated, the messages written into the topic partition earlier than the time corresponding to the current time partition can be written into the backend storage system.

[0074] Through the above steps, for any topic partition, firstly, for each message in the topic partition, according to the time when the message is written into the topic partition, it is determined whether the message belongs to the time period corresponding to the current time partition. If the message belongs to the time period corresponding to the current time partition, the message is written into the current time partition. If the message does not belong to the time period corresponding to the current time partition, a new time partition is generated until the message belongs to the current time partition, and then the message is written into the current time partition, so that each message in the topic partition can be written into the backend storage system in the data lake.

[0075] After the messages are written to the backend storage system, the number of messages can be processed according to the data size corresponding to the messages written to each topic partition and the data size queried in each time partition in the backend query system to obtain the number of results. The following is an introduction to the number of results process with reference to the accompanying drawings.

[0076] Figure 5 A schematic diagram of the process flow of core number processing provided in the embodiment of the present application is as follows: Figure 5 As shown, including:

[0077] S51, determine whether the core number thread pool is idle, if not, execute S51, if yes, execute S52.

[0078] S51 is an optional step. The server can perform the core count processing process through the core count thread pool. The server can check whether the thread pool is idle at a certain time interval (for example, 30 minutes, or other suitable time intervals). If there is no idle time, wait first, and execute S51 again after waiting for a certain time. If there is idle time, it is necessary to determine the waiting core count time partition in the multiple time partitions of the backend query system. The waiting core count time partition includes at least one first target data. The process can refer to S52 to S54.

[0079] S52, obtaining configuration information of each of the multiple time partitions, where the configuration information includes a core number instruction field, a core number frequency field, and a core number time field.

[0080] The server obtains the configuration information of multiple time partitions. The configuration information includes a core count instruction field, a core count frequency field, and a core count time field. The core count instruction field is used to indicate whether to turn off or on the core count, the core count frequency field is used to indicate the core count frequency, and the core count time field is used to indicate the time when the core count was last performed.

[0081] Exemplarily, the configuration information is obtained by a defined core count configuration table (CHECK_CONFIG), and the core count configuration table is shown in Table 1 below. The configuration information may include a processing thread primary key field, a core count instruction field, a core count frequency field, a core count time field, a core count status field, and an update time field. The processing thread primary key field is used to uniquely identify each processing thread; the core count instruction field is used to indicate whether to turn off or on the core count. Exemplarily, 0 can be used to indicate turning off the core count, and 1 can be used to indicate turning on the core count; the core count frequency field is used to indicate the core count frequency, which is divided in days. For example, the core count frequency can be 3 (once every 3 days), 1 / 24 (once every hour), and so on; the core count time field is used to indicate the time when the core count was last performed; the core count status field is used to indicate the core count status of the corresponding time partition, and the core count status includes not in the core count, in the core count, and failed in the core count. Exemplarily, 0 can be used to indicate not in the core count, 1 can be used to indicate in the core count, and 2 can indicate failed in the core count; the update time field is used to record the current time. The field types corresponding to the processing thread primary key field, core number instruction field, core number frequency field, core number time field, core number status field, and update time field are: String, Integer, String, String, Integer, and TIMESTAMP.

[0082]

[0083] S53, determining at least one candidate time partition from the multiple time partitions according to the core number instruction fields of the multiple time partitions, wherein the at least one candidate time partition is a time partition for enabling the core number.

[0084] A candidate time partition refers to a set of time partitions with cores turned on. Whether the cores are turned on is determined by the core count instruction field in the configuration information of each time partition. Specifically, among multiple time partitions, the core count instruction field in the configuration information of each time partition is checked. If the core count instruction field indicates turning on the cores, the time partition is determined to be a candidate time partition; if the core count instruction field indicates turning off the cores, the time partition is not counted and is not a candidate time partition.

[0085] After determining at least one candidate time partition, the server determines a time partition to be counted in the at least one candidate time partition according to the core count frequency of each of the at least one candidate time partition and the time of the last core count. This process can be seen in S54 in the following embodiment.

[0086] S54, determining whether at least one candidate time partition satisfies the core number condition. If not, execute S51; if so, execute S55.

[0087] The core count condition refers to the condition that needs to be met for performing core count on at least one candidate time partition. The core count condition is determined by the core count frequency field and the core count time field in the candidate time partition configuration information. The time partition that meets the core count condition is determined as the time partition to be counted. The server first determines the core count interval duration of at least one candidate time partition. The core count interval duration is the duration between the last time the candidate time partition was counted and the current time. Then, according to the core count frequency and core count interval duration of at least one candidate time partition, the server determines the time partition to be counted in at least one candidate time partition.

[0088] Specifically, the server checks the core number frequency field and the core number time field of each time partition in the candidate time partition, and determines the core number interval duration of the candidate time partition based on the core number time field. The core number interval duration is the duration between the last time the core number was performed in the candidate time partition and the current time. By comparing the core number interval duration of the candidate time partition with the duration corresponding to the core number frequency, it is determined whether the candidate time partition meets the core number condition. The duration corresponding to the core number frequency is the time interval that needs to be met between two adjacent core numbers. When the core number interval duration of the candidate time partition is less than the duration corresponding to the core number frequency, the candidate time partition does not meet the core number condition, and S51 is executed; when the core number interval duration of the candidate time partition is greater than or equal to the duration corresponding to the core number frequency, the candidate time partition meets the core number condition, and the candidate time partition is determined as the time partition to be counted, and S55 is executed.

[0089] Exemplarily, when there are multiple candidate time partitions that meet the core count conditions and the core count interval is greater than 1 day, the candidate time partitions are split according to days. Currently, only the candidate time partitions that are within one day from the last core count are counted, and the core count time is set to the moment corresponding to the latest time partition in the candidate time partitions of the current core count. For candidate time partitions that are more than one day away from the last core count, the above candidate time partitions are counted at the next core count.

[0090] S55, determining the sum of the first data amounts according to the data amount size of at least one first target data; determining the sum of the second data amounts according to the data amount size of at least one second target data.

[0091] If the candidate time partition meets the core count condition, the candidate time partition is determined as the time partition to be counted, and the messages in the time partition to be counted are counted. Optionally, the server updates the core count status field of the time partition to be counted, and updates the core count status field of the time partition to be counted to the core counting state. Then the server determines the sum of the first data amount according to the data amount of at least one first target data; and determines the sum of the second data amount according to the data amount of at least one second target data.

[0092] The first target data refers to the data stored in the time partition of the waiting number of cores. The data stored in the time partition of the waiting number of cores includes at least one message written from the data processing platform. According to the data volume corresponding to each message, the sum of the data volume of all the first target data in the time partition of the waiting number of cores can be determined, that is, the sum of the first data volume.

[0093] The second target data refers to the target data included in each target message. The target message refers to the message that the moment when it is written into the corresponding subject partition belongs to the time period corresponding to the time partition of the number to be verified. As long as the message meets the condition that the moment when it is written into the corresponding subject partition belongs to the time period corresponding to the time partition of the number to be verified, it can be used as the target information. Therefore, the second target data also includes at least one target data included in the target message that meets the above conditions. The sum of the second data amounts is determined according to the data amounts of each of the second target data.

[0094] Exemplarily, the sum of the second data volume can be obtained from a defined record read data volume table (RECORD_READ_COUNT), which is used to record the data volume of each message belonging to the corresponding topic partition in the current backend storage system when reading messages from each topic partition in the data processing platform. The record read data volume table is shown in Table 2 below:

[0095]

[0096] The table recording the amount of data read includes the following fields: processing thread primary key field, message number field, topic partition name field, amount of data read field, and update time field. The field types corresponding to each field are: String, Integer, String, Long (large integer), and TIMESTAMP. The processing thread primary key field is used to uniquely identify each processing thread; the message number field is used to record the number of each message in the processing thread; the topic partition name field is used to record the name of the topic partition to which the current message belongs; the amount of data read is used to record the amount of data of the current message; and the update time field is used to record the current time.

[0097] For each message belonging to the current time partition of the backend storage system, ie, the second target data, its specific information is recorded in Table 3, and then the data volumes corresponding to all the second target data are added together to obtain the sum of the second data volumes.

[0098] S56, performing core count processing on the messages in the time partition to be core counted according to the sum of the first data volume and the sum of the second data volume, and obtaining the core count result corresponding to the time partition to be core counted.

[0099] Compare whether the sum of the first data volume corresponding to the time partition to be counted is consistent with the sum of the second data volume, and obtain the core count result corresponding to the time partition to be counted. If the core count results are consistent, it means that the data in the time partition to be counted is complete. If the core count results are inconsistent, it means that the data in the time partition to be counted is incomplete.

[0100] Exemplarily, the check result can be obtained from the defined check record table (CHECK_RECORD), which is used to record the specific information of the time partition to be checked during the check process, as well as the sum of the first data volume and the sum of the second data volume obtained according to the time partition to be checked. The check record table is shown in Table 3 below:

[0101]

[0102] The core count record table includes the core count thread primary key field, the core count start time partition field, the core count end time partition field, the first data volume sum field, and the second data volume sum field. The field types corresponding to each field are: String, String, String, Long, and Long. The core count thread primary key field is used to uniquely identify each core count thread; the core count start time partition field is used to indicate the start time partition of the current core count thread core count; the core count end time partition is used to indicate the end time partition of the current core count thread core count; the sum of the first data volume is used to indicate the sum of the data volume of the messages in the waiting core count time partition determined in the backend query system; the sum of the second data volume is used to indicate the sum of the data volume of the messages in the waiting core count time partition corresponding to each topic partition written into the backend storage system. The core count result is obtained by comparing the first data volume sum field and the second data volume sum field. Optionally, after completing this core count, you can return to step S51 for the next core count.

[0103] Through the above steps, first determine at least one candidate time partition among multiple time partitions, then determine whether at least one candidate time partition meets the core count condition, determine the time partition that meets the core count condition as the time partition to be counted, and compare the sum of the first data volume of the partition to be counted obtained from the backend query system and the sum of the second data volume obtained from the corresponding subject partition to obtain the core count result. The scheme of the embodiment of the present application can obtain the sum of the first data volume by recording the data volume size of the target data written to each subject partition, and can obtain the sum of the second data volume by recording the data volume size of the target data of each time partition in the backend query system. The sum of the first data volume and the sum of the second data volume are counted, thereby verifying the integrity and reliability of the data in the data lake.

[0104] Figure 6 A schematic diagram of the structure of a core counting device for copying big data into a lake provided in an embodiment of the present application, such as Figure 6 As shown, the core device 60 for copying the big data into the lake includes:

[0105] The first processing module 61 is used to write multiple messages into corresponding topic partitions according to the source database, wherein for each message, the message includes corresponding target data in the source database and time information, and the time information is used to indicate the time when the message is written into the corresponding topic partition;

[0106] The second processing module 62 is used to write the multiple messages into each time partition in the backend storage system according to the time when the multiple messages are each written into the corresponding topic partition;

[0107] A mapping module 63, used to map the messages written into each time partition in the backend storage system to each time partition in the backend query system;

[0108] The count module 64 is used to perform count processing on multiple messages to obtain count results based on the data volume corresponding to the messages written into each topic partition and the data volume queried in each time partition in the backend query system.

[0109] In a possible implementation, the second processing module 62 is specifically configured to:

[0110] Read at least one message in the topic partition in sequence through the processing thread corresponding to the topic partition, wherein the at least one message is arranged in the order of the time when it is written into the topic partition;

[0111] Determine, according to the time when at least one message is written into the topic partition, the time partition to which each of the at least one message belongs in the backend storage system;

[0112] For each message in the at least one message, the message is written into a time partition to which the message belongs.

[0113] In a possible implementation, the second processing module 62 is specifically configured to:

[0114] Determine whether the moment when the i-th message is written into the topic partition belongs to the time period corresponding to the j-th time partition; the number of time partitions currently included in the backend storage system is j, and j is a positive integer greater than or equal to 1;

[0115] When the time when the i-th message is written into the topic partition belongs to the time period corresponding to the j-th time partition, the j-th time partition is determined as the time partition to which the i-th message belongs;

[0116] When the time when the i-th message is written into the topic partition does not belong to the time period corresponding to the j-th time partition, generate the j+1-th time partition, and determine the j+1-th time partition as the time partition to which the i-th message belongs; wherein the time period corresponding to the j-th time partition is earlier than the time period corresponding to the j+1-th time partition, and the time when the i-th message is written into the topic partition belongs to the time period corresponding to the j+1-th time partition;

[0117] Here, i is 1, 2, ..., M, M is the number of at least one message, and M is a positive integer.

[0118] In a possible implementation, the second processing module 62 is further configured to:

[0119] When the j+1th time partition is generated, the jth time partition is closed.

[0120] In a possible implementation, the core count module 64 is specifically used to:

[0121] Determine a time partition of the number to be verified among multiple time partitions of the backend query system, wherein the time partition of the number to be verified includes at least one first target data;

[0122] Determining the sum of the first data amounts according to the data amount of each of the at least one first target data;

[0123] According to the time period corresponding to the time partition of the number to be verified, at least one second target data is determined in each subject partition, the at least one second target data is the target data included in the target message, and the target message is a message written into the corresponding subject partition and the time period corresponds to the time partition of the number to be verified;

[0124] Determining the sum of the second data amounts according to the data amount of each of the at least one second target data;

[0125] According to the sum of the first data volume and the sum of the second data volume, core count processing is performed on the messages in the time partition to be core counted, and a core count result corresponding to the time partition to be core counted is obtained.

[0126] In a possible implementation, the core count module 64 is specifically used to:

[0127] Obtain configuration information of each of the multiple time partitions, the configuration information includes a core count instruction field, a core count frequency field, and a core count time field, the core count instruction field is used to indicate whether to turn off or on the core count, the core count frequency field is used to indicate the core count frequency, and the core count time field is used to indicate the time when the core count was last performed;

[0128] Determine at least one candidate time partition from the multiple time partitions according to the core number instruction fields of the respective multiple time partitions, wherein the at least one candidate time partition is a time partition for enabling the core number;

[0129] According to the core count frequency of each of the at least one candidate time partitions and the time of the last core count, a time partition to be core counted is determined in the at least one candidate time partition.

[0130] In a possible implementation, the core count module 64 is specifically used to:

[0131] Determine the core count interval duration of at least one candidate time partition, where the core count interval duration is the duration between the last core count time of the candidate time partition and the current time;

[0132] According to the core number frequency and the core number interval duration of each of the at least one candidate time partition, a time partition to be core counted is determined in the at least one candidate time partition.

[0133] In a possible implementation, the core count module 64 is further configured to:

[0134] The core count status field of the waiting count time partition is updated, and the core count status field after the update of the waiting count time partition indicates that the core count status is the counting status.

[0135] The core counting device 60 for copying big data into a lake provided in the embodiment of the present application can execute the technical solution in the above method embodiment, and its implementation principle and beneficial effects are similar, which will not be repeated here.

[0136] Figure 7 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application, such as Figure 7 As shown, it includes a memory 71 and a processor 72. Exemplarily, the memory 71 and the processor 72 are connected to each other via a bus 73. Wherein:

[0137] The memory 71 is used to store computer-executable instructions;

[0138] The processor 72 is configured to execute computer-executable instructions stored in the memory to implement the method in the above embodiment.

[0139] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily required by the present application.

[0140] It should be further noted that, although the various steps in the flow chart are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the flow chart may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these sub-steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of the sub-steps or stages of other steps.

[0141] It should be understood that the above-mentioned device embodiments are only illustrative, and the device of the present application can also be implemented in other ways. For example, the division of units / modules in the above-mentioned embodiments is only a logical function division, and there may be other division methods in actual implementation. For example, multiple units, modules or components can be combined, or can be integrated into another system, or some features can be ignored or not executed.

[0142] In addition, unless otherwise specified, each functional unit / module in each embodiment of the present application may be integrated into one unit / module, each unit / module may exist physically separately, or two or more units / modules may be integrated together. The above-mentioned integrated unit / module may be implemented in the form of hardware or in the form of a software program module.

[0143] If the integrated unit / module is implemented in the form of hardware, the hardware may be a digital circuit, an analog circuit, etc. The physical implementation of the hardware structure includes but is not limited to transistors, memristors, etc. Unless otherwise specified, the processor may be any appropriate hardware processor, such as a CPU, a GPU, an FPGA, a DSP, an ASIC, etc. Unless otherwise specified, the storage unit may be any appropriate magnetic storage medium or magneto-optical storage medium, such as a resistive random access memory (RRAM), a dynamic random access memory (DRAM), a static random access memory (SRAM), an enhanced dynamic random access memory (EDRAM), a high-bandwidth memory (HBM), a hybrid memory cube (HMC), etc.

[0144] If the integrated unit / module is implemented in the form of a software program module and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a memory, including a number of instructions to enable a computer device (which can be a personal computer, server or network device, etc.) to execute all or part of the steps of the various embodiments of the present application. The aforementioned memory includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk and other media that can store program codes.

[0145] In the above embodiments, the description of each embodiment has its own emphasis. For the part not described in detail in a certain embodiment, please refer to the relevant description of other embodiments. The technical features of the above embodiments can be combined arbitrarily. In order to make the description concise, all possible combinations of the technical features in the above embodiments are not described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0146] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present application. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the following claims.

[0147] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A core counting method for copying big data into a lake, characterized in that: The method comprises: According to the source database, a plurality of messages are written into corresponding topic partitions respectively, wherein for each message, the message includes corresponding target data in the source database and time information, and the time information is used to indicate the moment when the message is written into the corresponding topic partition; According to the time when each of the multiple messages is written into the corresponding topic partition, the multiple messages are written into each time partition in the backend storage system respectively; Mapping the messages written into each time partition in the backend storage system to each time partition in the backend query system; According to the data volume corresponding to the messages written into each topic partition and the data volume queried in each time partition in the backend query system, the multiple messages are processed by core count to obtain the core count result.

2. The method according to claim 1, characterized in that For each topic partition, writing the multiple messages into each time partition in the backend storage system according to the time when the multiple messages are each written into the corresponding topic partition includes: Reading at least one message in the topic partition in sequence through a processing thread corresponding to the topic partition, wherein the at least one message is arranged in a chronological order of a time when the message is written into the topic partition; Determine, according to the time when each of the at least one message is written into the topic partition, the time partition to which each of the at least one message belongs in the backend storage system; For each message in the at least one message, the message is written into the time partition to which the message belongs.

3. The method according to claim 2, characterized in that Determining the time partition to which each of the at least one message in the backend storage system belongs according to the time when each of the at least one message is written into the subject partition includes: Determine whether the moment when the i-th message is written into the topic partition belongs to the time period corresponding to the j-th time partition; the number of time partitions currently included in the backend storage system is j, and j is a positive integer greater than or equal to 1; In the case where the moment when the i-th message is written into the topic partition belongs to the time period corresponding to the j-th time partition, the j-th time partition is determined as the time partition to which the i-th message belongs; In the case where the moment when the i-th message is written into the topic partition does not belong to the time period corresponding to the j-th time partition, a j+1-th time partition is generated, and the j+1-th time partition is determined as the time partition to which the i-th message belongs; wherein the time period corresponding to the j-th time partition is earlier than the time period corresponding to the j+1-th time partition, and the moment when the i-th message is written into the topic partition belongs to the time period corresponding to the j+1-th time partition; Here, i is 1, 2, ..., M respectively, M is the number of the at least one message, and M is a positive integer.

4. The method according to claim 3, characterized in that The method further comprises: When the j+1th time partition is generated, the jth time partition is closed.

5. The method according to any one of claims 1 to 4, characterized in that: According to the data volume corresponding to the messages written into each topic partition and the data volume queried by each time partition in the backend query system, the multiple messages are processed to obtain the core count result, including: Determine a time partition of the number to be checked among the multiple time partitions of the backend query system, wherein the time partition of the number to be checked includes at least one first target data; Determining the sum of the first data amounts according to the data amounts of the at least one first target data; According to the time period corresponding to the time partition of the number to be verified, at least one second target data is determined in each of the subject partitions, wherein the at least one second target data is the target data included in the target message, and the target message is a message written into the corresponding subject partition at a time period corresponding to the time partition of the number to be verified; Determining the sum of the second data amounts according to the data amounts of the at least one second target data; According to the sum of the first data volume and the sum of the second data volume, core count processing is performed on the messages in the time partition to be core counted to obtain the core count result corresponding to the time partition to be core counted.

6. The method according to claim 5, characterized in that Determining the time partition of the number of pending checks among the multiple time partitions of the backend query system includes: Acquire configuration information of each of the multiple time partitions, the configuration information including a core count instruction field, a core count frequency field, and a core count time field, the core count instruction field is used to indicate whether to turn off or turn on the core count, the core count frequency field is used to indicate the core count frequency, and the core count time field is used to indicate the time when the core count was last performed; Determine at least one candidate time partition among the multiple time partitions according to the core number instruction fields of each of the multiple time partitions, wherein the at least one candidate time partition is a time partition for turning on the core number; The time partition to be counted is determined in the at least one candidate time partition according to the core count frequency of each of the at least one candidate time partition and the time when the core count was last performed.

7. The method according to claim 6, characterized in that The determining the time partition to be counted of cores in the at least one candidate time partition according to the core count frequency of each of the at least one candidate time partition and the time when the core count was last performed includes: Determine a core count interval duration of each of the at least one candidate time partition, where the core count interval duration is the duration between the last time the core count was performed on the candidate time partition and the current time; The waiting core count time partition is determined in the at least one candidate time partition according to the core count frequency and the core count interval duration of each of the at least one candidate time partition.

8. The method according to claim 6, characterized in that The configuration information further includes a core number status field, where the core number status field is used to indicate the core number status of the corresponding time partition. The method further includes: The core count status field of the waiting core count time partition is updated, and the core count status field of the waiting core count time partition after the update indicates that the core count status is the core count status.

9. A core counting device for copying big data into a lake, characterized in that: The device comprises: A first processing module is used to write multiple messages into corresponding topic partitions according to a source database, wherein for each message, the message includes corresponding target data and time information in the source database, and the time information is used to indicate the time when the message is written into the corresponding topic partition; A second processing module, used to write the multiple messages into each time partition in the backend storage system according to the time when the multiple messages are each written into the corresponding topic partition; A mapping module, used to map the messages written into each time partition in the backend storage system to each time partition in the backend query system; The core count module is used to perform core count processing on the multiple messages to obtain core count results based on the data volume corresponding to the messages written into each topic partition and the data volume queried in each time partition in the backend query system.

10. An electronic device, characterized in that: include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the method according to any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 8 when executed by a processor.

12. A computer program product, characterized in that The invention comprises a computer program, which implements the method according to any one of claims 1 to 8 when being executed by a processor.

Images