Data storage method, device, storage medium and electronic equipment

Abstract

The embodiment of the present invention discloses a data warehousing method, device, storage medium and electronic device, the method comprising: dividing the data in the first topic into batches based on a preset algorithm, and determining the first offset range of each batch of data respectively; writing each first offset range into the second topic respectively; in response to a data warehousing event being triggered, reading data from the second topic based on the consumer, and using the read data as the target offset range; reading target data corresponding to the target offset range from the first topic, and writing the target data into a target file to save the target file in the database; wherein the target file is a file named based on the target offset range. The solution of the embodiment of the present invention can effectively avoid data duplication or loss when generating a database file from consuming data in a topic, thereby ensuring data consistency.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of data processing technology, and in particular to a data warehousing method, device, storage medium and electronic device. Background Art

[0002] Traditional data warehouse systems (such as Hive) typically use formats such as ORC, Parquet, JSON, and TXT to store data on distributed file systems (such as HDFS). To improve data query and analysis performance, it is often necessary to accumulate data to a certain volume before generating a file. A common solution is to buffer data in a distributed message queue (such as Kafka, RocketMQ, and RabbitMQ). A streaming service then consumes data from the distributed message queue. Once the consumed data reaches a certain volume, an archive file is generated and stored on the distributed file system for query or analysis.

[0003] In related technologies, data consuming processes (i.e., consumers) consume data from a distributed message queue topic, batching the data by time or volume. Each batch generates a file, and after file generation is complete, the offset (i.e., the consumer's progress in consuming data) is submitted to the distributed message queue's consumer group. However, if the offset is submitted before the file is generated, there will be an atomicity issue: the offset is submitted, but the file is not saved. If the consumer fails at this time, data loss will occur. Conversely, if the file is generated before the offset is submitted, there will also be certain issues: the file is generated, but the offset is not submitted. If a consumer fails or loses connection, triggering a rebalance, the offset for this batch of data cannot be submitted to the distributed message queue's consumer group. At this time, the file has already been generated. After rebalancing, this batch of data will be consumed by other consuming processes, and new files will be generated, resulting in duplicate data. In data warehouse systems, deduplication of large amounts of data often consumes a large amount of system resources, placing unnecessary performance pressure on the cluster. Summary of the Invention

[0004] The embodiments of the present invention provide a data warehousing method, device, storage medium and electronic device, which can effectively avoid data duplication or loss when consuming data from a subject to generate a database file, thereby ensuring data consistency.

[0005] In a first aspect, an embodiment of the present invention provides a data storage method, comprising:

[0006] Divide the data in the first topic into batches based on a preset algorithm, and determine the first offset range of each batch of data;

[0007] Writing each first offset range into the second topic respectively;

[0008] In response to a data storage event being triggered, reading data from the second topic based on a consumer, and using the read data as a target offset range;

[0009] Target data corresponding to the target offset range is read from the first topic, and the target data is written into a target file to save the target file in a database; wherein the target file is a file named based on the target offset range.

[0010] In a second aspect, an embodiment of the present invention further provides a data storage device, comprising:

[0011] A first offset range determination module is configured to divide the data in the first topic into batches based on a preset algorithm and determine a first offset range for each batch of data;

[0012] A first offset range writing module, configured to write each first offset range into the second topic respectively;

[0013] a target offset range reading module, configured to respond to a data storage event being triggered, read data from the second topic based on a consumer, and use the read data as a target offset range;

[0014] A target data writing module is used to read target data corresponding to the target offset range from the first topic and write the target data into a target file to save the target file in a database; wherein the target file is a file named based on the target offset range.

[0015] In a third aspect, an embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the data warehousing method provided in an embodiment of the present invention.

[0016] In a fourth aspect, an embodiment of the present invention provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the data warehousing method provided in the embodiment of the present invention is implemented.

[0017] An embodiment of the present invention provides a data warehousing solution, which divides the data in the first topic into batches based on a preset algorithm, and determines the first offset range of each batch of data respectively; writes each first offset range into the second topic respectively; in response to a data warehousing event being triggered, reads data from the second topic based on the consumer, and uses the read data as the target offset range; reads the target data corresponding to the target offset range from the first topic, and writes the target data into a target file to save the target file in the database; wherein the target file is a file named based on the target offset range. The technical solution provided by the embodiment of the present invention pre-batches the data to be processed in a certain topic, and inserts the data offset range of each batch of data as a piece of data into another topic respectively, and then asynchronously generates a database file based on the predetermined data batches, which can effectively avoid data duplication or loss when generating a database file from consuming data in a topic, thereby ensuring data consistency. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 This is a flow chart of a data storage method provided by one embodiment of the present invention;

[0019] Figure 2 This is a schematic diagram of the data storage process provided by an embodiment of the present invention;

[0020] Figure 3 is a structural diagram of a data storage device provided by another embodiment of the present invention;

[0021] Figure 4 It is a structural diagram of an electronic device in another embodiment of the present invention. DETAILED DESCRIPTION

[0022] Embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be construed as limited to the embodiments described herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of the present invention. It should be understood that the drawings and embodiments of the present invention are for illustrative purposes only and are not intended to limit the scope of protection of the present invention.

[0023] It should be understood that the various steps described in the method embodiments of the present invention may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present invention is not limited in this respect.

[0024] As used herein, the term "including" and its variations are open-ended, i.e., "including but not limited to." The term "based on" means "based, at least in part, on." The term "one embodiment" means "at least one embodiment," the term "another embodiment" means "at least one additional embodiment," and the term "some embodiments" means "at least some embodiments." Other terms are defined in the following description.

[0025] It should be noted that the concepts of "first" and "second" mentioned in the present invention are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0026] It should be noted that the modifications of "one" and "multiple" mentioned in the present invention are illustrative rather than restrictive. Those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0027] The names of the messages or information exchanged between multiple devices in the embodiments of the present invention are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0028] Figure 1 This is a flow chart of a data storage method provided by an embodiment of the present invention. The embodiment of the present invention is applicable to the case of storing data in a database. The method can be executed by a data storage device, which can be composed of hardware and / or software and can generally be integrated into an electronic device. Figure 1 As shown, the method specifically includes the following steps:

[0029] Step 110 : Divide the data in the first topic into batches based on a preset algorithm, and determine a first offset range for each batch of data.

[0030] A topic is a type of distributed message queue. For example, a distributed messaging system can include producers, consumers, and topics. When a producer writes data, it publishes it to a topic of the corresponding type, and consumers consume data from the topic.

[0031] In an embodiment of the present invention, the data written into the first topic can be divided into batches based on a preset algorithm. Exemplarily, the data in the first topic can be divided according to the writing time of the data, wherein the data written into the first topic in the same time period can be divided into a batch of data, for example, the data written into the first topic in the time period of 8:00-8:05 is regarded as the same batch of data, the data written into the first topic in the time period of 8:05-8:10 is regarded as the same batch of data, the data written into the first topic in the time period of 8:10-8:15 is regarded as the same batch of data, and so on. Exemplarily, the data in the first topic can also be divided according to the amount of data, for example, starting from the first data written into the first topic, the amount of data written into the first topic can be monitored in real time, when the amount of data reaches a preset data amount threshold (such as 1000), the 1-1000 data in the first topic can be divided into the same batch of data, and then the amount of data written into the first topic can be monitored. When the amount of data reaches 2000, the 1001-2000 data in the first topic can be divided into the same batch of data, and so on. It should be noted that the embodiment of the present invention does not limit the batch division algorithm of the data in the first topic.

[0032] The offset is the progress of the consumer consuming data or the progress of the producer producing data. Since the consumer is consuming data from the first topic while the producer is writing data to the first topic, and the speed at which the producer writes data is not necessarily consistent with the speed at which the consumer consumes data, there is a deviation between the amount of data written to the first topic and the amount of data consumed from the first topic, that is, there is a deviation between the progress of the producer producing data and the progress of the consumer consuming data. The offset range can be determined based on the progress of the producer producing data and the progress of the consumer consuming data. In an embodiment of the present invention, the offset range of each batch of data is determined separately, and the offset range of each batch of data is referred to as the first offset range.

[0033] Among them, the first topic may include one partition or multiple partitions. A partition is actually a queue. A topic consists of one or more partitions. The data written to the topic will be distributed to each partition based on a specific strategy. When the first topic contains multiple partitions, the data of each partition in the first topic can be batched based on a preset algorithm, and the first offset range of the same batch of data in each partition of the first topic can be determined separately. It can be understood that when the first topic contains multiple partitions, the first offset range of the same batch of data can include the offset range of each partition in the first topic.

[0034] Step 120: Write each first offset range into the second topic respectively.

[0035] Among them, the second topic can be understood as a message queue for inserting the first offset range. Exemplarily, it can be determined whether there is an idle topic in the distributed messaging system (such as kafka). If so, the idle topic is used as the second topic. Otherwise, a new topic is created in the distributed messaging system as the second topic. In an embodiment of the present invention, the first offset range of the same batch of data in the first topic is written into the second topic as a piece of data. It can be understood that the first offset range of each batch of data in the first topic is each data written to the second topic. Among them, the second topic can contain one partition or multiple partitions. The number of partitions contained in the second topic can be the same as or different from the number of partitions contained in the first topic. The embodiment of the present invention does not limit the number of partitions in the second topic. When the second topic contains multiple partitions, the first offset range of each batch in the first topic can be written into each partition of the second topic based on a pre-set data writing strategy.

[0036] Step 130 : In response to a data warehousing event being triggered, data is read from the second topic based on a consumer, and the read data is used as a target offset range.

[0037] In an embodiment of the present invention, when it is monitored that the consumer of the second topic is successfully created, it can be determined that the data entry event is triggered, wherein the consumer of the second topic can be one or more, and the embodiment of the present invention does not limit the number of consumers of the second topic. Optionally, when a data entry instruction input by the user is detected, it can be determined that the data entry event is triggered. It should be noted that the embodiment of the present invention does not limit the triggering conditions of the data entry event. When it is detected that the data entry event is triggered, data is read from the second topic based on the consumer corresponding to the second topic. It can be understood that the data read from the second topic is one or more of the first offset ranges corresponding to all batch data of the first topic, that is, each first offset range read from the second topic is respectively used as the target offset range. When there are multiple consumers corresponding to the second topic, the data read by each consumer can be respectively used as the target offset range.

[0038] Step 140, read the target data corresponding to the target offset range from the first subject, and write the target data into a target file to save the target file in the database; wherein the target file is a file named based on the target offset range.

[0039] In an embodiment of the present invention, after reading the target offset range from the second topic, the data corresponding to the target offset range is determined from the first topic, and the data corresponding to the target offset range in the first topic is determined as the target data. The target data is then read from the first topic. It is understood that the target data is the data currently to be consumed, that is, the data to be written to the database. The target data is written to a target file named based on the target offset range to ensure the uniqueness of the target data written to the file. Optionally, before writing the target data to the target file, the method further includes: calculating a hash value of the target offset range based on a hash algorithm, and using the hash value as the file name of the target file. Exemplarily, the hash value of the target offset range is calculated based on MD5. For example, if the hash value of a target offset range is 0253475B30A4C600504367FC192D1571, the hash value is used as the file name of the target file, for example: 0253475B30A4C600504367FC192D1571.parquet. It should be noted that the embodiment of the present invention does not limit the specific hash algorithm.

[0040] In an embodiment of the present invention, the target file is stored in a database, that is, the target data is stored in the database in the form of the target file, wherein the database can be a distributed file system. Optionally, the target file is stored in a database, including: judging whether the database has a file with the same file name as the target file, if so, the target file is stored in the database, and the file with the same file name is overwritten based on the target file. Specifically, the target file is written into the database in an overwrite write mode, that is, judging whether the database has already stored a file with the same file name as the target file, if so, the file with the same file name is overwritten based on the target file, that is, the file with the same file name as the target file is deleted, and the target file is stored in the database. The advantage of such an arrangement is that the uniqueness of the files written into the database can be effectively guaranteed, and the duplication of the data written into the database can be effectively avoided.

[0041] Optionally, after writing the target data into the target file, the method further includes: submitting the offset of the consumption data in the second topic to the consumer group corresponding to the second topic. In an embodiment of the present invention, after writing the target data into the target file, the offset of the current consumption data in the second topic is submitted to the corresponding consumer group, so that the consumer group maintains the consumption progress of the current consumption data in the second topic, thereby facilitating consumers to read the currently unconsumed data from the second topic based on the consumption progress of the current consumption data in the second topic, thereby avoiding duplication of data consumption in the second topic.

[0042] An embodiment of the present invention provides a data warehousing method, which divides the data in a first topic into batches based on a preset algorithm, and determines the first offset range of each batch of data respectively; writes each first offset range into a second topic respectively; in response to a data warehousing event being triggered, reads data from the second topic based on the consumer, and uses the read data as the target offset range; reads the target data corresponding to the target offset range from the first topic, and writes the target data into a target file to save the target file in a database; wherein the target file is a file named based on the target offset range. The technical solution provided by an embodiment of the present invention pre-batches the data to be processed in a certain topic, and inserts the data offset range of each batch of data as a piece of data into another topic respectively, and then asynchronously generates a database file based on the predetermined data batches, which can effectively avoid data duplication or loss when generating a database file from consuming data in a topic, thereby ensuring data consistency.

[0043] In some embodiments, the data in the first topic is divided into batches based on a preset algorithm, including: polling at preset time intervals to obtain the production offset of the production data in each partition of the first topic and the consumption offset of the current consumption data in each partition of the first topic; for each partition in the first topic, according to the production offset and the consumption offset of the current partition, determining the offset range of the data to be consumed in the current partition; wherein the starting offset of the offset range is the consumption offset, and the ending offset of the offset range is the production offset; the data corresponding to the offset range of each partition in the first topic is divided into the same batch of data. The advantage of this setting is that the data in the first topic can be accurately batched, which helps to determine the offset range of each batch of data.

[0044] When a producer writes data to a topic, the amount of data written to the topic increases by 1 for each piece of data written by the producer, meaning the production offset increases by 1. It can be understood that the production offset is the number of data items written to the topic by the producer. Furthermore, the production offset increases continuously as data is written, meaning the production offset is a continuously changing quantity. Similarly, each time a consumer consumes a piece of data from a topic, the amount of data consumed increases by 1, meaning the consumption offset increases by 1. It can be understood that the consumption offset is the number of data items consumed by the consumer from the topic. Furthermore, the consumption offset increases continuously as data is consumed, meaning the consumption offset is a continuously changing quantity.

[0045] In an embodiment of the present invention, when the first topic contains multiple partitions, the progress of the producer writing data to each partition in the first topic can be the same or different; similarly, the progress of the consumer consuming data from each partition in the first topic can be the same or different. For example, the production offset of the production data in each partition in the first topic and the consumption offset of the currently consumed data in each partition in the first topic can be obtained by polling at preset intervals through a low-level API (Application Programming Interface). It should be noted that the initial value of the consumption offset of the currently consumed data in each partition in the first topic is 0. The offset range of each partition can be determined based on the production offset and consumption offset of each partition in the first topic obtained by the current polling. Specifically, the starting offset of the offset range of each partition is the consumption offset of the corresponding partition, and the ending offset is the production offset of the corresponding partition, that is, the offset range is [consumption offset, production offset]. It can be understood that the data corresponding to the offset range of each partition is the data to be consumed that has been written to the first topic but has not been consumed by the consumer.

[0046] Exemplarily, the first topic includes three partitions, namely the first partition partition1, the second partition partition2 and the third partition partition3. The production offsets of each partition in the first topic obtained by the first polling are as follows: the production offset of the first partition partition1 is 100, the production offset of the second partition partition2 is 130 and the production offset of the third partition partition3 is 110; the consumption offsets of each partition in the first topic obtained by the first polling are all 0. Then the offset ranges of each partition in the first topic obtained by the first polling are as follows: the offset range of the first partition partition1 is [0, 100], the offset range of the second partition partition2 is [0, 130] and the offset range of the third partition partition3 is [0, 110]. In an embodiment of the invention, the offset ranges of each partition in the first topic obtained by this polling can be recorded as follows:

[0047]

[0048]

[0049] In an embodiment of the present invention, the data corresponding to the offset range of each partition in the first topic obtained by each polling is divided into the same batch of data. It can be understood that the data corresponding to the offset range of each partition in the first topic obtained by the first polling is divided into the first batch of data; the data corresponding to the offset range of each partition in the first topic obtained by the second polling is divided into the second batch of data, and so on, thereby realizing the batch division of the data in the first topic. Optionally, after obtaining the offset range of each partition in the first topic by each polling, the amount of data corresponding to the offset range obtained by this polling can be calculated based on the offset range of each partition, wherein the amount of data is the sum of the amount of data contained in the offset range corresponding to each partition; when the amount of data corresponding to the offset range obtained by this polling is less than the preset data amount threshold, it is possible to wait for a period of time before performing the next polling operation to increase the offset range corresponding to each partition in the first topic obtained by this polling, thereby increasing the amount of data divided into the batch of data by this polling.

[0050] After the data in each partition of the first topic is divided into batches based on the polling method described above, the offset range corresponding to each partition in the first topic obtained in each polling is used as the first offset range for that batch of data in the first topic. It will be understood that the first offset range for the same batch is composed of the offset ranges corresponding to each partition.

[0051] In some embodiments, after the data corresponding to the offset ranges of each partition in the first topic are divided into the same batch of data, the method further includes: updating the consumption offset of the currently consumed data in the corresponding partition based on the end offset of the offset range of each partition in the first topic. In an embodiment of the present invention, after each polling obtains the offset ranges of each partition in the first topic and divides the offset ranges of each partition into the same batch of data, the consumption offset of the currently consumed data in the corresponding partition is updated based on the end offset of the offset range of each partition obtained in the polling. For example, the offset ranges of each partition in the first topic obtained by the first poll are as follows: the offset range of the first partition partition1 is [0, 100], the offset range of the second partition partition2 is [0, 130], and the offset range of the third partition partition3 is [0, 110]. Then, after the data corresponding to the offset ranges of each partition in the first topic are divided into the same batch of data in the first poll, the consumption offset of the current consumption data of each partition in the first topic is updated as follows: the consumption offset of the current consumption data of the first partition partition1 is updated to 100, the consumption offset of the current consumption data of the second partition partition2 is updated to 130, and the consumption offset of the current consumption data of the third partition partition3 is updated to 110. Specifically, after the data corresponding to the offset ranges of each partition in the first topic are divided into the same batch of data, the end offset based on the offset range of each partition in the first topic can be submitted to the consumer group corresponding to the first topic, so that the consumer group can maintain the data consumption progress of the consumer for each partition in the first topic. The consumer group is used to maintain the consumer data consumption progress (i.e., offset). During the consumption process, the consumer will submit the offset to its corresponding consumer group to save its current consumption progress. When multiple consumers use the same group, they share the same group consumption progress.

[0052] In some embodiments, the target offset range includes the second offset range of each partition in the first topic; before reading the target data corresponding to the target offset range from the first topic, it also includes: determining that the target offset range corresponds to each target partition in the first topic and the second offset range of each target partition; reading the target data corresponding to the target offset range from the first topic includes: reading the target data corresponding to the second offset range from each target partition in the first topic. In an embodiment of the present invention, when the first topic contains multiple partitions, each data written to the second topic (that is, the first offset range of each batch of data in the first topic) is composed of the offset range of each partition in the first topic. Therefore, the data read from the second topic (that is, the target offset range) also includes the offset range of each partition in the first topic. For the convenience of description, the offset range of each partition in the first topic included in the target offset range is called the second offset range. In an embodiment of the present invention, each target partition in the first topic corresponding to the target offset range and a second offset range of the target partition are determined, wherein the target partition can be understood as the partition in the first topic where the data in the first topic corresponding to the target offset range is located, and the second offset range can be understood as the offset range corresponding to the data in the first topic corresponding to the target offset range in each target partition of the first topic. When reading the target data corresponding to the target offset range from the first topic, the target data corresponding to the second offset range can be directly read from each target partition in the first topic respectively.

[0053] Figure 2 The following is a schematic diagram of the data storage process provided by the embodiment of the present invention. Figure 2 As shown, the first topic contains three partitions (the first partition, the second partition, and the third partition). After the data of each partition in the first topic is divided into batches, the first offset range of each same batch of data (including the offset range of each partition in the first topic) is written into each partition of the second topic. Among them, the second topic also contains three partitions (the first partition, the second partition, and the third partition). The data warehousing service contains three service nodes (that is, three consumers). The three nodes read data from each partition of the second topic respectively, and pull data from the first topic respectively according to the target offset range of the read data (including the corresponding target partition in the first topic and the second offset range of the target partition), and generate corresponding target files for the same batch of data pulled from the first topic and save them in the distributed file system.

[0054] Figure 3 This is a structural diagram of a data storage device provided by another embodiment of the present invention. Figure 3As shown, the device includes: a first offset range determining module 310, a first offset range writing module 320, a target offset range reading module 330 and a target data writing module 340.

[0055] A first offset range determination module 310 is configured to divide the data in the first topic into batches based on a preset algorithm and determine a first offset range for each batch of data;

[0056] A first offset range writing module 320 is configured to write each first offset range into the second topic;

[0057] a target offset range reading module 330 for reading data from the second topic based on a consumer in response to a data storage event being triggered, and using the read data as a target offset range;

[0058] The target data writing module 340 is used to read the target data corresponding to the target offset range from the first topic and write the target data into a target file to save the target file in the database; wherein the target file is a file named based on the target offset range.

[0059] An embodiment of the present invention provides a data warehousing device, which divides the data in a first topic into batches based on a preset algorithm, and determines the first offset range of each batch of data respectively; writes each first offset range into a second topic respectively; in response to a data warehousing event being triggered, reads data from the second topic based on a consumer, and uses the read data as a target offset range; reads target data corresponding to the target offset range from the first topic, and writes the target data into a target file to save the target file in a database; wherein the target file is a file named based on the target offset range. The technical solution provided by an embodiment of the present invention pre-batches the data to be processed in a certain topic, and inserts the data offset range of each batch of data as a piece of data into another topic respectively, and then asynchronously generates a database file based on the predetermined data batches, which can effectively avoid data duplication or loss when generating a database file from consuming data in a topic, thereby ensuring data consistency.

[0060] Optionally, the first offset range determining module is configured to:

[0061] Polling every preset time period to obtain the production offset of production data in each partition of the first topic and the consumption offset of current consumption data in each partition of the first topic;

[0062] For each partition in the first topic, determine an offset range of the data to be consumed in the current partition based on the production offset and the consumption offset of the current partition; wherein the starting offset of the offset range is the consumption offset, and the ending offset of the offset range is the production offset;

[0063] The data corresponding to the offset range of each partition in the first topic are divided into the same batch of data.

[0064] Optionally, the device further includes:

[0065] The consumption offset update module is used to update the consumption offset of the currently consumed data in the corresponding partition based on the end offset of the offset range of each partition in the first topic after dividing the data corresponding to the offset range of each partition in the first topic into the same batch of data.

[0066] Optionally, the target offset range includes the second offset range of each partition in the first topic;

[0067] The device further comprises:

[0068] A second offset range determining module is configured to determine, before reading target data corresponding to the target offset range from the first topic, each target partition in the first topic corresponding to the target offset range and a second offset range of each target partition;

[0069] The target data writing module is used to:

[0070] Target data corresponding to the second offset range is read from each target partition in the first topic.

[0071] Optionally, also include:

[0072] The file name determination module is used to calculate a hash value of the target offset range based on a hash algorithm before writing the target data into the target file, and use the hash value as the file name of the target file.

[0073] Optionally, the target data writing module is used to:

[0074] Determine whether there is a file with the same file name as the target file in the database. If so, save the target file in the database and overwrite the file with the same file name based on the target file.

[0075] Optionally, also include:

[0076] An offset recording module is used to submit the offset of the consumption data in the second topic to the consumer group corresponding to the second topic after writing the target data into the target file.

[0077] The above device can execute the methods provided by all the above embodiments of the present invention, and has the corresponding functional modules and beneficial effects of executing the above methods. For technical details not fully described in the embodiments of the present invention, please refer to the methods provided by all the above embodiments of the present invention.

[0078] An embodiment of the present invention further provides a storage medium containing computer-executable instructions, which, when executed by a computer processor, are used to execute the data warehousing method provided by an embodiment of the present invention.

[0079] Storage medium - any of various types of memory devices or storage devices. The term "storage medium" is intended to include: installation media, such as CD-ROMs, floppy disks, or tape drives; computer system memory or random access memory, such as DRAM, DDRRAM, SRAM, EDORAM, Rambus RAM, etc.; non-volatile memory, such as flash memory, magnetic media (such as hard disks or optical storage); registers or other similar types of memory elements, etc. Storage media may also include other types of memory or combinations thereof. In addition, the storage medium may be located in the first computer system in which the program is executed, or may be located in a different second computer system that is connected to the first computer system via a network (such as the Internet). The second computer system can provide program instructions to the first computer for execution. The term "storage medium" may include two or more storage media that can reside in different locations (e.g., in different computer systems connected via a network). The storage medium can store program instructions (e.g., embodied as a computer program) that can be executed by one or more processors.

[0080] Of course, the computer executable instructions of the storage medium provided by the embodiment of the present invention are not limited to the data storage operations described above, but can also execute related operations in the data storage method provided by any embodiment of the present invention.

[0081] An embodiment of the present invention provides an electronic device, into which the data storage device provided by the embodiment of the present invention can be integrated. Figure 4 This is a block diagram of an electronic device according to an embodiment of the present invention. The electronic device 400 may include a memory 401, a processor 402, and a computer program stored in the memory 401 and executable by the processor. When the processor 402 executes the computer program, the data storage method according to an embodiment of the present invention is implemented.

[0082] The electronic device provided in the embodiment of the present invention divides the data in the first topic into batches based on a preset algorithm, and determines the first offset range of each batch of data respectively; writes each first offset range into the second topic respectively; in response to the data warehousing event being triggered, reads data from the second topic based on the consumer, and uses the read data as the target offset range; reads the target data corresponding to the target offset range from the first topic, and writes the target data into a target file to save the target file in the database; wherein the target file is a file named based on the target offset range. The technical solution provided in the embodiment of the present invention pre-batches the data to be processed in a certain topic, and inserts the data offset range of each batch of data as a piece of data into another topic respectively, and then asynchronously generates a database file based on the predetermined data batches, which can effectively avoid data duplication or loss when generating a database file from consuming data in the topic, thereby ensuring data consistency.

[0083] The data storage devices, storage media, and electronic devices provided in the above embodiments can execute the data storage methods provided in any embodiment of the present invention, and have the corresponding functional modules and beneficial effects of executing the methods. For technical details not fully described in the above embodiments, please refer to the data storage methods provided in any embodiment of the present invention.

[0084] Note that the above are only preferred embodiments of the present invention and the technical principles employed. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and that various obvious changes, readjustments, and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in detail through the above embodiments, the present invention is not limited to the above embodiments and may include many other equivalent embodiments without departing from the concept of the present invention. The scope of the present invention is determined by the scope of the appended claims.

Claims

1. A data storage method, characterized in that: include: Divide the data in the first topic into batches based on a preset algorithm, and determine the first offset range of each batch of data; Writing each first offset range into the second topic respectively; In response to a data storage event being triggered, reading data from the second topic based on a consumer, and using the read data as a target offset range; Target data corresponding to the target offset range is read from the first topic, and the target data is written into a target file to save the target file in a database; wherein the target file is a file named based on the target offset range.

2. The method according to claim 1, characterized in that The data in the first topic is divided into batches based on a preset algorithm, including: Polling every preset time period to obtain the production offset of production data in each partition of the first topic and the consumption offset of current consumption data in each partition of the first topic; For each partition in the first topic, determine an offset range of the data to be consumed in the current partition based on the production offset and the consumption offset of the current partition; wherein the starting offset of the offset range is the consumption offset, and the ending offset of the offset range is the production offset; The data corresponding to the offset range of each partition in the first topic are divided into the same batch of data.

3. The method according to claim 2, characterized in that After dividing the data corresponding to the offset range of each partition in the first topic into the same batch of data, the method further includes: Based on the end offset of the offset range of each partition in the first topic, the consumption offset of the currently consumed data in the corresponding partition is updated.

4. The method according to claim 2, characterized in that The target offset range includes a second offset range for each partition in the first topic; Before reading the target data corresponding to the target offset range from the first topic, the method further includes: Determine the target offset range corresponding to each target partition in the first topic and a second offset range of each target partition; Reading target data corresponding to the target offset range from the first topic includes: Target data corresponding to the second offset range is read from each target partition in the first topic.

5. The method according to claim 1, wherein Before writing the target data into the target file, the method further includes: A hash value of the target offset range is calculated based on a hash algorithm, and the hash value is used as the file name of the target file.

6. The method according to claim 1, wherein Saving the target file in a database includes: Determine whether there is a file with the same file name as the target file in the database. If so, save the target file in the database and overwrite the file with the same file name based on the target file.

7. The method according to claim 1, characterized in that After writing the target data into the target file, the method further includes: Submit the offset of the consumed data in the second topic to the consumer group corresponding to the second topic.

8. A data storage device, characterized in that: include: A first offset range determination module is configured to divide the data in the first topic into batches based on a preset algorithm and determine a first offset range for each batch of data; A first offset range writing module, configured to write each first offset range into the second topic respectively; a target offset range reading module, configured to respond to a data storage event being triggered, read data from the second topic based on a consumer, and use the read data as a target offset range; A target data writing module is used to read target data corresponding to the target offset range from the first topic and write the target data into a target file to save the target file in a database; wherein the target file is a file named based on the target offset range.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processing device, the data storage method as described in any one of claims 1 to 7 is implemented.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable by the processor, wherein: When the processor executes the computer program, the data warehousing method as described in any one of claims 1-7 is implemented.

Images