Cold data migration method, acquisition method, device, medium and electronic equipment

Abstract

The present disclosure relates to a cold data migration method, acquisition method, device, medium, and electronic device, and relates to the field of big data processing technology. The method includes: obtaining a data directory of message data in a message queue, and determining the data generation time and message offset of the message data based on the data directory; determining whether the message data is cold data based on the data generation time; when it is determined that the message data is cold data, obtaining a data file corresponding to the message data from a local disk where the message queue is located based on the message offset; and migrating the data file corresponding to the message data from the local disk to a data storage bucket set in an object storage service. The present disclosure reduces the storage burden of the local disk.

Description

Technical Field

[0001] The embodiments of the present disclosure relate to the field of big data processing technology, and more specifically, to a cold data migration method, a cold data migration device, a cold data acquisition method, a cold data acquisition device, a computer-readable storage medium, and an electronic device. Background Art

[0002] In existing solutions, message data is stored in the message queue, while the corresponding data files are stored on the local disk. However, due to the limited storage space of the local disk, cold data files in the storage space need to be deleted, which in turn destroys the integrity of the data files.

[0003] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute prior art known to ordinary technicians in this field. Summary of the Invention

[0004] The purpose of the present disclosure is to provide a cold data migration method, a cold data migration device, a cold data acquisition method, a cold data acquisition device, a computer-readable storage medium, and an electronic device, thereby overcoming, at least to a certain extent, the problem of incomplete data files caused by the limitations and defects of related technologies.

[0005] According to one aspect of the present disclosure, a cold data migration method is provided, comprising:

[0006] Obtaining a data directory of message data in a message queue, and determining a data generation time of the message data and a message offset of the message data according to the data directory;

[0007] Determining whether the message data is cold data according to the data generation time;

[0008] When it is determined that the message data is cold data, obtaining a data file corresponding to the message data from a local disk where the message queue is located based on the message offset;

[0009] Migrate the data file corresponding to the message data from the local disk to a data storage bucket set in the object storage service.

[0010] In an exemplary embodiment of the present disclosure, determining whether the message data is cold data according to the data generation time includes:

[0011] Obtaining a current time node in the message queue, and calculating a time difference between the current time node and the data generation time;

[0012] It is determined whether the time difference is greater than a preset time threshold, and when it is determined that the time difference is greater than or equal to the preset time threshold, it is determined that the message data is cold data.

[0013] In an exemplary embodiment of the present disclosure, the message offset includes a start message offset and an end message offset;

[0014] Wherein, obtaining a data file corresponding to the message data from the local disk where the message queue is located based on the message offset includes:

[0015] Determining a file name of a data file associated with the message data based on the start message offset and the end message offset;

[0016] A data file corresponding to the message data in the message queue is obtained from a local disk where the message data is located based on the file name.

[0017] In an exemplary embodiment of the present disclosure, migrating a data file corresponding to the message data from the local disk to a data storage bucket set in an object storage service includes:

[0018] Configuring an interface address and an access key for the object storage service, and generating a data packet to be migrated based on the access key and a data file corresponding to the message data;

[0019] Sending the data package to be migrated to the object storage service, where the object storage service verifies the legitimacy of the data package to be migrated based on the access key in the data package to be migrated;

[0020] When the object storage service determines that the validity verification of the data packet to be migrated passes, the object storage service migrates the data file corresponding to the message data in the data packet to be migrated from the local disk to a data storage bucket set in the object storage service.

[0021] In an exemplary embodiment of the present disclosure, the cold data migration method further includes:

[0022] When the object storage service detects that the migration of the data file corresponding to the message data is complete, the object storage service generates an object unique identifier corresponding to the message data according to the bucket name and the message topic to which the message data belongs;

[0023] A mapping relationship between the object unique identifier and the data directory of the message data is established, and the object unique identifier is stored in a preset metadata database based on the mapping relationship.

[0024] In an exemplary embodiment of the present disclosure, the data directory is generated in the following manner:

[0025] Detecting whether there is new message data in the message queue, and when detecting that there is new message data in the message queue, obtaining the data generation time of the new message data and the message topic to which the new message data belongs;

[0026] Determining the number of data files of the newly added message data under the message topic according to the data volume of the newly added message data, and determining the starting message offset and the ending message offset of the newly added message data according to the number of data files;

[0027] A data directory of the newly added data message is generated according to the data generation time, the start message offset, and the end message offset, and the data directory is written into a preset metadata database.

[0028] In an exemplary embodiment of the present disclosure, obtaining a data directory of message data in a message queue includes:

[0029] Configuring a database address for the preset metadata database, and establishing a communication link with the preset metadata database based on the database address;

[0030] A data directory of the message data in the message queue is obtained from the preset metadata database based on the communication link.

[0031] According to one aspect of the present disclosure, a method for acquiring cold data is provided, comprising:

[0032] In response to a cold data acquisition request sent by a business acquisition party, determining a start query time of the data to be queried, and matching a data generation time of the data to be queried corresponding to the start query time in a preset metadata database based on the start query time;

[0033] Extracting a data directory including data generation time from the preset metadata database, constructing a data set of the data to be queried according to the data directory, and determining a start message offset and an end message offset of the data to be queried based on the data directory;

[0034] Sort the data to be queried in the data set based on the data generation time, the start message offset, and the end message offset to determine the start consumption position of the data to be queried;

[0035] Based on the starting consumption position, data files of the data to be queried are obtained in sequence from the data storage bucket set in the object storage service, and the data files of the data to be queried are loaded to the local disk for consumption by the business acquirer; wherein, the data files are stored in the data storage bucket by any of the cold data migration methods described above.

[0036] In an exemplary embodiment of the present disclosure, based on the starting consumption position, sequentially obtaining data files of the data to be queried from data storage buckets set in the object storage service includes:

[0037] Based on the starting consumption position, traverse the preset metadata database in sequence, and determine the object unique identifier of the data to be queried in the data storage bucket set in the object storage service based on the data directory of the data to be queried;

[0038] Based on the interface address and access key of the object storage service, the data file of the data to be queried is obtained from the data storage bucket corresponding to the object unique identifier.

[0039] In an exemplary embodiment of the present disclosure, loading the data file of the to-be-queried data into a local disk for consumption by the business acquirer includes:

[0040] The data file of the data to be queried is copied to the local disk where the message queue is located, and the data to be queried is loaded into the message queue, so that the service acquirer reads the data file of the data to be queried from the message queue.

[0041] According to one aspect of the present disclosure, a cold data migration device is provided, comprising:

[0042] A message offset determination module is used to obtain a data directory of message data in a message queue, and determine a data generation time of the message data and a message offset of the message data according to the data directory;

[0043] A cold data determination module, configured to determine whether the message data is cold data according to the data generation time;

[0044] a data file acquisition module, configured to acquire, when determining that the message data is cold data, a data file corresponding to the message data from a local disk where the message queue is located based on the message offset;

[0045] The data file migration module is used to migrate the data file corresponding to the message data from the local disk to a data storage bucket set in the object storage service.

[0046] According to one aspect of the present disclosure, a device for acquiring cold data is provided, comprising:

[0047] a data generation time matching module, configured to determine, in response to a cold data acquisition request sent by a business acquisition party, a start query time of the data to be queried, and match, based on the start query time, a data generation time of the data to be queried corresponding to the start query time in a preset metadata database;

[0048] a data set construction module, configured to extract a data directory including data generation time from the preset metadata database, construct a data set of the data to be queried based on the data directory, and determine a start message offset and an end message offset of the data to be queried based on the data directory;

[0049] a starting consumption position determining module, configured to sort the data to be queried in the data set based on the data generation time, the starting message offset, and the ending message offset, so as to determine the starting consumption position of the data to be queried;

[0050] A data file loading module is used to obtain the data files of the data to be queried from the data storage bucket set in the object storage service in sequence based on the starting consumption position, and load the data files of the data to be queried to the local disk for consumption by the business acquirer; wherein, the data files are stored in the data storage bucket through any of the cold data migration methods described above.

[0051] According to one aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the cold data migration method described in any of the above-mentioned example embodiments and the cold data acquisition method described in any of the above-mentioned example embodiments are implemented.

[0052] According to one aspect of the present disclosure, there is provided an electronic device, including:

[0053] processor; and

[0054] a memory for storing executable instructions of the processor;

[0055] The processor is configured to execute the cold data migration method described in any one of the above-mentioned example embodiments and the cold data acquisition method described in any one of the above-mentioned example embodiments by executing the executable instructions.

[0056] The embodiment of the present disclosure provides a cold data migration method. On the one hand, the method obtains the data directory of the message data in the message queue, and determines the data generation time of the message data and the message offset of the message data according to the data directory; then determines whether the message data is cold data according to the data generation time; and when it is determined that the message data is cold data, obtains the data file corresponding to the message data from the local disk where the message queue is located based on the message offset; finally, migrates the data file corresponding to the message data from the local disk to the data storage bucket set in the object storage service. Since the cold data can be migrated to the data storage bucket, the data stored in the local disk are all hot data, thereby solving the problem in the prior art of needing to delete the cold data files in the storage space, which destroys the integrity of the data files. On the other hand, since cold data can be migrated to the data storage bucket, it can be ensured that the data stored in the local disk is hot data, thereby greatly reducing the storage burden of the local disk, and solving the problem in the prior art that the message data itself is stored in the local disk, thereby making the local disk burden heavy; on the other hand, since the data file corresponding to the message data is obtained from the local disk where the message queue is located based on the message offset, and the data file corresponding to the message data is migrated from the local disk to the data storage bucket set in the object storage service, the data files under the same message topic are merged and stored, and all data files under the message topic can be obtained from the same data storage bucket during the data query process, thereby improving the data query efficiency.

[0057] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] The accompanying drawings are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification, are used to explain the principles of the present disclosure. Obviously, the drawings described below are only some embodiments of the present disclosure, and those skilled in the art can derive other drawings based on these drawings without inventive effort.

[0059] Figure 1 A flowchart of a cold data migration method according to an exemplary embodiment of the present disclosure is schematically shown.

[0060] Figure 2 A diagram schematically illustrates an example structure of a cold data migration system according to an example embodiment of the present disclosure.

[0061] Figure 3 The following schematically shows an example diagram of the structure of a Broker server according to an example embodiment of the present disclosure.

[0062] Figure 4 A scenario diagram of a message offset according to an exemplary embodiment of the present disclosure is schematically shown.

[0063] Figure 5 The flowchart schematically shows a process of generating a data directory of message data according to an exemplary embodiment of the present disclosure.

[0064] Figure 6 A flowchart schematically illustrates a method for acquiring cold data according to an exemplary embodiment of the present disclosure.

[0065] Figure 7 A diagram schematically illustrates an example scenario of storing or acquiring message data according to an example embodiment of the present disclosure.

[0066] Figure 8 A block diagram schematically illustrates a cold data migration apparatus according to an exemplary embodiment of the present disclosure.

[0067] Figure 9 A block diagram schematically illustrates a device for acquiring cold data according to an exemplary embodiment of the present disclosure.

[0068] Figure 10 An electronic device schematically illustrates a cold data migration method and a cold data acquisition method according to an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0069] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that the present disclosure will be more comprehensive and complete and will fully convey the concepts of the example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure may be practiced while omitting one or more of the specific details, or that other methods, components, devices, steps, etc. may be employed. In other cases, well-known technical solutions are not shown or described in detail to avoid obscuring various aspects of the present disclosure.

[0070] In addition, the accompanying drawings are merely schematic illustrations of the present disclosure and are not necessarily drawn to scale. Identical reference numerals in the figures denote identical or similar parts, and thus repetitive descriptions thereof will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities that do not necessarily correspond to physically or logically separate entities. These functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0071] In existing message data storage implementations, message data of message queue software is stored on the local hard disk of the Broker server because local hard disks have higher performance and lower latency. At the same time, some public cloud vendors also provide remote cloud hard disk services, which map hard disks on other servers to the current server via the network. From the perspective of the current server, these cloud hard disks look like local hard disks.

[0072] However, both local hard drives and remote cloud hard drives have high costs and limited storage capacity. Therefore, popular message queue software currently on the market has a message retention time (or storage capacity) limit. Taking Kafka software as an example, it supports setting an independent retention time for each topic, and messages exceeding this time are automatically deleted by the system. Under this premise, if the business needs to store data in the message queue for a long time so that the old data can be consumed at any time, the capacity or number of hard drives on the broker server needs to be increased. However, in many cases, this is difficult to achieve (that is, it is impossible to increase the capacity or number of hard drives on the server indefinitely) because the required hard drive accessories may not be available in the enterprise computer room environment, or the server's own hard drive slots are already full. If business needs are to be fully met, a large-scale cluster must be built at a high initial cost, which greatly increases the cost of cluster construction. Therefore, a solution is needed to achieve long-term storage of message data without increasing the capacity or number of hard drives on the server or building a large-scale cluster.

[0073] In addition, the special feature of message queues is that once message data is generated, it can be considered as immutable read-only data; and newly generated message data will only be appended to the end of the queue, and will not disrupt the order of historical message data; at the same time, in actual application, message receivers usually start from a certain position in the message queue and receive messages sequentially towards the end of the queue. This access pattern is highly predictable: that is, only a small section of continuous data needs to be read within a period of time, and there is generally no random access to any position. Based on this, the example embodiment of the present disclosure first provides a cold data migration method, which provides a low-cost message queue data storage mechanism, which can be implemented based on a public cloud object storage service and can provide users with nearly unlimited storage capacity.

[0074] In an exemplary embodiment, the cold data migration method can be run on a server, server cluster or cloud server; of course, those skilled in the art can also run the method disclosed herein on other platforms as needed, and this exemplary embodiment does not specifically limit this. Figure 1 As shown, the cold data migration method may include the following steps:

[0075] Step S110. Obtain a data directory of the message data in the message queue, and determine the data generation time of the message data and the message offset of the message data according to the data directory;

[0076] Step S120: Determine whether the message data is cold data based on the data generation time;

[0077] Step S130. When it is determined that the message data is cold data, obtaining a data file corresponding to the message data from the local disk where the message queue is located based on the message offset;

[0078] Step S140: Migrate the data file corresponding to the message data from the local disk to a data storage bucket set in the object storage service.

[0079] In the above-mentioned cold data migration method, on the one hand, the data directory of the message data in the message queue is obtained, and the data generation time of the message data and the message offset of the message data are determined according to the data directory; then, whether the message data is cold data is determined according to the data generation time; and when it is determined that the message data is cold data, the data file corresponding to the message data is obtained from the local disk where the message queue is located based on the message offset; finally, the data file corresponding to the message data is migrated from the local disk to the data storage bucket set in the object storage service. Since the cold data can be migrated to the data storage bucket, the data stored in the local disk is all hot data, thereby solving the problem in the prior art of needing to delete the cold data files in the storage space, thereby destroying the integrity of the data files; On the other hand, since cold data can be migrated to the data storage bucket, it can be ensured that the data stored in the local disk is hot data, thereby greatly reducing the storage burden of the local disk, and solving the problem in the existing technology that the message data itself is stored in the local disk, which makes the local disk burden heavy; on the other hand, since the data file corresponding to the message data is obtained from the local disk where the message queue is located based on the message offset, and the data file corresponding to the message data is migrated from the local disk to the data storage bucket set in the object storage service, the data files under the same message topic are merged and stored, so that all data files under the message topic can be obtained from the same data storage bucket during the data query process, thereby improving data query efficiency.

[0080] Hereinafter, the cold data migration method described in the exemplary embodiments of the present disclosure will be explained and illustrated in detail with reference to the accompanying drawings.

[0081] First, the terms involved in the exemplary embodiments of the present disclosure are explained and illustrated.

[0082] Object storage: Object storage can actually include two parts, objects and object storage; objects can be understood as data files in unstructured formats; object storage can be understood as a technology for storing and managing data files in unstructured formats. Specifically, in actual applications, in order to implement corresponding businesses, it is necessary to create and analyze a large number of data files in unstructured formats. Such unstructured data files may include, for example, photos, videos, emails, web pages, sensor data, and audio files, etc. In this scenario, in order to store and manage unstructured data files, it is necessary to implement it based on object storage. At the same time, in the object storage scenario, data files can be stored in a cloud object storage system, and then these data files can be distributed on multiple physical devices through the cloud object storage system.

[0083] In an example embodiment, in order to store data files in a cloud object storage system, it is necessary to introduce the concept of metadata corresponding to the data files. Metadata is crucial to the implementation of object storage technology; that is, metadata is a prerequisite for the implementation of object storage technology. In actual application, using object storage, data files can be stored in a single bucket, rather than storing data files as files in a folder. In addition, object storage can also integrate the data fragments that constitute the data file, add all user-created metadata to the data file, and attach a custom identifier. In order to implement the storage of the object, a flat structure called a bucket can be created to implement object storage, rather than using a hierarchical storage method or a graded storage method. In addition, any data file of any file type in the bucket can be retrieved and analyzed based on the functions and characteristics of the bucket.

[0084] In an example embodiment, object storage provides an architecture for a large number of data files, each of which can be stored as an object, and the object corresponding to the metadata also provides a unique identifier for easy access. Therefore, this architecture eliminates the scalability limitations of traditional storage, which is why object storage is cloud storage. In addition, since object storage can be implemented on multiple devices across systems and / or across data centers and / or across data regions, object storage can achieve almost unlimited scale expansion while providing higher data persistence, elasticity and availability. Furthermore, object storage also has the advantage of low cost, especially since various manufacturers have launched tiered object storage, and the cost and access latency of each level vary greatly. For example, the cost of "archive-type" storage is generally 1 / 3 of that of conventional object storage, while the latency is in minutes.

[0085] Message Queues: A message queue (MQ) is a communication method between different processes or between different threads of the same process. In practice, the role that sends messages to a message queue is called a message sender (or message producer); conversely, the role that retrieves messages from a message queue is called a message receiver (or message consumer). Furthermore, message queues are implemented asynchronously, allowing message receivers to retrieve messages long after they have been sent, which differs from most communication protocols. For example, HTTP (Hypertext Transfer Protocol), the most commonly used Internet protocol, is synchronous, meaning that the client must wait for the server to respond after sending a request. However, asynchronous communication protocols are often required, such as when one process notifies another of an event without waiting for a response. However, the asynchronous nature of message queues also has drawbacks: the receiver must poll the message queue to receive the latest message. Furthermore, in addition to serving as a buffer between different threads or processes, message queues can also be used to detect performance issues in the receiving thread or process by counting the number of messages currently in the message queue.

[0086] In an example embodiment, software with a message queue is usually run in a cluster manner. Currently, the most common software with a message queue may include MQ in a Kafka cluster and MQ in a Rabbit cluster, etc., which are all open source models and can be used, modified and redistributed by anyone for free; at the same time, each server in the cluster can be called a Broker, which can be responsible for receiving messages from the message sender and storing the messages sent by the message sender on its own hard disk; and then delivering these messages to the receiver according to the instructions of the message receiver; further, each cluster can support multiple message topics (called Topics), and each message can and can only belong to one of them.

[0087] Secondly, the migration system of cold data involved in the example embodiment of the present disclosure is explained and illustrated. Specifically, the migration system of cold data involved in the example embodiment of the present disclosure may include a producer 210 of message data, a Broker server 220, a metadata repository 230, and an object storage service 240. Among them, the Broker server is connected to the producer of message data, the metadata repository, and the object storage service in a wired network or a wireless network; in actual application, the producer of message data is used to produce message data, and the Broker server is used to implement the migration method of cold data recorded in the example embodiment of the present disclosure, and of course, the cold data acquisition method recorded in the example embodiment of the present disclosure can also be implemented; the metadata repository is used to store the data directory of message data, and the object storage server is used to store cold data.

[0088] In an exemplary embodiment, referring to Figure 3 As shown, the Broker server described above is provided with a message queue 301 and a message data migration component 302; wherein, the message queue is used to receive message data sent by the producer of the message data, and generate a data directory corresponding to the message data, and then store the data directory in the metadata database; the message data migration component user migrates the cold data in the message queue to the data storage bucket in the object storage service. Further, continue to refer to Figure 3 As shown, the object storage service described above is provided with a plurality of data storage buckets 303, which are used to store cold data.

[0089] It should be noted here that when the message queue receives message data sent by the producer of message data, it will automatically save the data file of the message data to the local disk. Specifically, taking Kafka software as an example, it is in 1GB size units by default; for example, under each message topic, each 1GB of message data generated is written to a new independent file (data file); at the same time, the file name of the data file is composed of the starting message offset and the ending message offset; among them, the offsets in the starting message offset and the ending message offset recorded here are defined as follows: the offset of the first data file under the message topic is 0, the offset of the second data file is 1, and so on; that is, assuming that the message topic T includes 5 data files, the name of the first data file (file 1) is : Topic T+0_A; the name of the second data file (file 2) is: Topic T+A_B; the name of the third data file (file 3) is: Topic T+B_C; the name of the fourth data file (file 4) is: Topic T+C_D; the name of the fifth data file (file 5) is: Topic T+D_E; among them, A, B, C, D, E in the name are all message offsets, offset A-offset B is the number of messages included in file 1, that is, the number of messages in the same data file; based on this, it can be inferred that the starting message offset of the message data is 0 and the ending message offset is E; among them, the specific scenario example diagram can be referred to. Figure 4 As shown; from this we can know that the offset is a monotonically increasing number.

[0090] However, since the length of each message data is different, the number of messages in each data file must also be different (but the total size of each data file is basically the same); further, files with a creation time (which can also be understood as data generation time) greater than or equal to the maximum message retention time of the message queue software are defined as cold data; and files with a time less than the maximum message retention time are defined as hot data; for example, in Kafka software, the maximum message retention time defaults to 7 days; from this, it can be seen that under the default configuration, message data older than 7 days can be considered cold data, and message data no older than 7 days can be considered hot data; at that time, the maximum message retention time involved here can be determined by yourself according to actual needs, and this example does not impose any special restrictions on this.

[0091] The following will be combined Figure 5 The specific generation process of the data directory involved in the exemplary embodiment of the present disclosure is explained and illustrated. Figure 5 As shown, the process of generating the data directory may include the following steps:

[0092] Step S510: Detect whether there is new message data in the message queue, and when it is detected that there is new message data in the message queue, obtain the data generation time of the new message data and the message topic to which the new message data belongs;

[0093] Step S520: determining the number of data files of the newly added message data under the message topic according to the data volume of the newly added message data, and determining the starting message offset and the ending message offset of the newly added message data according to the number of data files;

[0094] Step S530: Generate a data directory of the newly added data message according to the data generation time, the start message offset, and the end message offset, and write the data directory into a preset metadata database.

[0095] Hereinafter, steps S510 to S530 will be explained and illustrated. Specifically, in actual application, it is possible to continuously detect whether there is new message data in the message queue; and when the presence of new message data in the message queue is detected, the data generation time and data volume of the new message data can be obtained from the data attributes of the new message data, and then the corresponding number of data files is determined based on the data volume (for example, it can be divided into 1G units, and the number of data files is determined by the number of GB included). Of course, the specific number of data files can also be directly obtained from the local disk, and this example does not impose special restrictions on this. After obtaining the number of data files, the starting message offset and the ending message offset can be determined. Finally, based on the data generation time, the starting message offset and the ending message offset, the corresponding data directory can be obtained. The generated data directory can be, for example: message topic (Topic) + data generation time + starting message offset + ending message offset; of course, it can also be other forms of data directories, and this example does not impose special restrictions on this. Further, after obtaining the data directory, the data directory can be stored in the metadata database to facilitate data migration.

[0096] The following will be combined Figure 2-Figure 5 right Figure 1 The cold data migration method shown is further explained and illustrated. Specifically:

[0097] In step S110 , a data directory of message data in a message queue is obtained, and a data generation time of the message data and a message offset of the message data are determined according to the data directory.

[0098] In this example embodiment, first, the data directory of the message data in the message queue is obtained. Specifically, the data directory acquisition process can be implemented as follows: a database address is configured for the preset metadata repository, and a communication link is established with the preset metadata repository based on the database address; and the data directory of the message data in the message queue is obtained from the preset metadata repository based on the communication link. In other words, in order to obtain the data directory of the message data from the metadata repository, it is first necessary to ensure that the metadata repository can communicate normally. Under this premise, before the Broker server is started, the database address must be configured for the metadata repository, and after the Broker server is started, it is necessary to determine whether the database address can be connected normally. If an error occurs, the database address must be reconfigured for the metadata repository to ensure that a communication link can be established between the data migration component and the metadata repository based on the database address, and then the data directory is obtained based on the communication link. It should be noted here that once the message data in the message queue is generated, it can be considered as immutable read-only data; and the newly generated message data will only be appended to the end of the queue and will not disrupt the order of historical message data; from this, it can be inferred that the data directory in the metadata database is also arranged in sequence according to the sending time of the message data; under this premise, the message data migration component can determine the starting node of this data migration based on the final time node of the previous migration, and then extract the corresponding data directory from the metadata database based on the starting node; that is, the message data migration component here can be understood as a special message consumer, which can consume message data from the message queue and then migrate and store the data files.

[0099] Furthermore, after obtaining the data directory, the data generation time of the message data and the message offset of the message data can be determined based on the data directory; based on the above-mentioned recorded content, it can be known that the data directory is generated based on the data generation time and the message offset. Therefore, the corresponding data generation time, starting message offset and ending message offset can be directly extracted from the data directory here.

[0100] In step S120 , it is determined whether the message data is cold data according to the data generation time.

[0101] Specifically, in actual application, whether the message data is cold data can be determined based on the data generation time. This can be achieved in the following way: first, obtain the current time node in the message queue and calculate the time difference between the current time node and the data generation time; second, determine whether the time difference is greater than a preset time threshold, and when it is determined that the time difference is greater than or equal to the preset time threshold, determine that the message data is cold data. In other words, whether the message data is cold data can be determined based on the storage time (time difference) of the data file of the message data in the local disk; if the storage time exceeds the maximum message storage time (that is, the preset time threshold, for example, the value can be 7 days), the message data can be determined to be cold data; if it does not exceed the maximum message storage time, the message data can be determined to be hot data; if the message data is hot data, the data file corresponding to the hot data can be directly saved in the local disk without the need for migration.

[0102] In step S130 , when it is determined that the message data is cold data, a data file corresponding to the message data is obtained from the local disk where the message queue is located based on the message offset.

[0103] Specifically, retrieving the data file corresponding to the message data from the local disk where the message queue is located based on the message offset can be achieved as follows: determining the file name of the data file associated with the message data based on the starting message offset and the ending message offset; and retrieving the data file corresponding to the message data from the local disk where the message data in the message queue is located based on the file name. In other words, before migrating the message data to be migrated to the data storage bucket, it is necessary to first determine the data file to be migrated; and in the process of determining the data file to be migrated, the message offset can be used as the basis for the determination.

[0104] In step S140 , the data file corresponding to the message data is migrated from the local disk to a data storage bucket set in the object storage service.

[0105] Specifically, migrating the data file corresponding to the message data from the local disk to the data storage bucket set in the object storage service can be achieved in the following manner: first, configuring the interface address and access key for the object storage service, and generating a data packet to be migrated based on the access key and the data file corresponding to the message data; secondly, sending the data packet to be migrated to the object storage service, and the object storage service verifies the legitimacy of the data packet to be migrated based on the access key in the data packet to be migrated; then, when the object storage service determines that the legitimacy verification of the data packet to be migrated has passed, it migrates the data file corresponding to the message data in the data packet to be migrated from the local disk to the data storage bucket set in the object storage service. That is to say, in the actual application process, before migrating the data files of the message data to be migrated to the object storage service, you must first configure the corresponding interface address and access key for the object storage service, and configure the bucket name for the data storage bucket; then, when the Broker server starts, you need to first check whether the interface address can be connected normally, and determine whether the data storage bucket actually exists based on the bucket name. You also need to determine the accuracy of the access key. Only when it is determined that the interface address can be connected normally, the data storage bucket actually exists, and the access key is accurate, can the data files be correctly migrated to the data storage bucket. At the same time, the reason why the access key needs to be set here is to improve the security of the data files stored in the object storage service, and to avoid data leakage caused by unauthorized users during the data reading and writing process.

[0106] In an exemplary embodiment, after migrating the data file to the data storage bucket, the cold data migration method further includes: upon detecting that the migration of the data file corresponding to the message data is complete, the object storage service generates an object unique identifier corresponding to the message data based on the bucket name and the message subject to which the message data belongs; establishes a mapping relationship between the object unique identifier and the data directory of the message data, and stores the object unique identifier in a preset metadata repository based on the mapping relationship. That is, after the data file is successfully migrated, the object storage service also needs to feedback an object unique identifier to the broker server to facilitate subsequent data queries; wherein, in the object unique identifier generation process, the object unique identifier can be generated based on information such as the message subject, the bucket name, and the message offset; and, upon receiving the object unique identifier, the broker server can associate the object unique identifier with the data directory and store it in the metadata repository; further, upon receiving the object unique identifier, the broker server also needs to delete the corresponding data file from the local disk, thereby ensuring that the data files stored on the local disk are all hot data, thereby reducing the storage burden of the local disk.

[0107] The exemplary embodiment of the present disclosure also provides a method for obtaining cold data, which can be run on a server, server cluster, or cloud server, etc. Of course, those skilled in the art can also run the method of the present disclosure on other platforms as needed, and this exemplary embodiment does not specifically limit this. Figure 6 As shown, the cold data acquisition method may include the following steps:

[0108] Step S610: In response to the cold data acquisition request sent by the business acquisition party, determine the start query time of the data to be queried, and match the data generation time of the data to be queried corresponding to the start query time in a preset metadata database based on the start query time;

[0109] Step S620: extracting a data directory including data generation time from the preset metadata database, constructing a data set of the data to be queried based on the data directory, and determining a start message offset and an end message offset of the data to be queried based on the data directory;

[0110] Step S630: sorting the data to be queried in the data set based on the data generation time, the start message offset, and the end message offset to determine the start consumption position of the data to be queried;

[0111] Step S640, based on the starting consumption position, obtain the data files of the data to be queried from the data storage bucket set in the object storage service in sequence, and load the data files of the data to be queried to the local disk for consumption by the business acquirer; wherein, the data files are stored in the data storage bucket through any of the cold data migration methods described above.

[0112] In an example embodiment, based on the starting consumption position, data files of the data to be queried are obtained from the data storage bucket set in the object storage service in sequence, which can be achieved in the following manner: based on the starting consumption position, a preset metadata database is traversed in sequence, and based on the data directory of the data to be queried, the object unique identifier of the data to be queried in the data storage bucket set in the object storage service is determined; based on the interface address and access key of the object storage service, the data file of the data to be queried is obtained from the data storage bucket corresponding to the object unique identifier.

[0113] In an example embodiment, the data file of the data to be queried is loaded onto a local disk for consumption by the business acquirer, which can be achieved by copying the data file of the data to be queried to the local disk where the message queue is located, and loading the data to be queried into the message queue, so that the business acquirer reads the data file of the data to be queried from the message queue.

[0114] Below, steps S610 to S640 will be explained and illustrated. Specifically, in the actual application process, when the business acquirer needs to obtain cold data, it first passes in a cold data acquisition request including a target date (for example, "60 days ago"); when the Broker server receives the cold data acquisition request, it can determine the starting query time based on the target date and the current time node, and then retrieve the data set of all records with the closest data creation time (that is, it can be advanced or delayed by a certain time, and in the actual application process, it can be selected according to actual needs, and this example does not impose special restrictions on this) in the metadata database based on the starting query time, and sort them in positive order based on the starting offset dimension; at this time, the data The minimum value of the starting message offset in the records in the data collection is the starting position of consumption of the business message receiver; then, based on the object unique identifier corresponding to this record, the data file is obtained from the data storage bucket in the object storage service and copied to the local disk; at this point, the business message receiver starts to operate normally; further, in the process of the business acquirer consuming the data file, the Broker server can automatically detect the current receiving offset of the business message receiver. When it is close to the end message offset of the current file, it obtains the object unique identifier of the next record in the collection, and also obtains the file and copies it to the local disk; and so on, until all cold data are consumed.

[0115] So far, the cold data migration method and cold data acquisition method described in the exemplary embodiments of this disclosure have been fully implemented. Figure 7 The specific implementation process of data migration and data acquisition in the exemplary embodiment of the present disclosure is further explained and illustrated. Figure 7 As shown in the figure, in actual application, first, the interface address, bucket name, access key, metadata database address, etc. of the object storage service are configured. When the message data migration and recovery device is started, it will first check whether the interface address and metadata database address of the object storage service can be connected normally, check whether the bucket exists, and finally check whether the access key is correct. If any error occurs, it will exit the operation. Secondly, the files in the message queue data directory are continuously detected. Once a new file is generated, a record is written to the metadata database, including its following key attributes: creation time, start message offset, and end message offset. Then, based on the pre-configured retention time, if the file has exceeded the time, the file is immediately moved to the corresponding bucket of the object storage. Once the operation is successful, the object storage service will return the newly generated object unique identifier and then update the identifier and write it to the metadata database (the same record). Furthermore, when the business needs to obtain cold data, it first enters a target date. The system retrieves a set of all records with a creation time closest to the date in the metadata database and sorts them in ascending order based on the start message offset. At this time, the record with the smallest start message offset in the set is the consumption starting position of the business message receiver. Then, based on the object unique identifier corresponding to this record, the file is retrieved from the object storage service and copied locally. At this point, the business message receiver can begin normal operation. Finally, the business message receiver's current receiving offset is automatically detected. When it approaches the end message offset of the current file, the object unique identifier of the next record in the collection is obtained, and the file is also retrieved and copied locally. This process continues in this manner until all cold data is consumed.

[0116] In an exemplary embodiment, referring to Figure 7 As shown, the message producer is used to produce message data and store it in the message queue of the corresponding topic; wherein the starting message offset of the message data produced by the message producer can be 0, and the ending message offset (i.e. Figure 7 The end offset shown in is D; At the same time, in the actual application process, the message producer will continuously produce new message data and arrange them in the message queue under the topic according to the data generation time; Further, continue to refer to Figure 7As shown, the message receiver is used to consume the message data in the message queue under the topic; wherein, the message receiver recorded here may include one or more, each message receiver can consume message data with the same offset at the same time, or can consume message data with different offsets separately according to actual conditions. This example does not impose any special restrictions on this.

[0117] Based on the above records, we can know that after the data migration function is processed, only hot data will always be stored on the server's local hard disk, while cold data will be stored in the object storage service; at the same time, if cold data needs to be obtained, the cold data can be reloaded to the message queue server and read by the recipient; it should be noted here that if hot data needs to be obtained, it can be obtained directly from the local disk of the Broker server.

[0118] At this point, the methods described in the present disclosure have been fully implemented. Based on the aforementioned recorded content, it can be known that the methods provided by the example embodiments of the present disclosure can be applied to the hot and cold message separation mechanisms and processes of various current message queue software; and, it can also implement cold message storage and recovery methods using cheap object storage, as well as cold message file acquisition one by one (instead of copying all files in the object storage to the local hard disk at the beginning, because the local hard disk is usually small in capacity and expensive and cannot accommodate all cold messages) to achieve corresponding data storage and / or data acquisition.

[0119] The following are embodiments of the apparatus disclosed herein, which can be used to implement the method embodiments disclosed herein. For details not disclosed in the apparatus embodiments disclosed herein, please refer to the method embodiments disclosed herein.

[0120] The exemplary embodiment of the present disclosure also provides a cold data migration device. Figure 8 As shown, the cold data migration device may include a message offset determination module 810, a cold data determination module 820, a data file acquisition module 830, and a data file migration module 840.

[0121] The message offset determination module 810 may be configured to obtain a data directory of message data in a message queue and determine a data generation time of the message data and a message offset of the message data according to the data directory;

[0122] A cold data determination module 820 may be configured to determine whether the message data is cold data based on the data generation time;

[0123] The data file acquisition module 830 may be configured to acquire a data file corresponding to the message data from the local disk where the message queue is located based on the message offset when determining that the message data is cold data;

[0124] The data file migration module 840 may be configured to migrate the data file corresponding to the message data from the local disk to a data storage bucket provided in the object storage service.

[0125] In an exemplary embodiment of the present disclosure, determining whether the message data is cold data based on the data generation time includes: obtaining the current time node in the message queue, and calculating the time difference between the current time node and the data generation time; judging whether the time difference is greater than a preset time threshold, and determining that the message data is cold data when it is determined that the time difference is greater than or equal to the preset time threshold.

[0126] In an exemplary embodiment of the present disclosure, the message offset includes a starting message offset and an ending message offset; wherein, obtaining a data file corresponding to the message data from a local disk where the message queue is located based on the message offset includes: determining a file name of the data file associated with the message data based on the starting message offset and the ending message offset; and obtaining a data file corresponding to the message data from a local disk where the message data in the message queue is located based on the file name.

[0127] In an exemplary embodiment of the present disclosure, the data file corresponding to the message data is migrated from the local disk to a data storage bucket set in the object storage service, including: configuring an interface address and an access key for the object storage service, and generating a data packet to be migrated based on the access key and the data file corresponding to the message data; sending the data packet to be migrated to the object storage service, and the object storage service verifies the legitimacy of the data packet to be migrated based on the access key in the data packet to be migrated; when the object storage service determines that the legitimacy verification of the data packet to be migrated has passed, the data file corresponding to the message data in the data packet to be migrated is migrated from the local disk to the data storage bucket set in the object storage service.

[0128] In an exemplary embodiment of the present disclosure, the cold data migration device further includes:

[0129] The object unique identifier generation module may be used by the object storage service to generate an object unique identifier corresponding to the message data according to the bucket name and the message topic to which the message data belongs when detecting that the migration of the data file corresponding to the message data is completed;

[0130] The object unique identifier storage module can be used to establish a mapping relationship between the object unique identifier and the data directory of the message data, and store the object unique identifier in a preset metadata database based on the mapping relationship.

[0131] In an exemplary embodiment of the present disclosure, the data directory is generated in the following manner: detecting whether there is new message data in the message queue, and when detecting that there is new message data in the message queue, obtaining the data generation time of the new message data and the message topic to which the new message data belongs; determining the number of data files of the new message data under the message topic according to the data volume of the new message data, and determining the starting message offset and the ending message offset of the new message data according to the number of data files; generating a data directory of the new data message according to the data generation time, the starting message offset and the ending message offset, and writing the data directory into a preset metadata database.

[0132] In an exemplary embodiment of the present disclosure, obtaining a data directory of message data in a message queue includes: configuring a database address for the preset metadata database, and establishing a communication link with the preset metadata database based on the database address; and obtaining the data directory of the message data in the message queue from the preset metadata database based on the communication link.

[0133] The exemplary embodiment of the present disclosure also provides a device for acquiring cold data. Specifically, refer to Figure 9 As shown, the cold data acquisition device may include a data generation time matching module 910, a data set construction module 920, a starting consumption position determination module 930, and a data file loading module 940.

[0134] The data generation time matching module 910 may be configured to determine a start query time of the data to be queried in response to a cold data acquisition request sent by a business acquisition party, and match a data generation time of the data to be queried corresponding to the start query time in a preset metadata database based on the start query time;

[0135] The data set construction module 920 may be configured to extract a data directory including data generation time from the preset metadata database, construct a data set of the data to be queried based on the data directory, and determine a start message offset and an end message offset of the data to be queried based on the data directory;

[0136] The starting consumption position determining module 930 may be configured to sort the data to be queried in the data set based on the data generation time, the starting message offset, and the ending message offset to determine the starting consumption position of the data to be queried;

[0137] The data file loading module 940 can be used to obtain the data files of the data to be queried from the data storage bucket set in the object storage service in sequence based on the starting consumption position, and load the data files of the data to be queried to the local disk for consumption by the business acquirer; wherein, the data file is stored in the data storage bucket through any of the cold data migration methods described above.

[0138] In an exemplary embodiment of the present disclosure, based on the starting consumption position, data files of the data to be queried are obtained in sequence from the data storage bucket set in the object storage service, including: based on the starting consumption position, traversing the preset metadata database in sequence, and determining the object unique identifier of the data to be queried in the data storage bucket set in the object storage service based on the data directory of the data to be queried; based on the interface address and access key of the object storage service, obtaining the data file of the data to be queried from the data storage bucket corresponding to the object unique identifier.

[0139] In an exemplary embodiment of the present disclosure, the data file of the data to be queried is loaded into a local disk for consumption by a business acquirer, including: copying the data file of the data to be queried to the local disk where the message queue is located, and loading the data to be queried into the message queue, so that the business acquirer reads the data file of the data to be queried from the message queue.

[0140] The specific details of each module in the cold data migration device and the cold data acquisition device have been described in detail in the corresponding cold data migration method and the cold data acquisition method, and thus will not be repeated here.

[0141] It should be noted that although several modules or units of the device for action execution are mentioned in the detailed description above, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be concretized in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided into multiple modules or units to be concretized.

[0142] Furthermore, although the steps of the method of the present disclosure are described in a particular order in the accompanying drawings, this does not require or imply that the steps must be performed in this particular order, or that all steps shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps.

[0143] In an exemplary embodiment of the present disclosure, an electronic device capable of implementing the above method is also provided.

[0144] Those skilled in the art will appreciate that various aspects of the present disclosure may be implemented as systems, methods, or program products. Therefore, various aspects of the present disclosure may be implemented in the following forms: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software implementations, which may be collectively referred to herein as "circuits," "modules," or "systems."

[0145] Refer to the following Figure 10 1000 according to this embodiment of the present disclosure will be described. Figure 10 The electronic device 1000 shown is merely an example and should not limit the functions and scope of use of the embodiments of the present disclosure.

[0146] like Figure 10 As shown, electronic device 1000 is implemented as a general-purpose computing device. Components of electronic device 1000 may include, but are not limited to, the aforementioned at least one processing unit 1010, the aforementioned at least one storage unit 1020, a bus 1030 connecting various system components (including storage unit 1020 and processing unit 1010), and a display unit 1040.

[0147] The storage unit stores program codes, which can be executed by the processing unit 1010, so that the processing unit 1010 performs the steps described in the "Exemplary Method" section of the present disclosure according to various exemplary embodiments. For example, the processing unit 1010 can perform the following steps: Figure 1 Step S110 shown in: obtaining the data directory of the message data in the message queue, and determining the data generation time of the message data and the message offset of the message data according to the data directory; step S120: determining whether the message data is cold data according to the data generation time; step S130: when it is determined that the message data is cold data, obtaining the data file corresponding to the message data from the local disk where the message queue is located based on the message offset; step S140: migrating the data file corresponding to the message data from the local disk to the data storage bucket set in the object storage service.

[0148] For another example, the processing unit 1010 may execute the following Figure 6 Step S610 shown in: in response to the cold data acquisition request sent by the business acquirer, determining the starting query time of the data to be queried, and matching the data generation time of the data to be queried corresponding to the starting query time in the preset metadata database based on the starting query time; step S620: extracting the data directory including the data generation time in the preset metadata database, and constructing a data set of the data to be queried based on the data directory, and determining the starting message offset and the ending message offset of the data to be queried based on the data directory; step S630: sorting the data to be queried in the data set based on the data generation time, the starting message offset and the ending message offset to determine the starting consumption position of the data to be queried; step S640: based on the starting consumption position, obtaining the data files of the data to be queried from the data storage bucket set in the object storage service in sequence, and loading the data files of the data to be queried to the local disk for consumption by the business acquirer; wherein, the data files are stored in the data storage bucket by any of the cold data migration methods described above.

[0149] The storage unit 1020 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 10201 and / or a cache memory unit 10202 , and may further include a read-only memory unit (ROM) 10203 .

[0150] The storage unit 1020 may also include a program / utility 10204 having a set (at least one) of program modules 10205, such program modules 10205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0151] Bus 1030 may represent one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0152] The electronic device 1000 can also communicate with one or more external devices 1100 (e.g., a keyboard, a pointing device, a Bluetooth device, etc.), one or more devices that enable a user to interact with the electronic device 1000, and / or any device that enables the electronic device 1000 to communicate with one or more other computing devices (e.g., a router, a modem, etc.). Such communication can occur via an input / output (I / O) interface 1050. Furthermore, the electronic device 1000 can also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network such as the Internet) via a network adapter 1060. As shown, the network adapter 1060 communicates with other modules of the electronic device 1000 via a bus 1030. It should be understood that, although not shown in the figure, other hardware and / or software modules can be used in conjunction with the electronic device 1000, including but not limited to microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0153] Through the description of the above embodiments, it is easy for those skilled in the art to understand that the example embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solution according to the embodiments of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the embodiments of the present disclosure.

[0154] In exemplary embodiments of the present disclosure, a computer-readable storage medium is also provided, on which is stored a program product capable of implementing the aforementioned methods of this specification. In some possible implementations, various aspects of the present disclosure may also be implemented in the form of a program product comprising program code. When the program product is executed on a terminal device, the program code is configured to cause the terminal device to execute the steps described in the "Exemplary Methods" section of this specification according to various exemplary embodiments of the present disclosure.

[0155] According to an embodiment of the present disclosure, a program product for implementing the above-mentioned method can be a portable compact disc read-only memory (CD-ROM) and include program code, and can be run on a terminal device, such as a personal computer. However, the program product of the present disclosure is not limited thereto. In this document, a readable storage medium can be any tangible medium containing or storing a program, which can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0156] The program product may be implemented in any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or component, or any combination thereof. More specific examples (a non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0157] A computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium that can transmit, propagate, or transfer a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0158] The program code embodied on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0159] The program code for performing the operations of the present disclosure may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, and the like, as well as conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a stand-alone software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device may be connected to the user computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0160] Furthermore, the figures above are merely illustrative of the processes included in the methods according to exemplary embodiments of the present disclosure and are not intended to be limiting. It is readily understood that the processes illustrated in the figures above do not indicate or limit the temporal order of these processes. Furthermore, it is readily understood that these processes may be executed synchronously or asynchronously, for example, in multiple modules.

[0161] Other embodiments of the present disclosure will readily occur to those skilled in the art after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow from the general principles of the present disclosure and include common knowledge or customary techniques in the art not invented herein. The specification and examples are to be considered as exemplary only, with the true scope and spirit of the present disclosure being indicated by the claims.

Claims

1. A cold data migration method, characterized in that: include: Obtaining a data directory of message data in a message queue, and determining a data generation time of the message data and a message offset of the message data according to the data directory; Determining whether the message data is cold data according to the data generation time; When it is determined that the message data is cold data, obtaining a data file corresponding to the message data from a local disk where the message queue is located based on the message offset; Migrating a data file corresponding to the message data from the local disk to a data storage bucket set in the object storage service includes: Configuring an interface address and an access key for the object storage service, and generating a data packet to be migrated based on the access key and a data file corresponding to the message data; Sending the data package to be migrated to the object storage service, where the object storage service verifies the legitimacy of the data package to be migrated based on the access key in the data package to be migrated; When the object storage service determines that the legitimacy verification of the data packet to be migrated has passed, the object storage service migrates the data file corresponding to the message data in the data packet to be migrated from the local disk to a data storage bucket set in the object storage service; The method further comprises: When the object storage service detects that the migration of the data file corresponding to the message data is complete, the object storage service generates an object unique identifier corresponding to the message data according to the bucket name and the message topic to which the message data belongs; A mapping relationship between the object unique identifier and the data directory of the message data is established, and the object unique identifier is stored in a preset metadata database based on the mapping relationship.

2. The cold data migration method according to claim 1, characterized in that: Determining whether the message data is cold data according to the data generation time includes: Obtaining a current time node in the message queue, and calculating a time difference between the current time node and the data generation time; It is determined whether the time difference is greater than a preset time threshold, and when it is determined that the time difference is greater than or equal to the preset time threshold, it is determined that the message data is cold data.

3. The cold data migration method according to claim 1, characterized in that: The message offset includes a start message offset and an end message offset; Wherein, obtaining a data file corresponding to the message data from the local disk where the message queue is located based on the message offset includes: Determining a file name of a data file associated with the message data based on the start message offset and the end message offset; A data file corresponding to the message data in the message queue is obtained from the local disk where the message data is located based on the file name.

4. The cold data migration method according to claim 1, characterized in that: The data directory is generated in the following way: Detecting whether there is new message data in the message queue, and when detecting that there is new message data in the message queue, obtaining the data generation time of the new message data and the message topic to which the new message data belongs; Determining the number of data files of the newly added message data under the message topic according to the data volume of the newly added message data, and determining the starting message offset and the ending message offset of the newly added message data according to the number of data files; A data directory of the newly added message data is generated according to the data generation time, the start message offset, and the end message offset, and the data directory is written into a preset metadata database.

5. The cold data migration method according to claim 4, characterized in that: Get the data directory of the message data in the message queue, including: Configuring a database address for the preset metadata database, and establishing a communication link with the preset metadata database based on the database address; A data directory of the message data in the message queue is obtained from the preset metadata database based on the communication link.

6. A method for obtaining cold data, characterized in that: include: In response to a cold data acquisition request sent by a business acquisition party, determining a start query time of the data to be queried, and matching a data generation time of the data to be queried corresponding to the start query time in a preset metadata database based on the start query time; Extracting a data directory including data generation time from the preset metadata database, constructing a data set of the data to be queried according to the data directory, and determining a start message offset and an end message offset of the data to be queried based on the data directory; Sort the data to be queried in the data set based on the data generation time, the start message offset, and the end message offset to determine the start consumption position of the data to be queried; Based on the starting consumption position, data files of the data to be queried are obtained in sequence from the data storage bucket set in the object storage service, and the data files of the data to be queried are loaded to the local disk for consumption by the business acquirer; wherein, the data files are stored in the data storage bucket by the cold data migration method described in any one of claims 1-5.

7. The method for obtaining cold data according to claim 6, characterized in that: Based on the starting consumption position, sequentially obtaining data files of the data to be queried from the data storage buckets set in the object storage service includes: Based on the starting consumption position, traverse the preset metadata database in sequence, and determine the object unique identifier of the data to be queried in the data storage bucket set in the object storage service based on the data directory of the data to be queried; Based on the interface address and access key of the object storage service, the data file of the data to be queried is obtained from the data storage bucket corresponding to the object unique identifier.

8. The method for obtaining cold data according to claim 6, characterized in that: Loading the data file of the data to be queried to the local disk for consumption by the business acquirer includes: The data file of the data to be queried is copied to the local disk where the message queue is located, and the data to be queried is loaded into the message queue, so that the service acquirer reads the data file of the data to be queried from the message queue.

9. A cold data migration device, characterized in that: include: A message offset determination module is used to obtain a data directory of message data in a message queue, and determine a data generation time of the message data and a message offset of the message data according to the data directory; A cold data determination module, configured to determine whether the message data is cold data according to the data generation time; a data file acquisition module, configured to acquire, when determining that the message data is cold data, a data file corresponding to the message data from a local disk where the message queue is located based on the message offset; A data file migration module, for migrating the data file corresponding to the message data from the local disk to a data storage bucket set in the object storage service, including: configuring an interface address and an access key for the object storage service, and generating a data packet to be migrated based on the access key and the data file corresponding to the message data; sending the data packet to be migrated to the object storage service, and the object storage service verifying the legitimacy of the data packet to be migrated based on the access key in the data packet to be migrated; when the object storage service determines that the legitimacy verification of the data packet to be migrated has passed, migrating the data file corresponding to the message data in the data packet to be migrated from the local disk to a data storage bucket set in the object storage service; an object unique identifier generation module, configured to generate an object unique identifier corresponding to the message data according to the bucket name and the message topic to which the message data belongs, when the object storage service detects that the migration of the data file corresponding to the message data is complete; and A mapping relationship between the object unique identifier and the data directory of the message data is established, and the object unique identifier is stored in a preset metadata database based on the mapping relationship.

10. A cold data acquisition device, characterized in that: include: a data generation time matching module, configured to determine, in response to a cold data acquisition request sent by a business acquisition party, a start query time of the data to be queried, and match, based on the start query time, a data generation time of the data to be queried corresponding to the start query time in a preset metadata database; a data set construction module, configured to extract a data directory including data generation time from the preset metadata database, construct a data set of the data to be queried based on the data directory, and determine a start message offset and an end message offset of the data to be queried based on the data directory; a starting consumption position determining module, configured to sort the data to be queried in the data set based on the data generation time, the starting message offset, and the ending message offset, so as to determine the starting consumption position of the data to be queried; A data file loading module is used to obtain data files of the data to be queried from the data storage bucket set in the object storage service in sequence based on the starting consumption position, and load the data files of the data to be queried to the local disk for consumption by the business acquirer; wherein, the data file is stored in the data storage bucket through the cold data migration method described in any one of claims 1-5.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the cold data migration method according to any one of claims 1 to 5 and the cold data acquisition method according to any one of claims 6 to 8 are implemented.

12. An electronic device, characterized in that: include: processor; as well as a memory for storing executable instructions of the processor; The processor is configured to execute the cold data migration method according to any one of claims 1 to 5 and the cold data acquisition method according to any one of claims 6 to 8 by executing the executable instructions.

Images