Data storage system, storage and reading method, device, equipment and storage medium

Abstract

Embodiments of the present application provide a data storage system, a storage and reading method, a device, equipment and a storage medium, which are applied to the field of data processing, wherein the data storage system comprises a client, a service node and a storage node; the service node is configured to save storage mode labels of the storage nodes, the storage mode labels comprising a streaming storage label and a batch storage label; the client is configured to receive to-be-stored data and a storage requirement; the storage requirement is configured to indicate pure streaming storage, pure batch storage or streaming-batch integrated storage; a target streaming storage node and / or a target batch storage node are acquired from the storage nodes according to the storage requirement; and the target streaming storage node and / or the target batch storage node are configured to store the to-be-stored data. The embodiments of the present application can reduce the maintenance cost of the storage service.

Description

Technical Field

[0001] This invention relates to the field of data processing, and in particular to a data storage system, storage and retrieval method, apparatus, device and storage medium. Background Technology

[0002] In related technologies, to meet the storage needs of both streaming and batch storage business scenarios, such as when a data stream scenario requires both batch storage and streaming scenario, the data is stored using independent systems for streaming and batch storage respectively. That is, one system performs streaming storage and another system performs batch storage. Since the two storage systems are not related and are independent of each other, the two systems need to be maintained separately at the same time when maintaining the system, which makes the maintenance cost of storage services too high. Summary of the Invention

[0003] The purpose of this invention is to provide a data storage system, storage and retrieval method, apparatus, device, and storage medium to reduce the maintenance costs of storage services. The specific technical solution is as follows:

[0004] In a first aspect, embodiments of the present invention provide a data storage system, the data storage system comprising: a client, a service node, and a storage node;

[0005] The service node is used to store the storage method tags of each storage node. The storage method tags include streaming storage tags and batch storage tags. The streaming storage tag indicates that the storage node uses streaming storage to store data, and the batch storage tag indicates that the storage node uses batch storage to store data.

[0006] The client is used to receive data to be stored and storage requirements; the storage requirements are determined based on the application scenario of the data to be stored, and are used to indicate pure streaming storage, pure batch storage, or integrated streaming and batch storage; pure streaming storage stores the data to be stored only through the streaming storage node; pure batch storage stores the data to be stored only through the batch storage node; integrated streaming and batch storage stores the data to be stored through both the streaming storage node and the batch storage node; according to the storage requirements, a target streaming storage node and / or a target batch storage node are determined from each storage node; wherein, the target streaming storage node includes storage nodes with the storage method label of streaming storage, and the target batch storage node includes storage nodes with the storage method label of batch storage; the data to be stored is stored to the target streaming storage node and / or the target batch storage node;

[0007] The target streaming storage node and / or the target batch storage node are used to store the data to be stored.

[0008] Optionally, the client is specifically configured to send the storage requirement and the first target data identifier corresponding to the data to be stored to the service node;

[0009] The service node is specifically used to determine the target streaming storage node and / or target batch storage node corresponding to the first target data identifier according to the storage requirements, the first target data identifier corresponding to the data to be stored, and the storage method label of the storage node, and to send the node information of the target streaming storage node and / or target batch storage node corresponding to the first target data identifier to the client. The node information is used by the client to send the data to be stored to the target streaming storage node and / or target batch storage node so that the target streaming storage node and / or target batch storage node can store it.

[0010] The client is specifically used to receive the node information.

[0011] Optionally, the storage requirement is used to indicate integrated stream and batch storage;

[0012] The client is specifically configured to determine the target streaming storage node and the target batch storage node from each storage node according to the storage requirements indicated for integrated streaming and batch storage; and send the data to be stored to the target streaming storage node.

[0013] The target streaming storage node is specifically used to receive the data to be stored sent by the client, store the data to be stored in a streaming storage manner, and synchronize the data to be stored to the target batch storage node.

[0014] The target batch storage node is used to store copies of the data to be stored in a batch storage manner.

[0015] Secondly, embodiments of the present invention provide a data storage method, the method comprising:

[0016] The system receives data to be stored and storage requirements; the storage requirements are determined based on the application scenario of the data to be stored and are used to indicate pure streaming storage, pure batch storage, or integrated streaming and batch storage; pure streaming storage stores the data to be stored only through the streaming storage node; pure batch storage stores the data to be stored only through the batch storage node; integrated streaming and batch storage stores the data to be stored through both the streaming storage node and the batch storage node.

[0017] According to the storage requirements, target streaming storage nodes and / or target batch storage nodes are determined from each storage node; wherein, the target streaming storage node includes storage nodes with the storage method label of streaming storage, and the target batch storage node includes storage nodes with the storage method label of batch storage; the streaming storage label indicates that the storage method used by the storage node to store data is streaming storage, and the batch storage label indicates that the storage method used by the storage node to store data is batch storage;

[0018] The data to be stored is stored in the target streaming storage node and / or the target batch storage node.

[0019] Optionally, determining the target streaming storage node and / or target batch storage node from the storage nodes according to the storage requirements includes:

[0020] The storage requirement and the first target data identifier corresponding to the data to be stored are sent to the service node. The service node is used to determine the target streaming storage node and / or target batch storage node corresponding to the first target data identifier according to the storage requirement, the first target data identifier corresponding to the data to be stored, and the storage method label of each storage node.

[0021] The system receives node information of the target streaming storage node and / or target batch storage node corresponding to the first target data identifier. The node information is used by the client to send the data to be stored to the target streaming storage node and / or target batch storage node so that the target streaming storage node and / or target batch storage node can store the data.

[0022] Optionally, the storage requirement is used to indicate integrated stream and batch storage;

[0023] The step of determining the target streaming storage node and / or target batch storage node from each storage node according to the storage requirements includes:

[0024] According to the storage requirements for indicating integrated streaming and batch storage, the target streaming storage node and the target batch storage node corresponding to the first target data identifier are determined from each storage node;

[0025] The step of storing the data to be stored to the target streaming storage node and / or the target batch storage node includes:

[0026] The data to be stored is sent to the target streaming storage node so that the target streaming storage node stores the data to be stored and synchronizes the data to be stored to the target batch storage node.

[0027] Thirdly, embodiments of the present invention provide a data reading method, the method comprising:

[0028] Receive a data access request, the data access request carrying a data reading requirement, the data reading requirement being determined based on the application scenario of the data to be accessed indicated by the data access request, and used to indicate streaming reading or batch reading;

[0029] If the data read request is used to indicate streaming read, then a streaming storage node is determined from each storage node, and data is read from the streaming storage node;

[0030] If the data read request is used to indicate batch read, then the batch storage node is determined from each storage node, and data is read from the batch storage node;

[0031] The streaming storage node includes storage nodes with a storage method label of streaming storage, and the batch storage node includes storage nodes with a storage method label of batch storage. The streaming storage label indicates that the storage method used by the storage node to store data is streaming storage, and the batch storage label indicates that the storage method used by the storage node to store data is batch storage. The pure streaming storage stores the data to be stored only through the streaming storage node; the pure batch storage stores the data to be stored only through the batch storage node; and the integrated streaming and batch storage stores the data to be stored through both the streaming storage node and the batch storage node.

[0032] Optionally, the data access request further carries a second target data identifier, the second target data identifier being the data identifier corresponding to the data to be accessed indicated by the data access request, and the method further includes:

[0033] Obtain the correspondence between multiple data identifiers and storage nodes, wherein the correspondence includes the target streaming storage nodes and / or target batch storage nodes corresponding to the multiple data identifiers;

[0034] The step of determining the streaming storage node from each storage node includes:

[0035] Based on the correspondence, determine the streaming storage node corresponding to the second target data identifier from each storage node;

[0036] The step of determining the batch storage nodes from each storage node includes:

[0037] Based on the correspondence, the batch storage node corresponding to the second target data identifier is determined from each storage node.

[0038] Fourthly, embodiments of the present invention provide a data storage device, the device comprising:

[0039] A receiving module is used to receive data to be stored and storage requirements; the storage requirements are determined based on the application scenario of the data to be stored, and are used to indicate pure streaming storage, pure batch storage, or integrated streaming and batch storage; pure streaming storage stores the data to be stored only through the streaming storage node; pure batch storage stores the data to be stored only through the batch storage node; integrated streaming and batch storage stores the data to be stored through both the streaming storage node and the batch storage node.

[0040] The determination module is used to determine target streaming storage nodes and / or target batch storage nodes from each storage node according to the storage requirements; wherein, the target streaming storage node includes storage nodes with a storage method label of streaming storage, and the target batch storage node includes storage nodes with a storage method label of batch storage; the streaming storage label indicates that the storage method used by the storage node to store data is streaming storage, and the batch storage label indicates that the storage method used by the storage node to store data is batch storage;

[0041] A storage module is used to store the data to be stored to the target streaming storage node and / or the target batch storage node.

[0042] Optionally, the determining module is specifically used to send the storage requirement and the first target data identifier corresponding to the data to be stored to the service node. The service node is used to determine the target streaming storage node and / or target batch storage node corresponding to the first target data identifier according to the storage requirement, the first target data identifier corresponding to the data to be stored, and the storage method label of each storage node; and to receive the node information of the target streaming storage node and / or target batch storage node corresponding to the first target data identifier. The node information is used by the client to send the data to be stored to the target streaming storage node and / or target batch storage node so that the target streaming storage node and / or target batch storage node can store the data.

[0043] Optionally, the storage requirement is used to indicate integrated stream and batch storage;

[0044] The determining module is used to determine the target streaming storage node and the target batch storage node corresponding to the first target data identifier from each storage node according to the storage requirements for indicating integrated streaming and batch storage.

[0045] The storage module is used to send the data to be stored to the target streaming storage node, so that the target streaming storage node stores the data to be stored, and synchronizes the data to be stored to the target batch storage node.

[0046] Fifthly, embodiments of the present invention provide a data reading device, the device comprising:

[0047] A receiving module is used to receive data access requests, which carry data reading requirements. The data reading requirements are determined based on the application scenario of the data to be accessed indicated by the data access request, and are used to indicate streaming reading or batch reading.

[0048] A read module is configured to: if the data read request indicates streaming read, determine a streaming storage node from each storage node and read data from the streaming storage node; if the data read request indicates batch read, determine a batch storage node from each storage node and read data from the batch storage node; wherein, the streaming storage node includes storage nodes with a storage method label of streaming storage, and the batch storage node includes storage nodes with a storage method label of batch storage; the streaming storage label indicates that the storage method used by the storage node to store data is streaming storage, and the batch storage label indicates that the storage method used by the storage node to store data is batch storage; pure streaming storage stores the data to be stored only through the streaming storage node; pure batch storage stores the data to be stored only through the batch storage node; and integrated streaming and batch storage stores the data to be stored through both the streaming storage node and the batch storage node.

[0049] Optionally, the data access request further carries a second target data identifier, the second target data identifier being the data identifier corresponding to the data to be accessed indicated by the data access request, and the device further includes:

[0050] The acquisition module is used to acquire the correspondence between multiple data identifiers and storage nodes, wherein the correspondence includes the target streaming storage nodes and / or target batch storage nodes corresponding to the multiple data identifiers;

[0051] The reading module is specifically used to determine, based on the correspondence, the streaming storage node corresponding to the second target data identifier from each storage node, or, based on the correspondence, the batch storage node corresponding to the second target data identifier from each storage node.

[0052] In a sixth aspect, embodiments of the present invention provide an electronic device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus;

[0053] Memory, used to store computer programs;

[0054] When a processor executes a program stored in memory, it implements any of the method steps described in the second or third aspect above.

[0055] In a seventh aspect, embodiments of the present invention provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements any of the method steps described in the second or third aspect above.

[0056] In the technical solution provided by this invention, the data storage system includes a client, a service node, and storage nodes. The service node stores storage method tags for each storage node. Using these tags, the client receives the data to be stored and the storage requirements, and can then determine a target streaming storage node and / or a target batch storage node from among the storage nodes. The data to be stored is then stored using the target streaming storage node and / or the target batch storage node. The storage requirements indicate pure streaming storage, pure batch storage, or integrated streaming and batch storage. The data storage system can achieve data storage for different storage requirements (pure streaming, pure batch, or integrated streaming and batch storage) through the target streaming storage node and / or the target batch storage node. This avoids the need to maintain two independent systems simultaneously for streaming and batch storage, reducing the maintenance costs of the storage service. Furthermore, the data storage system in this invention can not only achieve pure streaming and pure batch storage, but also integrated streaming and batch storage, meaning it can perform both streaming and batch storage on the data. Attached Figure Description

[0057] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below.

[0058] Figure 1 This is a schematic diagram of the data storage system in an embodiment of the present invention.

[0059] Figure 2 This is a schematic diagram of the master node-worker node architecture in an embodiment of the present invention.

[0060] Figure 3 This is a flowchart of the data storage method in an embodiment of the present invention.

[0061] Figure 4 This is an example diagram illustrating the data storage method used in an embodiment of the present invention.

[0062] Figure 5 This is a flowchart of the data reading method in an embodiment of the present invention.

[0063] Figure 6 This is an example diagram illustrating the application of the data reading method in this embodiment of the invention.

[0064] Figure 7 This is a schematic diagram of the data storage device in an embodiment of the present invention.

[0065] Figure 8 This is a schematic diagram of the data reading device in an embodiment of the present invention.

[0066] Figure 9 This is a schematic diagram of the structure of an electronic device in an embodiment of the present invention. Detailed Implementation

[0067] The technical solutions of the present invention will now be described with reference to the accompanying drawings in the embodiments of the present invention.

[0068] With the development of big data technology, batch storage for batch data and streaming storage for streaming data have emerged. Streaming storage supports fine-grained data reading and writing, can support millisecond-level latency, and has low latency, but low compression ratio and high storage cost. Furthermore, because streaming storage is sensitive to latency, the storage service may experience excessive load in a short period of time during large-scale data queries or batch data imports, even affecting the processing performance of currently active read / write clients. This leads to increased latency in the data path, making streaming storage unsuitable for large-scale data queries. Batch storage, on the other hand, is suitable for reading and writing data in batches at hourly or daily granularities, supports large-scale data queries, and has low storage cost, but relatively high data latency. Due to the differences between streaming and batch storage, existing streaming and batch storage systems are independent of each other. For data used in both streaming and batch scenarios, it is necessary to store one copy of the data in each storage system, providing both batch and streaming storage methods, which increases the maintenance cost of the storage service.

[0069] To reduce the maintenance costs of storage services, embodiments of the present invention provide a data storage system, such as... Figure 1 As shown, the data storage system includes: client 102, service node 101, and storage node 103;

[0070] Service node 101 is used to store the storage method tags of each storage node. The storage method tags include streaming storage tags and batch storage tags. The streaming storage tag indicates that the storage node uses streaming storage, and the batch storage tag indicates that the storage node uses batch storage.

[0071] Client 102 receives the data to be stored and the storage requirements. The storage requirements are determined based on the application scenario of the data to be stored and indicate whether it is pure streaming storage, pure batch storage, or integrated streaming and batch storage. Pure streaming storage stores the data to be stored only through streaming storage nodes; pure batch storage stores the data to be stored only through batch storage nodes; integrated streaming and batch storage stores the data to be stored through both streaming and batch storage nodes. According to the storage requirements, client 102 obtains the target streaming storage node 1031 and / or target batch storage node 1032 corresponding to the data identifier of the data to be stored from each storage node 103, based on the service node 101. The target streaming storage node includes storage nodes with the storage method label of streaming storage, and the target batch storage node includes storage nodes with the storage method label of batch storage. The data to be stored is then stored in the target streaming storage node and / or target batch storage node.

[0072] Target streaming storage nodes and / or target batch storage nodes are used to store the data to be stored.

[0073] In this embodiment of the invention, the data storage system includes a client 102, a service node 101, and a storage node 103. The service node 101 stores the storage method tags of each storage node. Using these tags, the client receives the data to be stored and the storage requirements, and can then determine the target streaming storage node and / or the target batch storage node from among the storage nodes. Thus, the data to be stored is stored using the target streaming storage node and / or the target batch storage node. The storage requirements indicate pure streaming storage, pure batch storage, or integrated streaming and batch storage. The data storage system can achieve data storage for different storage requirements—pure streaming storage, pure batch storage, or integrated streaming and batch storage—through the target streaming storage node and / or the target batch storage node. This avoids the need to maintain two independent systems simultaneously performing streaming and batch storage, reducing the maintenance costs of the storage service. Furthermore, in this embodiment of the invention, the data storage system can not only achieve pure streaming storage and pure batch storage, but also integrated streaming and batch storage, meaning it can perform both streaming and batch storage on the data.

[0074] Reference Figure 1 The data storage system provided in the embodiments of the present invention will be described in detail.

[0075] Service node 101 is used to store the storage method tags of each storage node. The storage method tags include streaming storage tags and batch storage tags. The streaming storage tag indicates that the storage node uses streaming storage to store data, and the batch storage tag indicates that the storage node uses batch storage to store data.

[0076] In the data storage system, there is only one service node 101, also known as the master node. Service node 101 is responsible for data management and the management of storage nodes. It assigns and records tags to each storage node, meaning the storage service supports differentiating storage nodes by streaming and batch tags, allowing for on-demand data storage. Storage nodes are responsible for data storage and reading / writing. Streaming storage nodes, tagged with streaming data, organize data in a streaming manner to provide streaming data read / write services. Batch storage nodes, tagged with batch data, organize data in a batch manner to provide batch data read / write services. Organizing data in a streaming manner means storing and retrieving data using both streaming storage and streaming read methods. Organizing data in a batch manner means storing and retrieving data using both batch storage and batch read methods.

[0077] The service node assigns streaming storage tags and batch storage tags to the storage nodes, enabling the storage nodes to store data according to the storage method corresponding to the assigned tags. This embodiment of the invention achieves three storage modes simultaneously—pure streaming, pure batch, and integrated streaming and batch—through interaction between the service node, storage node, and client, all within a single system.

[0078] There are two types of tags: one for streaming storage and one for batch storage. Stream-batch integrated storage refers to a storage method that uses both storage methods to store data simultaneously, that is, it uses storage nodes with streaming storage tags and storage nodes with batch storage tags to store data. Therefore, it is understandable that a stream-batch integrated storage tag is not set.

[0079] Streaming storage tags and batch storage tags can be represented by different types of information, such as using the number "0" to represent a streaming storage tag and the number "1" to represent a batch storage tag. Alternatively, to make the storage method of the storage node intuitively represent the tag information, streaming storage tags can be "streaming" or "stream," and batch storage tags can be "batch" or "batch."

[0080] Service node 101 can assign storage method tags to each storage node and save the storage method tags of each storage node. Specifically, service node 101 can divide all storage nodes into two parts: assign storage method tags as streaming storage tags to one part of the storage nodes, that is, treat these storage nodes as streaming storage nodes, and store data through streaming storage; assign storage method tags as batch storage tags to the other part of the storage nodes, that is, treat these storage nodes as batch storage nodes, and store data through batch storage.

[0081] In streaming storage, data is stored in units of single data entries. Since streaming data is a large, fast, and continuously arriving sequence of data, it is stored in the order in which the data is generated to maintain its low latency.

[0082] Batch storage divides the overall data into blocks of a fixed size and stores the data in units of blocks. A block is the smallest unit of deployment and can contain multiple data entries. This reduces storage costs. At the same time, different blocks can be stored on different storage nodes, which also means that batch data storage does not care about the storage order.

[0083] Streaming storage nodes store data in units of single data entries, as is typical for streaming data storage, maintaining their low latency characteristics; batch storage nodes store data in units of data blocks, as is typical for batch data storage, maintaining their low storage cost characteristics.

[0084] Client 102 receives the data to be stored and the storage requirements. The storage requirements are determined based on the application scenario of the data to be stored and indicate whether it is pure streaming storage, pure batch storage, or integrated streaming and batch storage. Pure streaming storage stores the data to be stored only through streaming storage nodes. Pure batch storage stores the data to be stored only through batch storage nodes. Integrated streaming and batch storage stores the data to be stored through both streaming and batch storage nodes. According to the storage requirements, a target streaming storage node and / or a target batch storage node are determined from each storage node 103. The target streaming storage node includes storage nodes with the storage method label of streaming storage, and the target batch storage node includes storage nodes with the storage method label of batch storage. The data to be stored is stored to the target streaming storage node and / or the target batch storage node.

[0085] The service node configures two types of tags, stream and batch, for the storage node, enabling the storage node to store data in a manner corresponding to the tag. Based on the received data storage request, the client obtains the storage node corresponding to the data from the service node, and then sends the data to the corresponding storage node. The storage node receives and stores the data.

[0086] Data storage requirements are determined based on the application scenario of the data and are determined when the request is issued. For data that only needs to be used in streaming scenarios, the storage requirement can be pure streaming storage; for data that only needs to be used in batch scenarios, the storage requirement can be pure batch storage; for data that needs to be used in both streaming and batch scenarios, the storage requirement can be integrated streaming and batch storage.

[0087] The data identifier corresponds to the data and is used to identify the data. Understandably, the first target data identifier can correspond to the data to be stored and is used to determine the target storage node of the data to be stored.

[0088] In one possible implementation, the client 102 determines the target streaming storage node and / or target batch storage node from each storage node 103 according to storage requirements. Specifically, this may include: the client 102 sending the storage requirements and the first target data identifier corresponding to the data to be stored to the service node 101; the service node 101 determining the target streaming storage node and / or target batch storage node corresponding to the first target data identifier according to the storage requirements, the first target data identifier corresponding to the data to be stored, and the storage method label of each storage node; and sending the node information of the target streaming storage node and / or target batch storage node corresponding to the first target data identifier to the client 102. The node information is used by the client to send the data to be stored to the target streaming storage node and / or target batch storage node so that the target streaming storage node and / or target batch storage node can store the data. In this way, the client 102 can obtain the target streaming storage node and / or target batch storage node allocated by the service node 101.

[0089] Target streaming storage nodes and / or target batch storage nodes are used to store the data to be stored.

[0090] In this embodiment of the invention, the data storage system supports three storage methods: pure streaming storage, pure batch storage, or integrated streaming and batch storage. Clients can choose different storage methods to meet different storage needs.

[0091] When the storage demand indicates pure streaming storage, the client determines the target streaming storage node from among the storage nodes. In one implementation, the client 102 can send the storage demand indicating pure streaming storage and the first target data identifier corresponding to the data to be stored to the service node 101. Based on the received storage demand and the first target data identifier corresponding to the data to be stored, the service node 101 determines the storage node corresponding to the first target data identifier of the data to be stored, i.e., the target streaming storage node, and sends the node information of the determined target streaming storage node to the client. In this way, the client 102 can obtain the target streaming storage node allocated by the service node 101.

[0092] The node information may include the node's identifier, which indicates which of all storage nodes the node to be stored is based on the first target data identifier. Then, the client determines the target storage node based on the node information and sends the data to be stored to the target storage node for storage.

[0093] In one possible implementation, client 102 can send the data to be stored to a target streaming storage node, which then stores the data.

[0094] In one possible implementation, both the stored data and data replicas of pure streaming storage reside on the streaming storage nodes, employing streaming storage algorithms such as the Kafka algorithm, without any specific limitations.

[0095] For pure streaming storage, there can be one or more target streaming storage nodes. For example, data can be stored on one target streaming storage node and backed up on other target streaming storage nodes.

[0096] When the storage demand indicates pure batch storage, the client determines the target batch storage node from among the storage nodes. In one implementation, the client 102 can send the storage demand indicating pure batch storage and the first target data identifier corresponding to the data to be stored to the service node 101. Based on the received storage demand and the first target data identifier corresponding to the data to be stored, the service node 101 determines the storage node corresponding to the first target data identifier of the data to be stored, i.e., the target batch storage node, and sends the node information of the determined target batch storage node to the client. In this way, the client 102 can obtain the target batch storage node allocated by the service node 101.

[0097] In one possible implementation, client 102 can send the data to be stored to a target batch storage node, which then stores the data.

[0098] In one possible implementation, both the stored data and data replicas reside on the batch storage nodes, using batch data storage algorithms such as HDFS (Hadoop Distributed File System), without any specific limitations.

[0099] For pure batch storage, there can be one or more target batch storage nodes. For example, data can be stored on one target batch storage node and backed up on other target batch storage nodes.

[0100] When a storage demand indicates integrated streaming and batch storage, the client determines the target streaming storage node and the target batch storage node from among the storage nodes. In one implementation, the client 102 can send the storage demand indicating integrated streaming and batch storage and the first target data identifier corresponding to the data to be stored to the service node 101. Based on the received storage demand and the first target data identifier corresponding to the data to be stored, the service node 101 determines the storage nodes corresponding to the first target data identifier of the data to be stored, namely the target streaming storage node and the target batch storage node, and sends the node information of the determined target streaming storage node and target batch storage node to the client. In this way, the client 102 can obtain the target streaming storage node and target batch storage node allocated by the service node 101; and send the data to be stored to the target streaming storage node. The target streaming storage node receives the data to be stored sent by the client, stores the data to be stored in a streaming storage mode; and synchronizes the data to be stored to the target batch storage node. The target batch storage node is used to store a copy of the data to be stored in a batch storage mode.

[0101] Since there is more than one piece of data to be stored, different pieces of data will have different data identifiers. Understandably, a single storage node can store multiple pieces of data. For example, storage node 1 can store data A, B, and C. Therefore, the correspondence can be that multiple data identifiers correspond to one storage node. Furthermore, during storage, because data is backed up, the same data may exist on multiple different storage nodes. For example, data A may be stored on storage node 1, backed up on storage node 2, and backed up on storage node 3. Therefore, the correspondence can also be that one data identifier corresponds to multiple storage nodes. Thus, the correspondence can include multiple data identifiers and multiple storage nodes, that is, it includes multiple target streaming storage nodes and / or target batch storage nodes corresponding to multiple data identifiers.

[0102] In one possible implementation, the client 102 can select the storage node of the streaming storage tag from each storage node, and then arbitrarily select one or more of the storage nodes of the streaming storage tag as the target streaming storage node.

[0103] Client 102 can select the storage node of the batch storage tag from each storage node, and then arbitrarily select one or more of the storage nodes of the batch storage tag as the target batch storage node.

[0104] If client 102 arbitrarily selects a target streaming storage node and / or a target streaming storage node, after storing the data, it can record the correspondence between the data and the target streaming storage node and / or the target streaming storage node. The correspondence includes multiple target streaming storage nodes and / or target batch storage nodes corresponding to data identifiers. The correspondence between the data and the target streaming storage node and / or the target streaming storage node can be sent to service node 101. In this way, when reading data, the client can directly look up the target streaming storage node and / or the target streaming storage node corresponding to the data from the service node and quickly read the data from the target streaming storage node and / or the target streaming storage node.

[0105] Data storage typically requires backups, and related technologies often employ independent systems for streaming and batch storage. Thus, for a single piece of data, the technology requires storing both the data and its copies in a streaming system, and also in a batch system, typically with three copies to ensure high availability. For example, storing three copies of the same data in a streaming system guarantees high availability there, and storing three copies in a batch system guarantees high availability there as well. Therefore, for the same piece of data, a total of six copies may be required.

[0106] In this embodiment of the invention, the integrated stream-batch storage stores data in a streaming storage manner on streaming storage nodes and stores copies of the data in a batch storage manner on batch storage nodes. This allows batch storage nodes to back up data stored in a streaming storage manner, reducing redundant backups and lowering storage costs. For example, for a single piece of data, one copy can be stored on a streaming storage node and one copy on a batch storage node. Alternatively, to ensure high availability, three copies can be stored: two copies on a streaming storage node and one copy on a batch storage node, or one copy on a streaming storage node and two copies on a batch storage node. Thus, in this embodiment of the invention, the data storage system can satisfy both streaming and batch storage scenarios and ensure high availability with only three copies. Compared to existing technologies that require six copies to achieve both scenarios, the storage method of the data storage system in this embodiment of the invention can significantly reduce storage costs.

[0107] Understandably, data stored via streaming is ordered, while data stored via batch is unordered. Therefore, for integrated streaming and batch storage, if data is first stored in batch mode, the data to be stored will become unordered. In this case, even if the data is then stored via streaming, the data cannot become ordered. Therefore, the integrated streaming and batch storage process only includes the process of storing data via streaming and backing up data via batch; there is no process of storing data via batch and backing up data via streaming.

[0108] In an optional embodiment, service node 101 is further configured to determine the streaming storage node and / or batch storage node corresponding to each data identifier based on the storage method label of each storage node; and save the correspondence between each data identifier and the corresponding streaming storage node and / or batch storage node.

[0109] The client is specifically used to determine the target streaming storage node and / or target batch storage node corresponding to the first target data identifier from the correspondence according to the storage requirements. The first target data identifier is the data identifier corresponding to the data to be stored.

[0110] In this embodiment of the invention, the client 102 saves the data to the corresponding streaming storage node and / or batch storage node according to the pre-determined correspondence between the data identifier and each storage node. Furthermore, during the data reading process, data can also be read from each storage node according to the pre-determined correspondence between the data identifier and each storage node.

[0111] Service node 101 can divide the storage nodes corresponding to different data identifiers. For example, different data identifiers can have their own corresponding streaming storage nodes and / or batch storage nodes.

[0112] Data identifiers can be categorized based on data type, with different data types corresponding to different identifiers. Data type can be the type of data itself, such as different types based on data size; or, data type can be the business type corresponding to the data, with different business scenarios corresponding to different data identifiers.

[0113] In this way, the client 102 can obtain the correspondence between each data identifier and the corresponding streaming storage node and / or batch storage node from the service node 101, and then determine the target streaming storage node and / or target batch storage node corresponding to the first target data identifier from the correspondence according to the storage requirements.

[0114] In one possible implementation, the data to be stored can carry a data identifier, namely a first target data identifier. In this way, the correspondence between each data identifier and the corresponding streaming storage node and / or batch storage node can be found based on the first target data identifier, and the streaming storage node and / or batch storage node corresponding to the first target data identifier can be obtained.

[0115] In another possible implementation, after receiving the data to be stored, the data identifier corresponding to the data to be stored, i.e., the first target data identifier, can be determined. Then, based on the first target data identifier, the correspondence between each data identifier and the corresponding streaming storage node and / or batch storage node can be found, thereby obtaining the streaming storage node and / or batch storage node corresponding to the first target data identifier.

[0116] Specifically, according to storage requirements, the streaming storage node and / or batch storage node corresponding to the first target data identifier can be obtained. If the storage requirement indicates pure streaming storage, the streaming storage node corresponding to the first target data identifier is obtained; if the storage requirement indicates pure batch storage, the batch storage node corresponding to the first target data identifier is obtained; if the storage requirement indicates integrated streaming and batch storage, the streaming storage node and batch storage node corresponding to the first target data identifier are obtained.

[0117] In one example, different data identifiers represent different files, and service node 101 allocates storage nodes corresponding to different data identifiers, that is, it allocates storage nodes corresponding to different files.

[0118] The client receives the data to be stored, which may carry the target file where the data is expected to be stored, and obtains the first target data identifier representing the target file to be stored.

[0119] In specific application scenarios, the data storage system using embodiments of this invention can specifically adopt a master-worker distributed architecture. For example... Figure 2 As shown.

[0120] The master node (Master) 201, also known as the service node, is a single node used for data management and worker node management. It assigns and records labels for each worker node, records data metadata, and records the worker node where the data resides.

[0121] The storage service supports distinguishing storage nodes by streaming or batch tags, that is, assigning streaming storage tags or batch storage tags to storage nodes.

[0122] The tags can be either streaming storage tags or batch storage tags; the metadata of the data is the descriptive information of the data file, which may include, for example, the creator, creation time, permissions, format, etc.

[0123] Worker nodes are storage nodes in a data storage system, used for data storage and reading / writing. They are categorized based on different tags assigned to the master node (201), including streaming storage nodes and batch storage nodes, such as... Figure 2 As shown, streaming storage nodes can also be called streaming nodes, and batch storage nodes can also be called batch nodes. Worker nodes can specifically include:

[0124] Stream node-1 (Worker-1) is used for streaming data storage and streaming data reading and writing. The stored data can include data A and data B. More specifically, it can include records-1,-2, and-3 of data A and records-1,-2, and-3 of data B.

[0125] Stream node-2 (Worker-2) is used for streaming data storage and streaming data reading and writing. The stored data can include data A and data D. More specifically, it can include records-1,-2, and-3 of data A and records-1,-2, and-3 of data D.

[0126] Understandably, data A in stream node-2 (Worker-2) can be a copy of data A in stream node-1 (Worker-1). Conversely, data A in stream node-1 (Worker-1) can also be understood as a copy of data A in stream node-2 (Worker-2), meaning that data can exist as copies on multiple stream nodes.

[0127] Batch node-3 (Worker-3) is used for batch data storage and batch data reading and writing. It can store data D-block 1, data D-block 2, data E-block 1, and data E-block 2.

[0128] Batch node-4 (Worker-4) is used for batch data storage and batch data reading and writing. It can store data D-block 3, data D-block 4, data E-block 1, and data E-block 2.

[0129] Understandably, blocks 1, 2, 3, and 4 of data D in batch nodes-3 (Worker-3) and-4 (Worker-4) can be backups of data D in stream node-2 (Worker-2). Similarly, data E in batch node-4 (Worker-4) can be a copy of data E in batch node-3 (Worker-3). Conversely, data E in batch node-3 (Worker-3) can also be understood as a copy of data E in batch node-4 (Worker-4). In other words, the same data can be stored separately on multiple batch nodes.

[0130] Understandably, in practical application scenarios, stream nodes and batch nodes may include, but are not limited to, the above-mentioned stream node-1, stream node-2, batch node-1 and batch node-2; the data in stream nodes and batch nodes may include, but are not limited to, the above-mentioned data, and the data records may include, but are not limited to, record-1, record-2 and record-3; the data blocks may include, but are not limited to, block 1, block 2, block 3 and block 4.

[0131] The technical solution of this invention involves a master node centrally managing each worker node, assigning tags to each worker node, and enabling each worker node to provide corresponding storage data strategies. This integrates streaming and batch storage into a single system, while simultaneously implementing three storage strategies: pure streaming, pure batch, and integrated streaming and batch. This reduces the maintenance costs of the storage service.

[0132] In related technologies, streaming storage and batch storage need to be maintained separately to distinguish between them. The technical solution provided in this invention uses service nodes to tag storage nodes with streaming and batch labels, isolating streaming and batch read / write operations. It can simultaneously satisfy both streaming and batch storage scenarios. That is, the service nodes uniformly manage storage nodes, linking streaming and batch storage to achieve on-demand storage and reduce the maintenance, usage, and storage costs of storage services. In streaming scenarios, clients connect to streaming nodes to read and write data; in batch scenarios, clients connect to batch nodes to read and write data. Data read / write operations in the two scenarios are mutually isolated, and large-scale data writing or reading in batch mode does not affect the read / write latency of streaming data.

[0133] This invention also provides a data storage method that can be applied to a client. During data storage, the client can be understood as writing to the client. For example... Figure 3 As shown, the data storage method provided in this embodiment of the invention may include:

[0134] S301 receives the data to be stored and the storage requirement; the storage requirement is determined based on the application scenario of the data to be stored and is used to indicate pure streaming storage, pure batch storage, or integrated streaming and batch storage; pure streaming storage stores the data to be stored only through streaming storage nodes; pure batch storage stores the data to be stored only through batch storage nodes; integrated streaming and batch storage stores the data to be stored through both streaming storage nodes and batch storage nodes.

[0135] S302, according to storage requirements, determine the target streaming storage node and / or target batch storage node from each storage node; wherein, the target streaming storage node includes storage nodes with the storage method label of streaming storage, and the target batch storage node includes storage nodes with the storage method label of batch storage; the streaming storage label indicates that the storage node uses the streaming storage method to store data, and the batch storage label indicates that the storage node uses the batch storage method to store data;

[0136] S303 stores the data to be stored to the target streaming storage node and / or the target batch storage node.

[0137] In this embodiment of the invention, after receiving the data to be stored and the storage requirements, the client can determine the target streaming storage node and / or the target batch storage node from among the various storage nodes. Thus, the data to be stored is stored through the target streaming storage node and / or the target batch storage node. The storage requirements indicate pure streaming storage, pure batch storage, or integrated streaming-batch storage. This allows for data storage tailored to different storage needs, avoiding the need to maintain two independent systems simultaneously for streaming and batch storage, thereby reducing the maintenance costs of the storage service. Furthermore, this embodiment of the invention can not only implement pure streaming storage and pure batch storage, but also integrated streaming-batch storage, that is, data can be stored both streamingly and in batches.

[0138] In streaming scenarios, clients connect to streaming storage nodes to write data; in batch scenarios, clients connect to batch storage nodes to write data. Data writes in these two scenarios are isolated from each other, so writing large amounts of data in batch mode does not affect the write latency of streaming data.

[0139] In an optional embodiment, S302 may include:

[0140] Obtain the correspondence between each data identifier and streaming storage nodes and / or batch storage nodes; wherein, the correspondence includes multiple target streaming storage nodes and / or target batch storage nodes corresponding to data identifiers, and according to storage requirements, obtain the target streaming storage node and / or target batch storage node corresponding to the first target data identifier from the correspondence; wherein, the first target data identifier is the data identifier corresponding to the data to be stored.

[0141] In one alternative embodiment, storage requirements are used to indicate stream-batch integrated storage;

[0142] S302 may include: determining, from each storage node, the target streaming storage node and the target batch storage node corresponding to the first target data identifier, according to the storage requirements used to indicate the integrated streaming and batch storage;

[0143] S303 may include: sending the data to be stored to the target streaming storage node so that the target streaming storage node stores the data to be stored, and synchronizing the data to be stored to the target batch storage node.

[0144] For pure streaming storage, both the stored data and backups are stored on the streaming storage nodes; for pure batch storage, both the stored data and backups are stored on the batch storage nodes; for integrated streaming and batch storage, the stored data is stored on the streaming storage nodes, and the backups are stored on the batch storage nodes.

[0145] In an optional embodiment, S302 may include:

[0146] The storage requirement and the first target data identifier corresponding to the data to be stored are sent to the service node. The service node is used to determine the target streaming storage node and / or target batch storage node corresponding to the first target data identifier according to the storage requirement, the first target data identifier corresponding to the data to be stored, and the storage method label of each storage node. The service node receives the node information of the target streaming storage node and / or target batch storage node corresponding to the first target data identifier. The node information is used by the client to send the data to be stored to the target streaming storage node and / or target batch storage node so that the target streaming storage node and / or target batch storage node can store the data.

[0147] Client 102 can send its storage requirements and the first target data identifier corresponding to the data to be stored to service node 101. Service node 101, based on the storage requirements, the first target data identifier corresponding to the data to be stored, and the storage method tags of each storage node, determines the target streaming storage node and / or target batch storage node corresponding to the first target data identifier, and sends the node information of the target streaming storage node and / or target batch storage node corresponding to the first target data identifier to client 102. In this way, client 102 can obtain the target streaming storage node and / or target batch storage node allocated by service node 101. That is, the client only needs to send the data storage requirements to the service node, the service node allocates and records the correspondence between data and storage nodes based on the data storage requirements, and then returns it to the client. The client writes or reads data according to the returned storage node information.

[0148] In specific application scenarios, such as Figure 4 As shown, during the data writing process of the stream-batch integrated storage:

[0149] Write client 401 to send data storage requests to the master node, receive the target streaming storage node and target batch storage node returned by the master node, and send the data to be stored to the target streaming storage node.

[0150] The client 401 writes to obtain the data identifier of the data to be stored A. Based on the data identifier, the target storage file of the data to be stored A can be obtained.

[0151] The master node 201 is used to store the correspondence between multiple data identifiers and storage nodes, receive the data to be stored and the storage requirements, and obtain the target streaming storage node and target batch storage node corresponding to the first target data identifier from the correspondence according to the storage requirements used to indicate the integrated streaming and batch storage.

[0152] After the writing client 401 determines the target storage file of the data to be stored, the master node 201 finds the storage node where the target storage file is located and transmits the information of the storage node back to the writing client 401. In this way, the storage node corresponding to the data identifier is obtained, and then the client 102 sends the data A to be stored to the specified storage node for storage.

[0153] Stream node 402, also known as a streaming storage node, is used to receive the data to be stored sent by the write client 401, store the data to be stored in the streaming storage mode, and synchronize the data to be stored to batch node 403.

[0154] like Figure 4 The stream node 402 can be the target storage node for the data A to be stored. Specifically, it can store records-1,-2 and-3 of data A.

[0155] Batch node 403, also known as batch storage node, is used to receive data copies of the data to be stored sent by stream node 402 and store the data to be stored in a batch storage manner.

[0156] When batch node 403 stores data A, it can divide data A into different blocks based on data replicas, such as data A-block 1 and data A-block 2, and store data A synchronously in the form of data blocks.

[0157] The technical solution provided in this invention provides a detailed description of the storage process for integrated stream and batch storage. When the storage requirement is batch storage, according to the storage requirement indicating integrated stream and batch storage, the target streaming storage node and target batch storage node corresponding to the first target data identifier are obtained from the correspondence. The data to be stored is sent to the target streaming storage node so that the target streaming storage node stores the data to be stored and synchronizes the data to be stored to the target batch storage node. This integrates three storage strategies—pure streaming storage, pure batch storage, and integrated stream and batch storage—into a single storage system. This avoids using multiple systems for streaming and batch storage separately, which increases the maintenance cost of the storage service. Therefore, the maintenance cost of the storage service can be reduced.

[0158] This invention also provides a data reading method that can be applied to a client. During the data reading process, the client can be understood as reading from another client. For example... Figure 5 As shown, the data reading method provided in this embodiment of the invention may include:

[0159] S501, Receive data access request. The data access request carries data reading requirements. The data reading requirements are determined based on the application scenario of the data to be accessed indicated by the data access request, and are used to indicate streaming reading or batch reading.

[0160] S502, if the data read request is used to indicate streaming read, then the streaming storage node is determined from each storage node, and the data is read from the streaming storage node; if the data read request is used to indicate batch read, then the batch storage node is determined from each storage node, and the data is read from the batch storage node.

[0161] Among them, streaming storage nodes include storage nodes with the storage method label of streaming storage, and batch storage nodes include storage nodes with the storage method label of batch storage. The streaming storage label indicates that the storage node uses the streaming storage method to store data, and the batch storage label indicates that the storage node uses the batch storage method to store data. Pure streaming storage stores the data to be stored only through streaming storage nodes. Pure batch storage stores the data to be stored only through batch storage nodes. Integrated streaming and batch storage stores the data to be stored through both streaming storage nodes and batch storage nodes.

[0162] If the data reading requirement is used to indicate streaming reading, the reading client can be understood as a streaming reading client. This streaming reading client determines the streaming storage node from each storage node, establishes a connection with the streaming storage node, and reads data from the streaming storage node through the reading method corresponding to the streaming storage method, that is, reading data streaming.

[0163] If the data reading requirement is used to indicate batch reading, the reading client can be understood as a batch reading client. This batch reading client determines the batch storage node from each storage node, establishes a connection with the batch storage node, and reads data from the batch storage node through the reading method corresponding to the batch storage method, that is, reads data in batches.

[0164] In this embodiment of the invention, there is no need to maintain two separate systems to support streaming and batch data reading. Data can be read from the corresponding streaming or batch storage node according to actual data reading needs, reducing the maintenance cost of data reading services. Furthermore, since the streaming and batch storage nodes are independent of each other, the streaming and batch reading processes do not affect each other; the reading process of one will not cause delays in the other. Thus, data reading is more stable while reducing the maintenance cost of storage services.

[0165] In streaming scenarios, clients connect to streaming storage nodes to read data; in batch scenarios, clients connect to batch storage nodes to read data. Data reading in the two scenarios is isolated from each other, and reading large amounts of data in batch mode does not affect the latency of streaming data reading.

[0166] In an optional embodiment, the data access request further carries a second target data identifier, which is the data identifier corresponding to the data to be accessed indicated by the data access request. The data reading method further includes:

[0167] Obtain the mapping relationship between multiple data identifiers and storage nodes. The mapping relationship includes the target streaming storage nodes and / or target batch storage nodes corresponding to the multiple data identifiers.

[0168] In S502, the streaming storage node is determined from each storage node, including:

[0169] Based on the correspondence, determine the streaming storage node corresponding to the second target data identifier from each storage node;

[0170] In S502, batch storage nodes are determined from each storage node, including:

[0171] Based on the correspondence, determine the batch storage node corresponding to the second target data identifier from each storage node.

[0172] In the aforementioned data storage method, the correspondence between each data identifier and its corresponding streaming storage node and / or batch storage node is obtained. According to the storage requirements, the target streaming storage node and / or target batch storage node corresponding to the first target data identifier is obtained from the correspondence. The data to be stored is stored in the target streaming storage node and / or target batch storage node. During the data reading process, the correspondence between multiple data identifiers and storage nodes can be obtained. Based on the correspondence, the streaming storage node corresponding to the second target data identifier is determined from each storage node. Alternatively, based on the correspondence, the batch storage node corresponding to the second target data identifier is determined from each storage node to read data from the streaming storage node or the batch storage node.

[0173] like Figure 6 As shown, the data reading process in a specific application scenario is as follows:

[0174] The streaming client 601 sends a data reading request to the master node 201, receives the target streaming storage node returned by the master node 201, and reads data from the target streaming storage node.

[0175] Batch read client 602 is used to send data read requests to the master node, receive the target batch storage node returned by the master node, and read data from the target batch storage node.

[0176] The master node 201, also known as the service node, is used to store the correspondence between multiple data identifiers and storage nodes, receive the data to be read and the read request, and obtain the target streaming storage node or target batch storage node corresponding to the second target data identifier from the correspondence according to the read request.

[0177] The data identifier provides the file information for the stored data. The master node 201 queries the storage node where the stored file is located and transmits the information of that storage node to the reading client. This allows the reading client to access and read the data from the storage node containing the data to be read.

[0178] For example, a streaming client obtains the corresponding stream node 402 based on the data identifier of data A, and accesses stream node 402 to read records about data A; a batch reading client obtains the corresponding batch node 403 based on the data identifier of data A, and accesses batch node 403 to read blocks about data A.

[0179] Stream node 402, or streaming storage node, is used to store streaming data. Specifically, it can be used to store records-1,-2, and-3 of data A.

[0180] Batch node 403, or batch storage node, is used to store batch data. Specifically, it can be used to store data block A-1 and data block A-2.

[0181] The technical solution provided in this invention describes in detail the process of reading data in streaming and batch reading. Streaming reading only reads data from streaming storage nodes, and batch reading only reads data from batch storage nodes. The two reading methods should not affect each other. In this way, the data reading process is more stable and the usage cost is reduced while reducing the maintenance cost of storage services.

[0182] In this embodiment of the invention, the client connects to the streaming node to read and write data in the streaming scenario; in the batch scenario, the client connects to the batch node to read and write data. Data reading and writing in the two scenarios are isolated from each other, so that the read and write latency of streaming data is not affected when writing or reading large amounts of data in batch mode.

[0183] like Figure 7 As shown, an embodiment of the present invention provides a data storage device, including:

[0184] The receiving module 701 is used to receive the data to be stored and the storage requirements; the storage requirements are determined based on the application scenario of the data to be stored and are used to indicate pure streaming storage, pure batch storage, or integrated streaming and batch storage; pure streaming storage stores the data to be stored only through streaming storage nodes; pure batch storage stores the data to be stored only through batch storage nodes; integrated streaming and batch storage stores the data to be stored through both streaming storage nodes and batch storage nodes.

[0185] The determination module 702 is used to determine target streaming storage nodes and / or target batch storage nodes from each storage node according to storage requirements; wherein, the target streaming storage nodes include storage nodes with the storage method label of streaming storage, and the target batch storage nodes include storage nodes with the storage method label of batch storage; the streaming storage label indicates that the storage method used by the storage node to store data is streaming storage, and the batch storage label indicates that the storage method used by the storage node to store data is batch storage;

[0186] Storage module 703 is used to store data to a target streaming storage node and / or a target batch storage node.

[0187] Optionally, the determining module 702 is specifically used to send the storage requirement and the first target data identifier corresponding to the data to be stored to the service node. The service node is used to determine the target streaming storage node and / or target batch storage node corresponding to the first target data identifier according to the storage requirement, the first target data identifier corresponding to the data to be stored, and the storage method label of each storage node; and to receive the node information of the target streaming storage node and / or target batch storage node corresponding to the first target data identifier. The node information is used by the client to send the data to be stored to the target streaming storage node and / or target batch storage node so that the target streaming storage node and / or target batch storage node can store it.

[0188] Optionally, storage requirements are used to indicate stream-batch integrated storage;

[0189] The determination module 702 is used to determine the target streaming storage node and the target batch storage node corresponding to the first target data identifier from each storage node according to the storage requirements used to indicate the integrated streaming and batch storage;

[0190] Storage module 703 is used to send the data to be stored to the target streaming storage node so that the target streaming storage node stores the data to be stored and synchronizes the data to be stored to the target batch storage node.

[0191] like Figure 8 As shown, an embodiment of the present invention provides a data reading device, including:

[0192] The receiving module 801 is used to receive data access requests. The data access requests carry data reading requirements, which are determined based on the application scenario of the data to be accessed indicated by the data access requests, and are used to indicate streaming reading or batch reading.

[0193] The read module 802 is used to determine the streaming storage node from each storage node and read data from the streaming storage node if the data read request indicates streaming read; and to determine the batch storage node from each storage node and read data from the batch storage node if the data read request indicates batch read. Streaming storage nodes include storage nodes with the storage method label "streaming storage," and batch storage nodes include storage nodes with the storage method label "batch storage." The streaming storage label indicates that the storage node uses streaming storage, and the batch storage label indicates that the storage node uses batch storage. Pure streaming storage stores the data to be stored only through streaming storage nodes; pure batch storage stores the data to be stored only through batch storage nodes; and integrated streaming and batch storage stores the data to be stored through both streaming and batch storage nodes.

[0194] Optionally, the data access request further carries a second target data identifier, which is the data identifier corresponding to the data to be accessed indicated by the data access request. The device also includes:

[0195] The acquisition module is used to acquire the correspondence between multiple data identifiers and storage nodes. The correspondence includes the target streaming storage nodes and / or target batch storage nodes corresponding to the multiple data identifiers.

[0196] The reading module 802 is specifically used to determine, based on the correspondence, the streaming storage node corresponding to the second target data identifier from each storage node, or, based on the correspondence, the batch storage node corresponding to the second target data identifier from each storage node.

[0197] This invention also provides an electronic device, such as... Figure 9 As shown, it includes a processor 901, a communication interface 902, a memory 903, and a communication bus 904, wherein the processor 901, the communication interface 902, and the memory 903 communicate with each other through the communication bus 904.

[0198] Memory 903 is used to store computer programs;

[0199] The processor 901, when executing a program stored in the memory 903, implements the above-described data storage method or data reading method steps.

[0200] The communication bus mentioned above can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not mean that there is only one bus or one type of bus.

[0201] The communication interface is used for communication between the aforementioned terminal and other devices.

[0202] The memory may include random access memory (RAM) or non-volatile memory, such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0203] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0204] In another embodiment of the present invention, a computer-readable storage medium is also provided, wherein a computer program is stored therein, and when the computer program is executed by a processor, it implements any of the data storage methods or data reading methods described in the above embodiments.

[0205] In another embodiment of the present invention, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to execute any of the data storage method or data reading method described in the above embodiments.

[0206] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid state disk (SSD)).

[0207] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0208] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the embodiments of methods, apparatus, electronic devices, and computer storage media are basically similar to the system embodiments, so the descriptions are relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0209] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention are included within the scope of protection of the present invention.

Claims

1. A data storage system, characterized in that, The data storage system includes: clients, service nodes, and storage nodes; The service node is used to store the storage method tags of each storage node. The storage method tags include streaming storage tags and batch storage tags. The streaming storage tag indicates that the storage node uses streaming storage to store data, and the batch storage tag indicates that the storage node uses batch storage to store data. The client is used to receive data to be stored and storage requirements; the storage requirements are determined based on the application scenario of the data to be stored, and are used to indicate pure streaming storage, pure batch storage, or integrated streaming and batch storage; pure streaming storage stores the data to be stored only through streaming storage nodes; pure batch storage stores the data to be stored only through batch storage nodes; integrated streaming and batch storage stores the data to be stored through both streaming and batch storage nodes; according to the storage requirements, a target streaming storage node and / or a target batch storage node are determined from each storage node; wherein, the target streaming storage node includes storage nodes with the storage method label of streaming storage, and the target batch storage node includes storage nodes with the storage method label of batch storage; the data to be stored is stored to the target streaming storage node and / or the target batch storage node; The target streaming storage node and / or the target batch storage node are used to store the data to be stored.

2. The system according to claim 1, characterized in that, The client is specifically used to send the storage requirement and the first target data identifier corresponding to the data to be stored to the service node; The service node is specifically used to determine the target streaming storage node and / or target batch storage node corresponding to the first target data identifier according to the storage requirements, the first target data identifier corresponding to the data to be stored, and the storage method label of each storage node, and to send the node information of the target streaming storage node and / or target batch storage node corresponding to the first target data identifier to the client. The node information is used by the client to send the data to be stored to the target streaming storage node and / or target batch storage node so that the target streaming storage node and / or target batch storage node can store it. The client is specifically used to receive the node information.

3. The system according to claim 1, characterized in that, The storage requirements are used to indicate integrated batch and stream storage; The client is specifically used to determine the target streaming storage node and the target batch storage node from each storage node according to the storage requirements for indicating integrated streaming and batch storage. The data to be stored is sent to the target streaming storage node; The target streaming storage node is specifically used to receive the data to be stored sent by the client and store the data to be stored in a streaming storage manner. And synchronize the data to be stored to the target batch storage node; The target batch storage node is used to store copies of the data to be stored in a batch storage manner.

4. A data storage method, characterized in that, The method includes: The system receives data to be stored and storage requirements; the storage requirements are determined based on the application scenario of the data to be stored and are used to indicate pure streaming storage, pure batch storage, or integrated streaming and batch storage; pure streaming storage stores the data to be stored only through streaming storage nodes; pure batch storage stores the data to be stored only through batch storage nodes; integrated streaming and batch storage stores the data to be stored through both streaming storage nodes and batch storage nodes. According to the storage requirements, target streaming storage nodes and / or target batch storage nodes are determined from each storage node; wherein, the target streaming storage node includes storage nodes with the storage method label of streaming storage, and the target batch storage node includes storage nodes with the storage method label of batch storage; the streaming storage label indicates that the storage method used by the storage node to store data is streaming storage, and the batch storage label indicates that the storage method used by the storage node to store data is batch storage; The data to be stored is stored in the target streaming storage node and / or the target batch storage node.

5. The method according to claim 4, characterized in that, The step of determining the target streaming storage node and / or target batch storage node from each storage node according to the storage requirements includes: The storage requirement and the first target data identifier corresponding to the data to be stored are sent to the service node. The service node is used to determine the target streaming storage node and / or target batch storage node corresponding to the first target data identifier according to the storage requirement, the first target data identifier corresponding to the data to be stored, and the storage method label of each storage node. The system receives node information of the target streaming storage node and / or target batch storage node corresponding to the first target data identifier. The node information is used by the client to send the data to be stored to the target streaming storage node and / or target batch storage node so that the target streaming storage node and / or target batch storage node can store the data.

6. The method according to claim 4, characterized in that, The storage requirements are used to indicate integrated batch and stream storage; The step of determining the target streaming storage node and / or target batch storage node from each storage node according to the storage requirements includes: According to the storage requirements used to indicate the integrated streaming and batch storage, the target streaming storage node and the target batch storage node corresponding to the storage requirements and the first target data identifier corresponding to the data to be stored are determined from each storage node; The step of storing the data to be stored to the target streaming storage node and / or the target batch storage node includes: The data to be stored is sent to the target streaming storage node so that the target streaming storage node stores the data to be stored and synchronizes the data to be stored to the target batch storage node.

7. A data reading method, characterized in that, The method includes: Receive a data access request, the data access request carrying a data reading requirement, the data reading requirement being determined based on the application scenario of the data to be accessed indicated by the data access request, and used to indicate streaming reading or batch reading; If the data read request is used to indicate streaming read, then a streaming storage node is determined from each storage node, and data is read from the streaming storage node; If the data read request is used to indicate batch read, then the batch storage node is determined from each storage node, and data is read from the batch storage node; The streaming storage node includes storage nodes with a storage method label of streaming storage, and the batch storage node includes storage nodes with a storage method label of batch storage. The streaming storage label indicates that the storage method used by the storage node to store data is streaming storage, and the batch storage label indicates that the storage method used by the storage node to store data is batch storage. Pure streaming storage stores the data to be stored only through the streaming storage node; pure batch storage stores the data to be stored only through the batch storage node; and integrated streaming and batch storage stores the data to be stored through both the streaming storage node and the batch storage node.

8. The method according to claim 7, characterized in that, The data access request also carries a second target data identifier, which is the data identifier corresponding to the data to be accessed indicated by the data access request. The method further includes: Obtain the correspondence between multiple data identifiers and storage nodes, wherein the correspondence includes the target streaming storage nodes and / or target batch storage nodes corresponding to the multiple data identifiers; The step of determining the streaming storage node from each storage node includes: Based on the correspondence, determine the streaming storage node corresponding to the second target data identifier from each storage node; The step of determining the batch storage nodes from each storage node includes: Based on the correspondence, the batch storage node corresponding to the second target data identifier is determined from each storage node.

9. A data storage device, characterized in that, The device includes: A receiving module is used to receive data to be stored and storage requirements; the storage requirements are determined based on the application scenario of the data to be stored and are used to indicate pure streaming storage, pure batch storage, or integrated streaming and batch storage; pure streaming storage stores the data to be stored only through streaming storage nodes; pure batch storage stores the data to be stored only through batch storage nodes; integrated streaming and batch storage stores the data to be stored through both streaming storage nodes and batch storage nodes. The determination module is used to determine target streaming storage nodes and / or target batch storage nodes from each storage node according to the storage requirements; wherein, the target streaming storage node includes storage nodes with a storage method label of streaming storage, and the target batch storage node includes storage nodes with a storage method label of batch storage; the streaming storage label indicates that the storage method used by the storage node to store data is streaming storage, and the batch storage label indicates that the storage method used by the storage node to store data is batch storage; A storage module is used to store the data to be stored to the target streaming storage node and / or the target batch storage node.

10. A data reading device, characterized in that, The device includes: A receiving module is used to receive data access requests, which carry data reading requirements. The data reading requirements are determined based on the application scenario of the data to be accessed indicated by the data access request, and are used to indicate streaming reading or batch reading. A read module is configured to: if the data read request indicates streaming read, determine a streaming storage node from each storage node and read data from the streaming storage node; if the data read request indicates batch read, determine a batch storage node from each storage node and read data from the batch storage node; wherein, the streaming storage node includes storage nodes with a storage method label of streaming storage, and the batch storage node includes storage nodes with a storage method label of batch storage; the streaming storage label indicates that the storage method used by the storage node to store data is streaming storage, and the batch storage label indicates that the storage method used by the storage node to store data is batch storage; pure streaming storage stores the data to be stored only through the streaming storage node; pure batch storage stores the data to be stored only through the batch storage node; and integrated streaming and batch storage stores the data to be stored through both the streaming storage node and the batch storage node.

11. An electronic device, characterized in that, It includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, when executing a program stored in memory, implements the steps of the method described in any one of claims 4-8.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the method described in any one of claims 4-8.

Images