Data storage method, device and equipment based on streaming data and storage medium

Abstract

The invention provides a data storage method, device and equipment based on streaming data and a storage medium, and relates to the field of computers, in particular to the field of object storage and streaming storage. According to the specific implementation scheme, the method comprises the steps of obtaining a to-be-stored data stream, and performing segmentation processing on the to-be-stored data stream to obtain a plurality of data segments; wherein the plurality of data segments form the data stream to be stored, and the data segments are divided into a plurality of data fragments; determining a storage unit corresponding to the data segment; wherein the storage unit is a part of a storage space of a disk; and storing the data segments into the storage unit according to the arrangement sequence of the data fragments in the data segments. The streaming data can be effectively managed, and the data storage efficiency and precision are improved.

Description

Technical Field

[0001] The present disclosure relates to the field of object storage and streaming storage in the computer field, and in particular to a data storage method, device, equipment and storage medium based on streaming data. Background Art

[0002] With the increasing popularity of big data cloud computing and AI (Artificial Intelligence) training, users have higher requirements for storage throughput performance. Currently, when storing data, EC (Error Correction) can be performed on the data to divide the data into smaller fragments for storage.

[0003] However, this approach will result in small IO (Input / Output) and fragmented storage, which is difficult to meet the needs of high-throughput scenarios and affects storage efficiency and accuracy. Summary of the invention

[0004] The present invention provides a data storage method, device, equipment and storage medium based on streaming data.

[0005] According to a first aspect of the present disclosure, a data storage method based on streaming data is provided, comprising:

[0006] Acquire a data stream to be stored, and segment the data stream to be stored to obtain a plurality of data segments; wherein the plurality of data segments constitute the data stream to be stored, and the data segments are divided into a plurality of data fragments;

[0007] Determine a storage unit corresponding to the data segment; wherein the storage unit is a portion of the storage space of the disk;

[0008] The data segment is stored in the storage unit according to the arrangement order of the data slices in the data segment.

[0009] According to a second aspect of the present disclosure, there is provided a data storage device based on streaming data, comprising:

[0010] A segmentation unit, used for acquiring a data stream to be stored, and segmenting the data stream to be stored to obtain a plurality of data segments; wherein the plurality of data segments constitute the data stream to be stored, and the data segments are divided into a plurality of data fragments;

[0011] A determination unit, used to determine a storage unit corresponding to the data segment; wherein the storage unit is a part of the storage space of the disk;

[0012] A storage unit is used to store the data segment in the storage unit according to the arrangement order of the data slices in the data segment.

[0013] According to a third aspect of the present disclosure, there is provided an electronic device, including:

[0014] at least one processor; and

[0015] a memory communicatively coupled to the at least one processor;

[0016] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method described in the first aspect of the present disclosure.

[0017] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause the computer to execute the method according to the first aspect of the present disclosure.

[0018] According to a fifth aspect of the present disclosure, a computer program product is provided, comprising a computer program, which implements the steps of the method described in the first aspect of the present disclosure when executed by a processor.

[0019] According to the technology disclosed in the present invention, effective management of streaming data is achieved, and the efficiency and accuracy of data storage are improved.

[0020] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0022] Figure 1 It is a flow chart of a data storage method based on streaming data provided according to an embodiment of the present disclosure;

[0023] Figure 2 is a schematic diagram of segmenting a data stream to be stored according to an embodiment of the present disclosure;

[0024] Figure 3 is a schematic diagram of the relationship between data segments and storage units provided according to an embodiment of the present disclosure;

[0025] Figure 4 It is a flow chart of a data storage method based on streaming data provided according to an embodiment of the present disclosure;

[0026] Figure 5is a schematic diagram of a truncated data stream to be stored provided according to an embodiment of the present disclosure;

[0027] Figure 6 It is a flow chart of a data storage method based on streaming data provided according to an embodiment of the present disclosure;

[0028] Figure 7 is a schematic diagram of data reading provided according to an embodiment of the present disclosure;

[0029] Figure 8 is a structural block diagram of a data storage device based on streaming data provided according to an embodiment of the present disclosure;

[0030] Fig. 9 is a structural block diagram of a data storage device based on streaming data provided according to an embodiment of the present disclosure;

[0031] Fig.10 is a block diagram of an electronic device for implementing the data storage method based on streaming data according to an embodiment of the present disclosure;

[0032] Fig.11 It is a block diagram of an electronic device used to implement the data storage method based on streaming data according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0033] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0034] With the increasing popularity of big data cloud computing and AI training, users have higher requirements for storage throughput performance. Existing online EC processing is prone to small IO and fragmented storage, especially in HDD (Hard Disk Drive) scenarios, it is difficult to achieve high-throughput data reading and storage behavior, and cannot meet the needs of high-throughput scenarios.

[0035] EC processing can divide a large file into multiple blobs (data shards), each of which corresponds to multiple data blocks and check blocks. Usually, a blob does not exceed 8MB in size. After a one-time EC processing, the blob can be distributed and written to the target node in the preset cluster to complete persistence. However, the excessive number of blobs leads to a waste of IO resources. For example, if a 1MB data is EC-ed according to 8 data blocks plus 4 check blocks, 12 128KB data will be written, and at least 8 128KB data will be read when reading. For HDD, a relatively small IO Size will waste IO capacity significantly. In addition, too many blobs will increase the difficulty of data management, reduce the efficiency and accuracy of data storage, and place higher requirements on reliability.

[0036] The present invention provides a data storage method, device, equipment and storage medium based on streaming data, which are applied to the object storage and streaming storage fields in the computer field to achieve effective management of streaming data and improve the efficiency and accuracy of data storage.

[0037] It should be noted that the data in this embodiment is not targeted at a specific user and cannot reflect the personal information of a specific user. It should be noted that the data in this embodiment comes from a public data set.

[0038] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0039] In order to enable readers to more deeply understand the implementation principle of the present disclosure, the following Figure 1-Figure 11 The embodiment is further refined.

[0040] Figure 1 FIG. 1 is a flow chart of a data storage method based on streaming data provided according to an embodiment of the present disclosure. The method can be executed by a data storage device based on streaming data. Figure 1 As shown, the method comprises the following steps:

[0041] S101, obtaining a data stream to be stored, and segmenting the data stream to be stored to obtain multiple data segments; wherein the multiple data segments constitute the data stream to be stored, and the data segments are divided into multiple data fragments.

[0042] Exemplarily, the data stream to be stored may be a file to be stored, and a user may upload the file to be stored in advance. The file to be stored is acquired, and the file to be stored is stored in the form of streaming data.

[0043] After receiving the file to be stored, the entire file is regarded as a data stream to be stored, and the data stream to be stored is segmented so that the data stream to be stored becomes streaming data connected by multiple data segments. That is, the data stream to be stored is segmented to obtain multiple data segments, and these data segments constitute the data stream to be stored. In this embodiment, the data stream can be called a Stream, and the data segment can be called a Block.

[0044] The number of blocks in the data stream to be stored can be preset, and the data stream to be stored can be segmented according to the preset number to obtain a preset number of blocks. The size of each block can also be preset, and the data stream to be stored can be evenly divided according to the preset size to obtain multiple blocks of the same size. The size of the stream is usually at the GB (gigabyte) level, and the size of the block is usually at the hundreds of megabytes to GB level. In this embodiment, the segmentation method is not specifically limited. It is worth noting that when segmenting, the data stream to be stored needs to be segmented in sequence, that is, starting from the first character of the data stream to be stored until it is segmented to the last character. The blocks are spliced ​​together in sequence to obtain a complete data stream to be stored.

[0045] In this embodiment, it also includes: slicing the data segment to obtain multiple data slices; wherein the multiple data slices constitute the data segment.

[0046] Specifically, after obtaining a Block, each Block may be further divided into smaller components. For each Block, the Block may be sharded into multiple data shards. In this embodiment, the data shards may be referred to as Blobs. For example, EC processing may be performed on a Block to obtain multiple Blobs.

[0047] The number of blobs in a Block can be preset, and the Block stream can be segmented according to the preset number to obtain a preset number of blobs. The size of each Blob can also be preset, and the Block can be evenly divided according to the preset size to obtain multiple blobs of the same size. The size of a Blob is usually in the megabyte range, and the number of blobs in different Blocks can be different. In this embodiment, the segmentation method is not specifically limited. It is worth noting that when segmenting, the Block needs to be segmented in sequence, that is, starting from the first character of the Block until the last character is segmented. The blobs are spliced ​​together in sequence to obtain a complete Block.

[0048] Figure 2 The figure is a schematic diagram of segmenting the data stream to be stored. Figure 2In the data storage system, the data stream to be stored is divided into one or more blocks, and the blocks are divided into one or more blobs. Figure 2 The Stream in can be divided into three blocks, namely Block1, Block2, and Block3. Block1 is divided into three blobs, Block2 is divided into three blobs, and Block3 is divided into one blob.

[0049] The beneficial effect of this setting is that the size concept of Stream is too different from the size concept of Blob. For a 1GB file, a Blob may be 1 megabyte, which means there will be millions of Blobs. Too many Blobs make management difficult and require high reliability. Therefore, Stream can be composed of a series of Blocks, and Block can be composed of a series of Blobs, which reduces the difficulty of management, avoids data confusion, and improves the efficiency and accuracy of data storage.

[0050] S102. Determine a storage unit corresponding to the data segment; wherein the storage unit is a portion of the storage space of the disk.

[0051] Exemplarily, for each data segment, the data segment can be stored in a disk of a computer. The disk can be pre-divided into multiple storage units, that is, each storage unit is a part of the storage space of the disk. Different data segments can be stored in different or the same storage units. In this embodiment, the storage unit can be referred to as a volume or a volume.

[0052] For each data segment, a storage unit can be randomly determined from multiple storage units as the storage unit corresponding to the data segment; the storage unit corresponding to the current data segment can also be selected according to the preset arrangement order of the storage units; the storage unit corresponding to the data segment can also be manually specified. The storage units corresponding to each data segment can be the same or different, but one data segment corresponds to only one storage unit.

[0053] S103. Store the data segment in a storage unit according to the arrangement order of the data fragments in the data segment.

[0054] Exemplarily, the data fragments in the data segment are divided according to the order of the data content in the data segment, and the data fragments in each data segment have their own arrangement order. According to the arrangement order of the data fragments in the data segment, the data segment is stored in the corresponding storage unit. When all the data segments in the data stream to be stored are stored in their corresponding storage units, the storage of the data stream to be stored is completed.

[0055] The data contents in the Blob can be stored in the same Volume in sequence according to the arrangement order of the Blobs in the Block. That is, the Blocks are written sequentially to ensure the physical clustering characteristics of the Blocks and avoid scattered data distribution that affects subsequent data reading. Figure 3 Schematic diagram of the relationship between data segments and storage units. Figure 3 In this example, Block2 is stored in Volume1, and Block1 and Block3 are stored in Volume2.

[0056] In the disclosed embodiment, the data stream to be stored is divided into multiple data segments, and the data segments are further divided into multiple data slices, so as to perform data management of different granularities, reduce the difficulty of management, and realize effective and reliable management of the data stream to be stored. The computer disk is divided into multiple storage units, the storage unit corresponding to the data segment is determined, and sequential writing is performed when storing the data segment according to the order of the data slices, and the content in the data segment is stored in the same storage unit, so as to realize the physical aggregation characteristics of the data segment and improve the efficiency and accuracy of data storage.

[0057] Figure 4 A flowchart of a data storage method based on streaming data is provided in an embodiment of the present disclosure.

[0058] In this embodiment, according to the arrangement order of data shards in the data segment, the data segment is stored in the storage unit, including: determining multiple data units corresponding to the data shards; wherein the data unit includes at least one data block and at least one check block, the number of data blocks corresponding to different data shards is the same, and the number of check blocks corresponding to different data shards is the same; for each data shard, determining the arrangement order of the data units corresponding to the data shard; according to the arrangement order of the data shards in the data segment and the arrangement order of the data units corresponding to each data shard, the data segment is stored in the storage unit.

[0059] This embodiment is based on the above embodiment. Figure 4 As shown, the method comprises the following steps:

[0060] S401, obtaining a data stream to be stored, and segmenting the data stream to be stored to obtain multiple data segments; wherein the multiple data segments constitute the data stream to be stored, and the data segments are divided into multiple data fragments.

[0061] Exemplarily, this step may refer to the above-mentioned step S101 and will not be described in detail.

[0062] S402. Determine a storage unit corresponding to the data segment; wherein the storage unit is a portion of the storage space of the disk.

[0063] Exemplarily, this step may refer to the above-mentioned step S102 and will not be described in detail.

[0064] S403. Determine multiple data units corresponding to the data shards; wherein the data unit includes at least one data block and at least one check block, the number of data blocks corresponding to different data shards is the same, and the number of check blocks corresponding to different data shards is the same.

[0065] Exemplarily, each data shard may correspond to a plurality of data units, and a data unit may be a data block or a check block, and each data shard may correspond to at least one data block and at least one check block. In this embodiment, the number of data blocks in each data shard is the same, and the number of check blocks in each data shard is also the same. For example, each data shard may correspond to 8 data blocks and 4 check blocks. A data block may be a smaller data unit divided from a data shard, and a check block may be redundant data attached to a data block for detecting and correcting data errors.

[0066] The data block and the check block may be collectively referred to as a data unit. In this embodiment, a division rule of the data unit may be preset, and the data unit corresponding to the data shard may be determined according to the preset division rule. For example, the preset division rule may be to divide the data shard into 8 data blocks, and generate a check block for every two data blocks. In this embodiment, the preset division rule is not specifically limited.

[0067] In this embodiment, determining multiple data units corresponding to the data shard includes: dividing the data shard into a first preset number of data blocks; generating a second preset number of check blocks based on the first preset number of data blocks; wherein the check blocks are arranged after all data blocks of the data shard.

[0068] Specifically, the data slice is segmented and divided into a plurality of data blocks. The number of data blocks can be preset as a first preset number. For example, if the first preset number is 8, the data slice is divided into 8 data blocks. The data slice can be evenly divided into the first preset number of data blocks, or divided into the first preset number of data blocks of different sizes. That is, the size of each data block in the data slice can be the same or different. When dividing the data blocks, the data blocks can be divided sequentially according to the order of the data in the data slice, that is, the data blocks can be spliced ​​sequentially to obtain a complete data slice.

[0069] The number of check blocks is preset as the second preset number. After obtaining the first preset number of data blocks, the second preset number of check blocks can be generated according to the first preset number of data blocks. A check block generation algorithm is preset, for example, the preset algorithm can be a hash algorithm. In this embodiment, the generation method of the check block is not specifically limited.

[0070] Arrange the data blocks and check blocks in sequence, with the check blocks behind the data blocks. For example, if there are 8 data blocks and 4 check blocks, then in all the data units corresponding to the data shards, the first 8 data units are data blocks and the last 4 data units are check blocks.

[0071] The beneficial effect of such a setting is that the data slices are subdivided into data units, which is conducive to local sequential writing of data in the data slices, achieving high throughput in the HDD scenario, and improving the efficiency of data storage.

[0072] S404: For each data slice, determine the arrangement order of the data units corresponding to the data slice.

[0073] Exemplarily, after the data shards are divided, the arrangement order of the data units corresponding to the data shards is determined. In this embodiment, the data units are arranged in sequence, with the data block in front and the check block in the back. The arrangement order between the data units is determined by the data content of the data shards themselves. In the same data stream to be stored, the number of data units in each data shard is the same, and the arrangement order is that the data block is arranged before the check block.

[0074] S405: Store the data segment in a storage unit according to the arrangement order of the data segments in the data segment and the arrangement order of the data units corresponding to each data segment.

[0075] Exemplarily, for each data segment, the arrangement order of the data segments in the data segment is determined, that is, the arrangement order of the blobs in the block is determined. The blobs in each block are arranged in sequence according to the data content of the block itself.

[0076] A data segment includes multiple data slices. For each data slice in the data segment, determine the arrangement order of the data units corresponding to the data slice. According to the arrangement order of the data slices in the data segment and the arrangement order of the data units corresponding to all the data slices in the data segment, the data segment is stored in the corresponding storage unit. For example, the data segment includes 3 data slices, each data slice corresponds to 12 data units, the arrangement order of the data slices is data slice 1, data slice 2, data slice 3, and the arrangement order of the data units corresponding to each data slice is data unit 1 to data unit 12. According to the order of data slices 1 to data slice 3, and the order of the three groups of data units 1 to data unit 12, the data segment is stored in the corresponding storage unit. For example, the data units in each data slice can be stored in the corresponding storage unit in sequence in units of data units.

[0077] In this embodiment, sequential writing is performed according to the order of blobs in a block and the order of data units in a blob, so as to realize the orderly storage of data and improve the efficiency and accuracy of data storage. Moreover, the data in a data segment is stored in a storage unit, so as to realize the physical aggregation characteristics of the data segment and improve the efficiency of subsequent data reading.

[0078] In this embodiment, the storage units are distributed in multiple disks; the data segments are stored in the storage units according to the arrangement order of the data shards in the data segment and the arrangement order of the data units corresponding to each data shard, including: determining the data units located at the same sorting position in each data shard according to the arrangement order of the data units corresponding to each data shard; storing the data units located at the same sorting position in each data shard in the storage units of the same disk according to the arrangement order of the data shards in the data segment.

[0079] Specifically, the same storage unit can be distributed in multiple disks, that is, multiple disks can be pre-equipped, each disk is divided, and a part of the storage space is taken from each disk as a complete storage unit. In other words, the disk space is divided in advance, and each disk is divided into N storage units. That is, N volumes are provided, and each volume is distributed on a different disk. For example, there are 12 disks, N is 3, and each disk is divided into three parts. The first part of the storage space in each disk is used as storage unit 1; the second part of the storage space in each disk is used as storage unit 2; the third part of the storage space in each disk is used as storage unit 3. In other words, one storage unit is distributed in 12 disks.

[0080] Determine the order of data units corresponding to each data slice in the data segment, and determine the data units at the same sorting position in the data slices according to the order of data units corresponding to each data slice. For example, determine the first data unit in each data slice, that is, each data slice must determine a first data unit.

[0081] For the data units corresponding to the data shards, each sorting position corresponds to a disk. The disks corresponding to different sorting positions can be the same or different. After determining the storage unit, the data units at the same sorting position in each data shard are stored in the storage unit of the corresponding disk. That is, the data units at the same sorting position in the data shards need to be stored in the same storage unit of the same disk.

[0082] According to the arrangement order of each data shard in the data segment, the data units at the same sorting position in each data shard are stored in sequence. For example, the first data unit of Blob1 in Block1 is stored in a storage unit of any disk; the first data unit of Blob2 in Block1 is also stored in the storage unit of the disk and is located after the first data unit of Blob1; the first data unit of Blob3 in Block1 is also stored in the storage unit of the disk and is located after the first data unit of Blob2. The second data unit of each Blob in Block1 can be stored in the storage unit of another disk in sequence.

[0083] The beneficial effect of this setting is that for each Block, the arrangement order of each Blob in the Block is determined. According to the arrangement order of the Blobs, the data units at the same position in each Blob are stored on the same volume in the same disk. Local sequential writing is implemented during data storage, which can not only disperse the data in the data segment on different disks to avoid excessive disk pressure, but also avoid random writing. Implementing random writing is a very complex operation. Not supporting random writing will greatly simplify the implementation of the data plane and improve the efficiency of data storage.

[0084] In this embodiment, data units located at the same sorting position in each data shard are determined according to the arrangement order of data units corresponding to each data shard, including: determining a target position according to the arrangement order of data units corresponding to each data shard; wherein the target position represents the sorting position of the first data unit that has not yet been stored in the data shard; and determining the data unit located at the target position in each data shard.

[0085] Specifically, the arrangement order of the data units corresponding to each data shard in the data segment is determined, and a sorting position is determined from the arrangement order as the target position. The target position represents the sorting position of the first data unit that has not been stored in the data shard, that is, the data unit at the target position needs to be stored at present. After the data unit at the target position is stored, a new target position can be determined until all the data units in the data shard are stored, that is, the data segment is stored.

[0086] After determining the target position, the data unit located at the target position in each data shard can be obtained. As many data units as there are data shards, the obtained data units are stored in the corresponding storage units according to the arrangement order between the data shards. Starting from the first data unit in the data shard, the target position can be determined, and then the data unit at the target position is stored according to the arrangement order of the data shards. That is, the first data unit in each data shard is stored first, and then the second data unit in each data shard is stored.

[0087] The beneficial effect of such a setting is that data is stored sequentially starting from the first data unit in each data shard, thereby avoiding data omission and improving the efficiency and accuracy of data storage.

[0088] In this embodiment, the disk is deployed in a computer node, and the computer node is a node in a preset distributed storage cluster; according to the arrangement order of the data shards in the data segment, the data units located at the same sorting position in each data shard are stored in the storage unit of the same disk, including: determining a computer node from the preset distributed storage cluster as the target node; storing the data units located at the same sorting position in each data shard in the storage unit of the disk of the target node in sequence according to the arrangement order of the data shards in the data segment.

[0089] Specifically, a distributed storage cluster is pre-set, and the distributed storage cluster includes multiple nodes, each of which can be a computer instance, that is, the distributed storage cluster includes multiple computer nodes. A disk is deployed in each computer node, each disk is divided into N volumes, and each volume is distributed on these disks.

[0090] Determine the data units at the same sorting position in each data shard in the data segment, that is, determine the data units at the target position in each data shard, and store these data units in the same volume of the same disk. A computer node can be determined from a preset distributed storage cluster as the target node, and the above data units are stored in the disk of the target node. For example, a target node can be determined randomly, or a target node can be manually specified, or the current target node can be determined in sequence according to the sorting of the computer nodes.

[0091] The beneficial effect of such a setting is that by setting up a distributed storage cluster, the data stream to be stored can be distributedly stored, which is conducive to flexible expansion of storage space, improving storage efficiency, and avoiding data loss. Data units with the same sorting position are stored on the same computer to achieve physical aggregation of data.

[0092] Figure 5 A schematic diagram of a truncation of the data stream to be stored. Figure 5After storing the data stream to be stored, if an error is found in the header data, the header of the data stream to be stored can be truncated, that is, the data in the dotted box is recovered and stored again. Currently, when storing files, only tail truncation is supported, which results in the head data of the file being unable to be recovered before the tail data. In the disclosed embodiment, data in different blocks are stored in different disks or different storage units. When an error occurs in the head block, there is no need to recover the tail block first, and the correct data can be directly replaced in the storage unit of the disk corresponding to the head block, thereby improving the efficiency of data storage.

[0093] In the disclosed embodiment, the data stream to be stored is divided into multiple data segments, and the data segments are further divided into multiple data slices, so as to reduce the difficulty of management and realize effective and reliable management of the data stream to be stored. The computer disk is divided into multiple storage units, the storage unit corresponding to the data segment is determined, and the data segment is sequentially written in the order of the data slices, and the content in the data segment is stored in the same storage unit, so as to realize the physical aggregation characteristics of the data segment and improve the efficiency and accuracy of data storage.

[0094] Figure 6 A flowchart of a data storage method based on streaming data is provided in an embodiment of the present disclosure.

[0095] In this embodiment, the method also includes: determining the stream identifier of the data stream to be read in response to a data stream read instruction; determining the target identifier from the segment identifier associated with the stream identifier of the data stream to be read; and obtaining the data segment of the target identifier in the data stream to be read from the storage unit of the storage identifier associated with the target identifier.

[0096] This embodiment is based on the above embodiment. Figure 6 As shown, the method comprises the following steps:

[0097] S601, obtaining a data stream to be stored, and segmenting the data stream to be stored to obtain multiple data segments; wherein the multiple data segments constitute the data stream to be stored, and the data segments are divided into multiple data fragments.

[0098] Exemplarily, this step may refer to the above-mentioned step S101 and will not be described in detail.

[0099] S602. Determine a storage unit corresponding to the data segment; wherein the storage unit is a portion of the storage space of the disk.

[0100] Exemplarily, this step may refer to the above-mentioned step S102 and will not be described in detail.

[0101] S603: Store the data segment in a storage unit according to the arrangement order of the data fragments in the data segment.

[0102] Exemplarily, this step may refer to the above-mentioned step S103 and will not be described in detail.

[0103] S604: In response to the data stream read instruction, determine the stream identifier of the data stream to be read.

[0104] Exemplarily, after storing a file, the user can read the stored file. The user can issue a data stream read instruction to indicate that he wants to read data. The data stream read instruction may include a stream identifier, and the stream identifier may be a unique ID of the data stream to be read. In this embodiment, the stream identifier may be a 128-bit ID. In response to the data stream read instruction, the stream identifier is obtained, and the file corresponding to the stream identifier is read, that is, the data stream to be read is obtained.

[0105] In this embodiment, a stream identifier is used instead of a file name. In current file storage systems, all objects, such as files, directories, links, etc., must have their own names. The file name exists as an intuitive index to facilitate management by applications. However, the file name is actually for people's own convenience to understand, and the management of the name brings corresponding costs, especially the complexity cost. In the early days in a single-machine environment, these costs were relatively small, but in today's distributed environment, the anomalies and uncertainties of the distributed environment are superimposed, and the costs are magnified to a very high level. Therefore, the disclosed embodiment adopts a 128-bit ID to avoid ID duplication, which not only greatly simplifies the complexity of file management, but also realizes efficient and accurate management of file data.

[0106] In this embodiment, it also includes: determining the flow identifier of the data stream to be stored, the segment identifier of the data segment in the data stream to be stored, and the storage identifier of the storage unit corresponding to the data segment; wherein the flow identifier represents the data stream, the segment identifier represents the data segment, and the storage identifier represents the storage unit; and the flow identifier, the segment identifier, and the storage identifier are associated and stored.

[0107] Specifically, when storing a data stream to be stored, each data stream to be stored corresponds to its own stream identifier, and a data segment in the data stream to be stored may also correspond to its own segment identifier, and each storage unit also corresponds to its own storage identifier. The stream identifier can uniquely represent the data stream, the segment identifier can uniquely represent the data segment, and the storage identifier can uniquely represent the storage unit.

[0108] After the data stream to be stored is stored, the storage unit where each data segment in the data stream to be stored is determined. The storage identifier of the storage unit is determined, and the stream identifier, the segment identifier, and the corresponding storage identifier are stored in association. That is, for each stream identifier, the storage identifiers corresponding to each segment identifier can be stored in association. For example, for the data stream to be stored, the stream identifier is 001, and there are two data segments in the data stream to be stored, and the segment identifiers are 00a and 00b respectively. The storage identifier of the storage unit corresponding to 00a is 0A, and the storage identifier of the storage unit corresponding to 00b is 0B. Then 001, 00a, and 0A can be stored in association, and 001, 00b, and 0B can be stored in association.

[0109] The beneficial effect of such a setting is that the data identifier and the storage location are stored in association, thereby achieving effective management of the data and facilitating subsequent accurate reading of the data.

[0110] S605: Determine a target identifier from the segment identifier associated with the stream identifier of the data stream to be read.

[0111] Exemplarily, a data stream to be read may include multiple data segments, so a stream identifier may be associated with multiple segment identifiers. The segment identifiers associated with the stream identifier of the data stream to be read are determined, and a segment identifier is determined from these segment identifiers as a target identifier, so as to read the data segment corresponding to the target identifier. When the data segment corresponding to the target identifier is read, a new target identifier can be determined, until all segment identifiers associated with the stream identifier of the data stream to be read are used as target identifiers for data reading.

[0112] In this embodiment, determining a target identifier from segment identifiers associated with the stream identifier of the data stream to be read includes: determining all segment identifiers associated with the stream identifier of the data stream to be read; and determining the target identifier from all segment identifiers according to the arrangement order of all segment identifiers associated with the stream identifier of the data stream to be read.

[0113] Specifically, all segment identifiers associated with the stream identifier of the data stream to be read are searched according to the pre-stored association relationship, and the target identifier is determined from these segment identifiers. For example, the target identifier can be randomly determined from these segment identifiers.

[0114] The arrangement order of all segment identifiers associated with the stream identifier of the data stream to be read may also be determined, and the arrangement order may represent the order of the data segments in the data stream to be read. According to the arrangement order, the target identifier may be determined from all segment identifiers in sequence, that is, the segment identifier of the first data segment in the data stream to be read is first used as the target identifier for data reading, and then the segment identifier of the second data segment in the data stream to be read is used as the target identifier for data reading, until all data segments are read.

[0115] The beneficial effect of such a setting is that data segments are read sequentially, which can avoid data reading omissions and improve the efficiency and accuracy of data reading.

[0116] S606: Obtain the data segment of the target identifier in the data stream to be read from the storage unit of the storage identifier associated with the target identifier.

[0117] Exemplarily, a storage identifier associated with the target identifier is determined, and a data segment of the target identifier in the data stream to be read is obtained from a storage unit corresponding to the storage identifier. When all segment identifiers are read, a complete data stream to be read is obtained.

[0118] In this embodiment, each data segment in the data stream to be read is read to achieve orderly reading of the data stream to be read, that is, sequential reading is achieved, which fully utilizes the performance characteristics of the HDD, improves data throughput, and further improves data reading efficiency.

[0119] In this embodiment, a data segment is divided into data slices with a preset number of slices; storage units are distributed in multiple disks, and the disks are deployed in computer nodes, which are nodes in a preset distributed storage cluster; from the storage unit of the storage identifier associated with the target identifier, a data segment of the target identifier in the data stream to be read is obtained, including: for the disk of each computer node in the preset distributed storage cluster, from the storage unit of the storage identifier associated with the target identifier, the first data unit containing the target identifier is determined as the starting unit; starting from the starting unit, a preset number of slices of data units are obtained; and the preset number of slices of data units obtained from the disk of each computer node are determined as the data segment of the target identifier in the data stream to be read.

[0120] Specifically, a preset number of shards is pre-set, and the preset shard data volume is the number of data shards in the data segment. When storing the data stream, sharding can be performed according to the preset number of shards to obtain the preset number of data shards. For the data stream to be read, the data segment includes the preset number of data shards.

[0121] A distributed storage cluster is pre-set, and the distributed storage cluster includes multiple computer nodes. A disk is deployed in each computer node, and each disk is divided into N storage units, and one storage unit is distributed in each disk. For each data segment in the data stream to be read, data units at different positions in the data slice of the data segment are stored in different computer nodes, and the stored data unit may contain a segment identifier of the corresponding data segment.

[0122] Determine the storage unit of the storage identifier associated with the target identifier. When reading data, for each computer node in the distributed storage cluster, that is, for each disk, determine whether a data unit containing the target identifier is stored in the storage unit. If not, it means that the disk does not have the data in the data segment corresponding to the target identifier; if so, since the data is written sequentially according to the order of the data shards during storage, the first data unit containing the target identifier in the storage unit can be determined as the starting unit, and starting from the starting unit, the data units of the preset number of shards are obtained. That is, all data related to the data segment of the target identifier in the storage unit of the disk are obtained.

[0123] Acquire all data related to the data segment of the target identifier in each disk, that is, for each disk containing the target identifier, a preset number of data units of shards can be acquired. The acquired data units are determined as the data segment of the target identifier in the data stream to be read. The data units in the storage unit can be marked with the sorting position of the data unit in the data shard, and the sorting between the data units in the same disk can represent the sorting position of the data shard to which the data unit belongs. According to the sorting position of the data unit in the data shard and the sorting position of the data shard to which the data unit belongs, the acquired data can be sorted to obtain accurate data segments. For example, there are two disks containing relevant data of the target identifier, the data units in the first disk are A and C, and the data units in the second disk are B and D, the sorting position of the data units marked in A and C is the first, and the sorting position of the data units marked in B and D is the second. That is, A is the first data unit in the first data fragment in the data segment, B is the second data unit in the first data fragment in the data segment, C is the first data unit in the second data fragment in the data segment, and D is the second data unit in the second data fragment in the data segment. The complete data segment is ABCD.

[0124] The beneficial effect of such a setting is that batch sequential reading can be performed according to the preset number of shards, thereby realizing pre-reading when reading data and improving the efficiency of data reading.

[0125] Figure 7 Schematic diagram for data reading. Figure 7There are three data streams to be read, namely Stream1, Stream2, and Stream3. The data segments in Stream1 include Block1-1, Block1-2, Block1-3, and Block1-4. The data segments in Stream2 include Block2-1, Block2-2, and Block2-3. The data segments in Stream3 include Block3-1. When reading the data streams, read them in the order of the data segments. For example, for Stream1, first read Block1-1 from the storage unit Volume2, then read Block1-2 from Volume1, then read Block1-3 from Volume2, and then read Block1-4 from Volume3; for Stream2, first read Block2-1 from Volume1, then read Block2-2 from Volume2, and then read Block3-3 from Volume3; for Stream3, read Block3-1 from Volume1.

[0126] In the disclosed embodiment, the data stream to be stored is divided into multiple data segments, and the data segments are further divided into multiple data slices, so as to reduce the difficulty of management and realize effective and reliable management of the data stream to be stored. The computer disk is divided into multiple storage units, the storage unit corresponding to the data segment is determined, and the data segment is sequentially written in the order of the data slices, and the content in the data segment is stored in the same storage unit, so as to realize the physical aggregation characteristics of the data segment and improve the efficiency and accuracy of data storage.

[0127] Figure 8 The following is a structural block diagram of a data storage device based on streaming data provided by an embodiment of the present disclosure. For ease of explanation, only the parts related to the embodiment of the present disclosure are shown. Figure 8 The data storage device 800 based on streaming data includes: a segmentation unit 801, a determination unit 802 and a storage unit 803.

[0128] The segmentation unit 801 is used to obtain a data stream to be stored, and segment the data stream to be stored to obtain a plurality of data segments; wherein the plurality of data segments constitute the data stream to be stored, and the data segments are divided into a plurality of data fragments;

[0129] A determination unit 802 is used to determine a storage unit corresponding to the data segment; wherein the storage unit is a part of the storage space of the disk;

[0130] The storage unit 803 is used to store the data segment in the storage unit according to the arrangement order of the data slices in the data segment.

[0131] Fig. 9 A structural block diagram of a data storage device based on streaming data provided by an embodiment of the present disclosure, such as Fig. 9 As shown, the data storage device 900 based on streaming data includes a segmentation unit 901 , a determination unit 902 and a storage unit 903 , wherein the storage unit 903 includes a unit determination module 9031 , a sequence determination module 9032 and a data storage module 9033 .

[0132] The unit determination module 9031 is used to determine a plurality of data units corresponding to the data shards; wherein the data unit includes at least one data block and at least one check block, and the number of data blocks corresponding to different data shards is the same, and the number of check blocks corresponding to different data shards is the same;

[0133] An order determination module 9032 is used to determine, for each of the data slices, an arrangement order of the data units corresponding to the data slice;

[0134] The data storage module 9033 is used to store the data segment in the storage unit according to the arrangement order of the data slices in the data segment and the arrangement order of the data units corresponding to each of the data slices.

[0135] In one example, the unit determination module 9031 includes:

[0136] A data block division submodule, used for dividing the data slice into a first preset number of data blocks;

[0137] The check block generation submodule is used to generate a second preset number of check blocks according to the first preset number of data blocks; wherein the check blocks are arranged after all the data blocks of the data slice.

[0138] In one example, the storage unit is distributed in multiple disks; the data storage module 9033 includes:

[0139] A position determination submodule, used to determine the data units located at the same sorting position in each of the data slices according to the arrangement order of the data units corresponding to each of the data slices;

[0140] The data storage submodule is used to store the data units at the same sorting position in each of the data slices into the storage unit of the same disk according to the arrangement order of the data slices in the data segment.

[0141] In one example, the location determination submodule is specifically used to:

[0142] Determine a target position according to the arrangement order of the data units corresponding to each of the data shards; wherein the target position represents the sorting position of the first data unit that has not yet been stored in the data shard;

[0143] The data unit located at the target position in each of the data slices is determined.

[0144] In one example, the disk is deployed in a computer node, and the computer node is a node in a preset distributed storage cluster; the data storage submodule is specifically used for:

[0145] Determine a computer node from the preset distributed storage cluster as the target node;

[0146] The data units at the same sorting position in each of the data slices are stored in sequence in the storage unit in the disk of the target node according to the arrangement order of the data slices in the data segment.

[0147] In one example, it also includes:

[0148] The sharding unit is used to perform sharding processing on the data segment to obtain a plurality of data shards; wherein the plurality of data shards constitute the data segment.

[0149] In one example, it also includes:

[0150] an identification determination unit, used to determine a flow identification of the data flow to be stored, a segment identification of a data segment in the data flow to be stored, and a storage identification of a storage unit corresponding to the data segment; wherein the flow identification represents the data flow, the segment identification represents the data segment, and the storage identification represents the storage unit;

[0151] The associated storage unit is used to associate and store the flow identifier, the segment identifier, and the storage identifier.

[0152] In one example, it also includes:

[0153] An instruction response unit, configured to determine a stream identifier of a data stream to be read in response to a data stream read instruction;

[0154] a target determination unit, configured to determine a target identifier from a segment identifier associated with a stream identifier of the data stream to be read;

[0155] The data reading unit is used to obtain the data segment of the target identifier in the data stream to be read from the storage unit of the storage identifier associated with the target identifier.

[0156] In one example, the target determination unit includes:

[0157] An identifier search module, used for determining all segment identifiers associated with the stream identifier of the data stream to be read;

[0158] The target determination module is used to determine the target identifier from all the segment identifiers according to the arrangement order of all the segment identifiers associated with the stream identifier of the data stream to be read.

[0159] In one example, a data segment is divided into a preset number of data shards; the storage unit is distributed in a plurality of disks, the disks are deployed in a computer node, and the computer node is a node in a preset distributed storage cluster; the data reading unit includes:

[0160] A starting determination module, configured to determine, for each computer node in the preset distributed storage cluster, from the storage units of the storage identifier associated with the target identifier, the first data unit containing the target identifier as the starting unit;

[0161] A unit acquisition module, used to acquire the data units of the preset number of fragments starting from the starting unit;

[0162] The data reading module is used to determine the data units of the preset number of fragments obtained from the disk of each computer node as the data segment of the target identifier in the data stream to be read.

[0163] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device.

[0164] Fig.10 A structural block diagram of an electronic device provided in an embodiment of the present disclosure, such as Fig.10 As shown, the electronic device 1000 includes: at least one processor 1002; and a memory 1001 communicatively connected to the at least one processor 1002; wherein the memory stores instructions executable by the at least one processor 1002, and the instructions are executed by the at least one processor 1002 so that the at least one processor 1002 can execute the data storage method based on streaming data disclosed in the present invention.

[0165] The electronic device 1000 further includes a receiver 1003 and a transmitter 1004. The receiver 1003 is used to receive instructions and data sent by other devices, and the transmitter 1004 is used to send instructions and data to external devices.

[0166] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0167] According to an embodiment of the present disclosure, the present disclosure also provides a computer program product, which includes: a computer program, the computer program is stored in a readable storage medium, at least one processor of an electronic device can read the computer program from the readable storage medium, and at least one processor executes the computer program so that the electronic device executes the solution provided by any of the above embodiments.

[0168] Fig.11 A schematic block diagram of an example electronic device 1100 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0169] like Fig.11 As shown, the device 1100 includes a computing unit 1101, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1102 or a computer program loaded from a storage unit 1108 into a random access memory (RAM) 1103. In the RAM 1103, various programs and data required for the operation of the device 1100 can also be stored. The computing unit 1101, the ROM 1102, and the RAM 1103 are connected to each other via a bus 1104. An input / output (I / O) interface 1105 is also connected to the bus 1104.

[0170] A number of components in the device 1100 are connected to the I / O interface 1105, including: an input unit 1106, such as a keyboard, a mouse, etc.; an output unit 1107, such as various types of displays, speakers, etc.; a storage unit 1108, such as a disk, an optical disk, etc.; and a communication unit 1109, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 1109 allows the device 1100 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0171] The computing unit 1101 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 1101 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 1101 performs the various methods and processes described above, such as a data storage method based on streaming data. For example, in some embodiments, a data storage method based on streaming data may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 1108. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 1100 via the ROM 1102 and / or the communication unit 1109. When the computer program is loaded into the RAM 1103 and executed by the computing unit 1101, one or more steps of the data storage method based on streaming data described above may be performed. Alternatively, in other embodiments, the computing unit 1101 may be configured to execute the data storage method based on streaming data in any other appropriate manner (for example, by means of firmware).

[0172] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0173] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0174] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0175] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0176] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0177] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services ("Virtual Private Server", or "VPS" for short). The server may also be a server of a distributed system, or a server combined with a blockchain.

[0178] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0179] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A data storage method based on streaming data, comprising: Acquire a data stream to be stored, and segment the data stream to be stored to obtain a plurality of data segments; wherein the plurality of data segments constitute the data stream to be stored, and the data segments are divided into a plurality of data fragments; Determine a storage unit corresponding to the data segment; wherein the storage unit is a portion of the storage space of the disk; The data segment is stored in the storage unit according to the arrangement order of the data slices in the data segment.

2. The method according to claim 1, wherein: The storing the data segment into the storage unit according to the arrangement order of the data slices in the data segment comprises: Determine a plurality of data units corresponding to the data shards; wherein the data unit includes at least one data block and at least one check block, and the number of data blocks corresponding to different data shards is the same, and the number of check blocks corresponding to different data shards is the same; For each of the data slices, determining an arrangement order of data units corresponding to the data slice; The data segment is stored in the storage unit according to the arrangement order of the data slices in the data segment and the arrangement order of the data units corresponding to each of the data slices.

3. The method according to claim 2, wherein: The determining of the plurality of data units corresponding to the data slices includes: Dividing the data slice into a first preset number of data blocks; A second preset number of check blocks are generated according to the first preset number of data blocks; wherein the check blocks are arranged after all the data blocks of the data slice.

4. The method according to claim 2 or 3, wherein: The storage units are distributed in a plurality of disks; storing the data segments in the storage units according to the arrangement order of the data segments in the data segments and the arrangement order of the data units corresponding to the data segments comprises: Determining data units at the same sorting position in each of the data slices according to the arrangement order of the data units corresponding to each of the data slices; According to the arrangement order of the data slices in the data segment, the data units located at the same sorting position in each of the data slices are stored in the storage unit of the same disk.

5. The method according to claim 4, wherein: The step of determining the data units located at the same sorting position in each of the data slices according to the arrangement order of the data units corresponding to each of the data slices comprises: Determine a target position according to the arrangement order of the data units corresponding to each of the data shards; wherein the target position represents the sorting position of the first data unit that has not yet been stored in the data shard; The data unit located at the target position in each of the data slices is determined.

6. The method according to claim 4 or 5, wherein: The disk is deployed in a computer node, and the computer node is a node in a preset distributed storage cluster; according to the arrangement order of the data slices in the data segment, the data units located at the same sorting position in each of the data slices are stored in the storage unit of the same disk, including: Determine a computer node from the preset distributed storage cluster as the target node; The data units at the same sorting position in each of the data slices are stored in sequence in the storage unit in the disk of the target node according to the arrangement order of the data slices in the data segment.

7. The method according to any one of claims 1 to 6, further comprising: The data segment is fragmented to obtain a plurality of data fragments; wherein the plurality of data fragments constitute the data segment.

8. The method according to any one of claims 1 to 7, further comprising: Determine a flow identifier of the data flow to be stored, a segment identifier of a data segment in the data flow to be stored, and a storage identifier of a storage unit corresponding to the data segment; wherein the flow identifier represents the data flow, the segment identifier represents the data segment, and the storage identifier represents the storage unit; The flow identifier, segment identifier, and storage identifier are associated and stored.

9. The method according to claim 8, further comprising: In response to the data stream read instruction, determining a stream identifier of the data stream to be read; Determining a target identifier from a segment identifier associated with a stream identifier of the data stream to be read; The data segment of the target identifier in the to-be-read data stream is obtained from a storage unit storing an identifier associated with the target identifier.

10. The method according to claim 9, wherein: The step of determining the target identifier from the segment identifier associated with the stream identifier of the data stream to be read comprises: Determine all segment identifiers associated with the stream identifier of the data stream to be read; The target identifier is determined from all the segment identifiers according to the arrangement order of all the segment identifiers associated with the stream identifier of the data stream to be read.

11. The method according to claim 9 or 10, wherein: The data segment is divided into a preset number of data shards; the storage unit is distributed in a plurality of disks, the disks are deployed in computer nodes, and the computer nodes are nodes in a preset distributed storage cluster; The step of acquiring the data segment of the target identifier in the to-be-read data stream from a storage unit storing the identifier associated with the target identifier comprises: For the disk of each computer node in the preset distributed storage cluster, determine the first data unit containing the target identifier from the storage units of the storage identifier associated with the target identifier as the starting unit; Starting from the starting unit, obtaining the data units of the preset number of fragments; The preset number of fragmented data units obtained from the disk of each computer node are determined as the data segment of the target identifier in the data stream to be read.

12. A data storage device based on streaming data, comprising: A segmentation unit, used for acquiring a data stream to be stored, and segmenting the data stream to be stored to obtain a plurality of data segments; wherein the plurality of data segments constitute the data stream to be stored, and the data segments are divided into a plurality of data fragments; A determination unit, used to determine a storage unit corresponding to the data segment; wherein the storage unit is a part of the storage space of the disk; A storage unit is used to store the data segment in the storage unit according to the arrangement order of the data slices in the data segment.

13. The device according to claim 12, wherein: The storage unit comprises: A unit determination module, used to determine a plurality of data units corresponding to the data shards; wherein the data unit includes at least one data block and at least one check block, and the number of data blocks corresponding to different data shards is the same, and the number of check blocks corresponding to different data shards is the same; An order determination module, used to determine, for each of the data slices, an arrangement order of data units corresponding to the data slice; The data storage module is used to store the data segment in the storage unit according to the arrangement order of the data slices in the data segment and the arrangement order of the data units corresponding to each of the data slices.

14. The device according to claim 13, wherein: The unit determination module comprises: A data block division submodule, used for dividing the data slice into a first preset number of data blocks; The check block generation submodule is used to generate a second preset number of check blocks according to the first preset number of data blocks; wherein the check blocks are arranged after all the data blocks of the data slice.

15. The device according to claim 13 or 14, wherein: The storage unit is distributed in multiple disks; The data storage module comprises: A position determination submodule, used to determine the data units located at the same sorting position in each of the data slices according to the arrangement order of the data units corresponding to each of the data slices; The data storage submodule is used to store the data units at the same sorting position in each of the data slices into the storage unit of the same disk according to the arrangement order of the data slices in the data segment.

16. The device according to claim 15, wherein: The position determination submodule is specifically used for: Determine a target position according to the arrangement order of the data units corresponding to each of the data shards; wherein the target position represents the sorting position of the first data unit that has not yet been stored in the data shard; The data unit located at the target position in each of the data slices is determined.

17. The device according to claim 15 or 16, wherein: The disk is deployed in a computer node, which is a node in a preset distributed storage cluster; the data storage submodule is specifically used for: Determine a computer node from the preset distributed storage cluster as the target node; The data units at the same sorting position in each of the data slices are stored in sequence in the storage unit in the disk of the target node according to the arrangement order of the data slices in the data segment.

18. The apparatus according to any one of claims 12 to 17, further comprising: The sharding unit is used to perform sharding processing on the data segment to obtain a plurality of data shards; wherein the plurality of data shards constitute the data segment.

19. The apparatus according to any one of claims 12 to 18, further comprising: an identification determination unit, used to determine a flow identification of the data flow to be stored, a segment identification of a data segment in the data flow to be stored, and a storage identification of a storage unit corresponding to the data segment; wherein the flow identification represents the data flow, the segment identification represents the data segment, and the storage identification represents the storage unit; The associated storage unit is used to associate and store the flow identifier, the segment identifier, and the storage identifier.

20. The apparatus according to claim 19, further comprising: An instruction response unit, configured to determine a stream identifier of a data stream to be read in response to a data stream read instruction; a target determination unit, configured to determine a target identifier from a segment identifier associated with a stream identifier of the data stream to be read; The data reading unit is used to obtain the data segment of the target identifier in the data stream to be read from the storage unit of the storage identifier associated with the target identifier.

21. The device according to claim 20, wherein: The target determination unit comprises: An identifier search module, used for determining all segment identifiers associated with the stream identifier of the data stream to be read; The target determination module is used to determine the target identifier from all the segment identifiers according to the arrangement order of all the segment identifiers associated with the stream identifier of the data stream to be read.

22. The device according to claim 20 or 21, wherein: The data segment is divided into a preset number of data shards; the storage unit is distributed in a plurality of disks, the disks are deployed in computer nodes, and the computer nodes are nodes in a preset distributed storage cluster; The data reading unit comprises: A starting determination module, configured to determine, for each computer node in the preset distributed storage cluster, from the storage units of the storage identifier associated with the target identifier, the first data unit containing the target identifier as the starting unit; A unit acquisition module, used to acquire the data units of the preset number of fragments starting from the starting unit; The data reading module is used to determine the data units of the preset number of fragments obtained from the disk of each computer node as the data segment of the target identifier in the data stream to be read.

23. An electronic device, comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 11.

24. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-11.

25. A computer program product, wherein: The invention comprises a computer program, which implements the steps of the method according to any one of claims 1 to 11 when being executed by a processor.

Images