Sequence data processing method, storage device and storage system

Abstract

The invention discloses a sequence data processing method, storage equipment and a storage system. The processing method is applied to the storage device, the storage device or the storage system carries out data processing operation in the form of a sequence data stream, the method comprises the steps that the storage device carries out data processing operation in the form of the sequence data stream, and the method comprises the steps that the storage device receives sequence data sent by a host end; the storage device stores the sequence data in the form of a sequence data stream, and sets a corresponding sequence data index; and the storage device receives a data operation instruction sent by the host end for the sequence data, and performs corresponding processing on the corresponding sequence data based on the sequence data index. Through the mode, a software stack of the host end can be greatly simplified at the host end, and the occupation of storage access of the host end on a processor and a memory of the host end is reduced.

Description

Technical Field

[0001] The present application relates to the technical field of sequence data processing, and in particular to a sequence data processing method, storage device and storage system. Background Art

[0002] Storage devices are widely used to store various data. However, the relevant storage devices are usually storage devices based on block storage interfaces, such as various storage devices based on NAND media. Because storage devices are based on block storage, data needs to be divided into block logic when it is stored and stored in corresponding storage units.

[0003] Therefore, when storing sequence data, the sequence data needs to be encoded into a continuous sequence file in a proprietary sequence data format. The sequence file is split, mapped and stored in the block address of the block storage according to the logic of the block in the file system; when reading, the data in the corresponding block is obtained, and then the data is reassembled to obtain the sequence file, and the sequence file is decoded to obtain the sequence data. This process of reading and writing sequence data is cumbersome, and the sequence data saved in the storage will also lose the corresponding sequence information. The storage of sequence data cannot be efficiently organized and managed, which reduces storage efficiency and performance. Summary of the invention

[0004] The present application provides a method for processing sequence data, a storage device and a storage system, which can greatly simplify the host software stack, reduce the occupancy of the host processor by storage access on the host side, improve storage efficiency and performance in practical applications, and increase the service life of the memory.

[0005] In a first aspect, the present application provides a method for processing sequence data, which is applied to a storage device, and the storage device performs data processing operations in the form of a sequence data stream. The method includes: the storage device receives sequence data sent by a host; the storage device stores the sequence data in the form of a sequence data stream and sets a corresponding sequence data index; the storage device receives a data operation instruction for the sequence data sent by the host, and performs corresponding processing on the corresponding sequence data based on the sequence data index.

[0006] The storage device includes at least one sequence container, each sequence container stores one or more sequence data, and each sequence container has a unique container ID; the sequence container sets the attributes of the sequence container when it is created and / or after it is created; wherein the attributes of the sequence container store first description information of the stored sequence data, and the first description information is used to enable the host end to determine the way to access and use the relevant sequence data through the first description information.

[0007] In response to deleting the sequence container in the storage device, all sequence data stored in the sequence container is deleted; when deleting the sequence container, in response to the storage device being set to a data protection mode, deletion of the sequence container is not allowed.

[0008] There is at least one sequence data stream in each sequence container, and each sequence data stream stores the same type of sequence data and has a unique sequence data stream ID; the storage device responds to a sequence data stream deletion instruction and deletes all sequence data on the sequence data stream corresponding to the sequence data stream deletion instruction, wherein the sequence data stream can be set to be unable to be deleted; the sequence data stream sets the attributes of the sequence data stream after creation; the attributes of the sequence data stream include second descriptive information of the data stored in the sequence data stream.

[0009] Among them, the sequence container has a unique sequence space, the sequence space has a corresponding sequence ID, the sequence ID is used to describe the position of the sequence data in the sequence data stream in the sequence container in the sequence space; the new and old relationship of the sequence data in the sequence data stream is determined by the size of the sequence ID; the sequence data stream ID and the sequence ID of the sequence data stream constitute the sequence data index, and the sequence data index corresponds to the unique sequence data in the sequence container, and other sequence data can only replace the original sequence data or be added to the original sequence data.

[0010] Among them, the sequence container supports a cyclic overwrite function, and the cyclic overwrite function is set with a reserved sequence ID length; when the cyclic overwrite function is turned on, in response to new sequence data being written into the sequence container, the sequence data outside the interval formed by the new sequence ID and the reserved sequence ID length is deleted.

[0011] Among them, a sequence data index set is used to identify multiple sequence data indexes, which are used in sequence data instructions to operate on multiple sequence data at the same time; the sequence data index set is a combination of multiple sequence data indexes, or the sequence data index set is a combination of multiple sequence data stream IDs and sequence IDs; or, the sequence data index set is a combination of a sequence ID interval and at least one sequence data stream ID, which is used to identify all sequence data of at least one sequence data stream within the sequence ID interval; wherein the sequence ID interval is used to represent the set of all sequence IDs located in the interval.

[0012] In which, in response to the sequence data deletion instruction, the sequence data corresponding to the sequence data index set in the target sequence container corresponding to the data deletion instruction is deleted.

[0013] In which, in response to a sequence data compression instruction, the sequence data corresponding to the data index set in the sequence data compression instruction is compressed, and the compressed sequence data is used to replace the original sequence data; in response to the sequence data stream being set to a compression mode, the sequence data written into the sequence data stream is compressed using a given compression algorithm; and before the compressed sequence data is read or calculated for use, it is decompressed using a given decompression algorithm to obtain the sequence data before compression.

[0014] The storage device responds to the sequence data merging instruction and performs a merging operation on the sequence data on the sequence data stream corresponding to the sequence data merging instruction.

[0015] Among them, in response to the data protection instruction, data protection is enabled for the sequence data corresponding to the data index set in the data protection instruction, and the sequence data corresponding to the data index set and the corresponding sequence data index cannot be deleted or changed; in response to the sequence container and / or the sequence data stream being set as a data protection option, all sequence data on the corresponding sequence data stream in the data protection option and the corresponding sequence data index cannot be deleted or changed.

[0016] In which, in response to the data protection function of a sequence container or a sequence data stream being enabled, deletion and / or change of sequence data already written in the protected data stream is not allowed; and / or, in response to the data protection function of a sequence container or a sequence data stream being enabled, only sequence data whose sequence ID is newer than the latest sequence ID in the sequence data stream is allowed to be written into the protected sequence data stream; and / or, in response to the data protection function of a sequence container or a sequence data stream being enabled, only sequence data whose sequence ID is within a sequence ID protection interval is allowed to be written into all sequence data streams in the sequence container; wherein the sequence ID protection interval is determined by the length of the interval between the latest sequence ID in the sequence container and the sequence ID protection zone in the data protection function of the sequence container.

[0017] Wherein, when the data protection function of the sequence data stream is turned on, in response to turning on the sequence data stream protection function, a sequence signature of the corresponding sequence data is added to the signature field of the sequence data tag to protect the data integrity of the corresponding data stream.

[0018] Among them, each sequence container with the data protection function enabled has a sequence signature private key, the sequence signature private key cannot be obtained externally, and the sequence signature public key corresponding to the sequence signature private key is stored in the attributes of the sequence container, which is used to verify the data integrity of the protected sequence data stream; wherein, the attributes of the sequence container include the sequence signature public key, related information, and the signature of the sequence signature public key and related information by the sequence memory security private key; in response to the first verification instruction, the signature of the related information is verified using the sequence memory security public key obtained by the sequence memory to prove the integrity of the sequence signature public key and related information; in response to the second verification instruction, the sequence data signature is used to verify the integrity of the current sequence ID, current sequence data, current sequence data label, and the sequence ID and signature of the previous sequence data; wherein, the sequence data signature is obtained by using the sequence signature private key to sign the sequence ID and signature of the previous sequence data of the sequence data stream, the current sequence ID, the current sequence data, and the current sequence data label.

[0019] In response to enabling the sequence data stream protection function, the sequence ID and signature information of the latest sequence data on all protected sequence data streams in the sequence container are signed to obtain a container signature, and the container signature is stored in the attributes of the sequence container. The container signature is used to ensure the integrity of all protected sequence data streams in the sequence storage.

[0020] In which, in response to a sequence data encryption instruction, sequence data corresponding to a sequence data index set in the sequence data encryption instruction is encrypted and stored; in response to a sequence container turning on a sequence data encryption function or a sequence data stream turning on a sequence data encryption function, the corresponding sequence data is encrypted and stored using a specified sequence encryption key; wherein the sequence encryption key is obtained by key generation or exchange.

[0021] The configuration range of the configuration information of the sequence data stream is within the configuration range of the configuration information of the sequence container.

[0022] The configuration information of the sequence data stream includes at least one of the storage requirements, size, format, label, query method and operation method of the data.

[0023] Among them, at least one sequence data tag is configured on the sequence data stream, and the configuration content includes the type, length, default value, and at least one of the tag operations that can be performed on the tag; in response to all sequence data written or modified on the sequence data stream, and the sequence data stream is configured with a sequence data tag, the data of the configured sequence data tag is stored as a supplementary part of the sequence data index; in response to an operation instruction on the sequence data stream, a rule operation is performed on at least one tag item in the sequence data tag, and when the result of the rule operation meets the requirements, the operation corresponding to the operation instruction is performed on the sequence data stream; in response to the sequence data stream being configured with a sequence data tag, and the operation instruction requires the use of a sequence data index set, the sequence data index set is filtered and adjusted using a tag filtering operation to obtain a filtered sequence data index set.

[0024] The sequence container includes at least one sequence data stream; in response to the configuration lock of the sequence data stream, no new sequence data stream can be added to the sequence container, and the configuration information of at least one data stream cannot be changed.

[0025] The sequence ID is determined by the time when the sequence data is generated on the host side.

[0026] Among them, the sequence data includes audio and video data; the description information of the audio and video data is saved in the attributes of the sequence container; the audio or video stream data from the same source and / or the same encoding in the audio and video data is stored using the same sequence data stream, and the corresponding audio or video stream information is saved in the corresponding sequence data stream attributes; audio or video stream data from different sources or different encodings are stored using different sequence data streams.

[0027] In which, in response to a counting operation on at least one sequence data stream, the number of data on a given sequence data index expression in at least one sequence data stream is counted; or, in response to a list operation on at least one sequence data stream, the number, data size and / or label of data within a given index interval in at least one sequence data stream is determined.

[0028] The sequence data includes audio and video data. The storage device responds to the transcoding instruction and uses the codec circuit in the storage device to decode and encode the audio and video data of the first coding standard corresponding to the transcoding instruction to obtain the audio and video data of the second coding standard.

[0029] Among them, the sequence data is blockchain data; the storage device stores the sequence data in the form of a sequence data stream and sets a corresponding sequence data index, including: using the height value of the blockchain data as the sequence ID, storing the block data of the blockchain data in the form of a data stream, and setting a corresponding sequence data index.

[0030] Among them, the blockchain data includes main chain data and forked chain data; the height value of the blockchain data is used as the sequence ID, the block data of the blockchain data is stored in the form of a data stream, and the corresponding sequence data index is set, including: using the height value of the main chain data as the sequence ID, using the first data stream to store the main chain data, using the height value of the forked chain data as the sequence ID, and using the second data stream to store the forked chain data.

[0031] Among them, the blockchain data includes main chain data and side chain data; the height value of the blockchain data is used as the sequence ID, the block data of the blockchain data is stored in the form of a data stream, and the corresponding sequence data index is set, including: using the height value of the main chain data as the sequence ID, using the first data stream to store the main chain data, using the height value of the side chain data as the sequence ID, and using the second data stream to store the side chain data.

[0032] In a second aspect, the present application provides a storage device, which includes a processor, a storage unit, a memory, and an interface unit. The storage unit is used to store computer programs and sequence data, and the processor is used to execute the computer program to implement the method provided in the first aspect on the storage device. The interface unit is connected to the processor of the host side to complete data communication between the host side and the storage device.

[0033] In a third aspect, the present application provides a storage system, which includes: a host end; a storage device, the storage device is communicatively connected to the host end, and the storage device is such as the storage device provided in the second aspect.

[0034] The beneficial effects of the present application are as follows: Different from the prior art, the sequence data processing method provided by the present application stores the sequence data in the form of a data stream, and sets a corresponding sequence data index, and searches for the corresponding sequence data through the sequence data index, which can greatly simplify the host software stack on the host side, reduce the occupancy of the host processor by the host storage access, improve the storage efficiency and performance in practical applications, and improve the service life of the memory; on the storage device side, various storage media can be organized more effectively, unnecessary garbage collection can be reduced, the wear of the storage medium can be reduced, the service life of the storage device can be improved, the use of over-allocated space and dynamic random access memory can be reduced, and the cost of the storage device can be reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0036] in:

[0037] Figure 1 It is a flowchart of the first embodiment of the method for processing sequence data provided by the present application;

[0038] Figure 2 It is a flow chart of the second embodiment of the method for processing sequence data provided by the present application;

[0039] Figure 3 It is a schematic diagram of the relationship between the sequence container sequence ID and the sequence data flow provided by this application;

[0040] Figure 4 It is a schematic diagram of an application scenario of a method for processing sequence data provided in this application;

[0041] Figure 5 It is a schematic diagram of an application scenario of a method for processing sequence data provided in this application;

[0042] Figure 6 It is a schematic diagram of an application scenario of a method for processing sequence data provided in this application;

[0043] Figure 7 It is a flowchart of the third embodiment of the method for processing sequence data provided by the present application;

[0044] Figure 8 It is a flowchart of a fourth embodiment of the method for processing sequence data provided by the present application;

[0045] Fig. 9 is a structural schematic diagram of an embodiment of a storage device provided by the present application;

[0046] Fig.10 It is a structural diagram of an embodiment of a storage system provided by the present application. DETAILED DESCRIPTION

[0047] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. It will be understood that the specific embodiments described herein are only used to explain the present application, rather than to limit the present application. It should also be noted that, for ease of description, only some but not all structures related to the present application are shown in the drawings. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.

[0048] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0049] See also Figure 1 , Figure 1 1 is a flow chart of a first embodiment of a method for processing sequence data provided by the present application. Applied to a storage device, the storage device performs data processing operations in the form of a sequence data stream. The data processing operations may be read, write, delete, query and / or calculation operations. The method comprises:

[0050] Step 11: The storage device receives the sequence data sent by the host.

[0051] In some embodiments, the storage device can be applied to one or more scenarios of audio and video data scenarios, surveillance video scenarios, autonomous driving scenarios, accident emergency data storage scenarios, Internet of Things scenarios, and blockchain data scenarios.

[0052] Therefore, the host side can be an electronic device in the corresponding scene. For example, in the monitoring video scene, the host side manages the processing equipment of multiple monitoring cameras for collecting and processing image sequence data. Among them, each sequence data stream stores all image data of a certain camera, and each image sequence data has time information as a sequence ID. For example, in the autonomous driving scene, the host side can be an autonomous driving system, which collects, processes, and perceives data from various sensors, makes autonomous driving decisions, and completes autonomous driving control. Among them, the sequence memory can store data including various autonomous driving sensors such as cameras, ultrasonic waves, millimeter wave radars, and lidars, as well as the perception results of the autonomous driving system on these automatic sensor data, the decision results of the autonomous driving system, other sensor data, and the status of various actuators such as throttles and steering wheels, etc. All data related to autonomous driving. Among them, each of the above data is stored in an independent sequence data stream in the form of sequence data, and each sequence data uses the time of collection or generation as the sequence ID, which can completely restore the entire process of autonomous driving. Among them, the sequence container supports cyclic coverage, which can efficiently and continuously save all autonomous driving data. Among them, the sequence container can enable digital signature protection, which can prove the validity and integrity of the saved data, and even the autonomous driving system itself cannot forge or tamper with the stored autonomous driving data afterwards. When a collision occurs, the emergency data protection function can be enabled to ensure that the data before and after the collision will not be deleted or changed in the sequence memory, which serves as effective evidence for accident analysis. The sequence memory can be compatible with autonomous driving systems with different sensors and architectures through a sequence data stream. When the corresponding sensor data encoding method adopts a standard encoding method, the same accident analysis software can be used to read, test and analyze accident data from autonomous driving systems with different sensors and different architectures.

[0053] Step 12: The storage device stores the sequence data in the form of a data stream and sets a corresponding sequence data index.

[0054] Among them, a data stream is a set of ordered data sequences of bytes with a starting point and an end point. Therefore, the sequence data can be stored in the form of a data stream, and a corresponding sequence data index can be set. In some embodiments, the sequence data index is composed of a sequence data stream ID and a sequence ID, and the corresponding sequence data can be uniquely indexed in the storage device. That is, each sequence data has a corresponding sequence data index.

[0055] After the sequence data is stored in the form of a data stream, the stored sequence data may be managed based on the sequence data index corresponding to the data stream.

[0056] Different sequence data streams are created for data of different sources, formats and meanings to store corresponding sequence data. For example, separate sequence data streams are created for data from different hosts to store the sequence data of the corresponding hosts; for example, sequence data streams are created for sequence data from different sensors to store the sequence data of the corresponding sensors; for example, separate sequence data streams are created for data of different meanings, such as sensor data, sensor data perception results, automated decision results, and decision result execution status. Sequence data indexes can be established for sequence data in sequence data streams. When processing sequence data, corresponding sequence data can be read according to the sequence data index and further logical operations can be performed.

[0057] By storing the sequence data in the form of a sequence data stream, the sequence data will not be split and can be stored and retrieved in its entirety. That is, the corresponding processing logic between the host side and the storage device also becomes simple. The host side only needs to form the sequence data to store the sequence data in the storage device through the communication interface.

[0058] Step 13: The storage device receives a data operation instruction for the sequence data sent by the host, and performs corresponding processing on the corresponding sequence data based on the sequence data index.

[0059] The data operation instruction may be a read, write, delete, query and / or operation instruction for the sequence data. The storage device receives the read, write, delete, query and / or operation instruction for the sequence data sent by the host, and can quickly query the corresponding sequence data through the sequence data index and perform corresponding processing on it.

[0060] In this embodiment, compared with the memory of the block storage interface used in the related art, such as SSD (Solid State Disk), UFS (Universal Flash Storage), SD card (Secure Digital Memory Card) and USB flash drive, the sequence data is stored in the form of a sequence data stream, and the corresponding sequence data index is set, which can greatly simplify the host software stack on the host side, reduce the occupancy of the host processor by the host storage access, improve the storage efficiency and performance in practical applications, and improve the service life of the memory; on the storage device side, various storage media can be organized more effectively, unnecessary garbage collection can be reduced, the wear of the storage medium can be reduced, the service life of the storage device can be improved, the use of over-allocated space and dynamic random access memory can be reduced, and the cost of the storage device can be reduced.

[0061] See also Figure 2 , Figure 21 is a flow chart of the second embodiment of the method for processing sequence data provided by the present application. Applied to a storage device, the storage device performs data processing operations in the form of a sequence data stream. The storage device includes at least one sequence container, each sequence container is configured to store one or more sequence data. A sequence container is a unit for storing sequence data in a storage device, all sequence data are stored in a corresponding sequence container, and each sequence container has a unique sequence container ID; wherein the storage device can add a new sequence container to store different sequence data. The method includes:

[0062] Step 21: Receive the sequence data sent by the host.

[0063] The storage device may have multiple communication interfaces, and each communication interface may communicate with a host end. Specifically, the sequence data of each host end may be divided into categories. For example, the categories may be divided according to the host end, or the categories may be divided according to the data of the host end. That is, a corresponding category may be set for each sequence container in the storage device or storage system to store the corresponding category of sequence data.

[0064] Among them, each sequence container can have different configurations. According to the sequence container configuration, the sequence container can have different size limits. Different sequence container description information stores sequence data with different characteristics, supports different types of sequence data streams, provides different reading, writing, querying, and computing capabilities, and realizes different data storage security and performance.

[0065] Furthermore, on the host side, the corresponding storage service program can be matched according to the configuration of the sequence container, and the corresponding data services can be directly provided to the upper-level applications on the host side. By standardizing the sequence container configuration, a software ecosystem and applications corresponding to the storage device can be built on the host side.

[0066] Step 22: Determine a target sequence container corresponding to the type of sequence data.

[0067] In an application scenario, a storage device includes multiple sequence containers, such as sequence container A, sequence container B, and sequence container C. Sequence container A corresponds to type a, sequence container B corresponds to type b, and sequence container C corresponds to type c. If the type of sequence data currently received is type b, sequence container B may be determined as a target sequence container.

[0068] Step 23: Store the sequence data in the target sequence container in the form of a sequence data stream, and set the corresponding sequence data index.

[0069] Store the sequence data in the target sequence container in the form of a sequence data stream and set the corresponding sequence data index.

[0070] The sequence container has a unique sequence space, and the sequence space has a corresponding sequence ID. The sequence ID is used to describe the position of the data in the sequence data stream of the sequence container in the sequence space. Step 23 can store the sequence data in the corresponding data space according to the sequence ID and sequence data stream ID of the sequence data, and set the corresponding sequence data index. That is, the sequence data stream ID and the sequence ID of the sequence data stream constitute the sequence data index.

[0071] The old and new relationship of the sequence data in the sequence data stream is determined by the size of the sequence ID. The sequence ID can be compared, interval and / or added and subtracted. In the same sequence data stream, the sequence data is unique, that is, there can be at most one copy of a sequence data ID in a certain sequence data stream.

[0072] The sequence ID is represented by a number not limited to 32 bits, 64 bits, 128 bits or 256 bits in length. Specifically, the length of the sequence data index can be set according to actual needs. The longer the length of the sequence data index is, the more sequence data can be stored.

[0073] In an application scenario, in response to deleting a target sequence container in a storage device, all sequence data stored in the target sequence container will be deleted. Deleting a sequence container may be performed in full or in part, such as deleting a sequence container.

[0074] In some embodiments, the sequence container of the storage device may be set to a data protection mode, and the data protection mode is used to protect the sequence container from being deleted and to protect the sequence data stored in the sequence container. That is, when deleting a target sequence container, in response to the storage device being set to the data protection mode, the target sequence container is not allowed to be deleted.

[0075] Furthermore, the sequence container sets the properties of the sequence container when it is created and / or after it is created; wherein the properties of the sequence container store the first description information of the stored sequence data, and the first description information is used to enable the host end to determine the way to access and use the relevant sequence data through the first description information. That is, the host end determines the way to access and use the relevant sequence data by obtaining the first description information of the data in the properties of the sequence container.

[0076] In some embodiments, each sequence container may contain at least one sequence data stream, each sequence data stream stores the same type of sequence data and has a unique sequence data stream ID, wherein each sequence container may add a new sequence data stream.

[0077] The storage device responds to the sequence data stream deletion instruction and deletes all sequence data on the sequence data stream corresponding to the sequence data stream deletion instruction, wherein the target sequence data stream can be set to be unable to be deleted to protect the data on the sequence data stream from being deleted. The sequence data stream sets the attributes of the sequence data stream after creation; the attributes of the sequence data stream include the second description information of the data stored on the sequence data stream, and the second description information can help the host end access and use the data on the sequence data stream.

[0078] In some embodiments, the above sequence container has a unique sequence space, and the sequence space has a corresponding sequence ID, and the sequence ID is used to describe the position of the data in the sequence data stream in the sequence container in the sequence space. Figure 3 As shown, the sequence data in each sequence data stream in the sequence space corresponds to a corresponding sequence ID. The sequence data in sequence data stream 1 correspond to sequence ID1, sequence ID2, sequence ID3, sequence ID4, sequence IDN-2, sequence IDN-1, and sequence IDN. The sequence data in sequence data stream 2 correspond to sequence ID1, sequence ID3, sequence ID4, sequence IDN-1, and sequence IDN. The sequence data in sequence data stream 3 correspond to sequence ID1, sequence ID2, sequence ID3, sequence ID4, sequence IDN-2, sequence IDN-1, and sequence IDN. The sequence data in sequence data stream N correspond to sequence ID1, sequence ID2, sequence ID3, sequence ID4, sequence IDN-2, sequence IDN-1, and sequence IDN.

[0079] The sequence ID is represented by a number of any length, 32bit, 64bit, 128bit or 256bit. In the same sequence data stream, the same sequence ID has only unique sequence data. The sequence ID has an increasing or decreasing relationship, which is determined when the sequence container is created. The new and old relationship of the sequence data in the sequence data stream is determined by the size of the sequence ID, that is, the new and old relationship of the sequence data is determined by comparing the size of the sequence ID. The sequence ID can perform interval addition and subtraction operations, and can be used to describe the space within a certain sequence distance before and after a certain sequence space position, and can correspond to the description and judgment of sequence data in the sequence space.

[0080] The sequence data stream ID and the sequence ID constitute a sequence data index. In the sequence container, the sequence data index corresponds to unique sequence data. There is at most one copy of sequence data on the same sequence data index, and other sequence data can only replace the original sequence data or be added to the original sequence data.

[0081] The sequence data is stored in the sequence container of the storage device and is organized and stored according to the sequence data index.

[0082] Further, the sequence container supports a cyclic overwrite function, and the cyclic overwrite function is provided with a reserved sequence ID length. Specifically, when the cyclic overwrite function is turned on, in response to writing new sequence data in the sequence container, the sequence data outside the interval formed by the new sequence ID and the reserved sequence ID length is deleted. That is, the sequence ID length reserved for storage is specified in the cyclic overwrite function. When new sequence data is written in the sequence container, the old sequence data outside the interval formed by the latest sequence ID and the reserved sequence ID length will be automatically deleted, and the storage medium space occupied by the corresponding deleted sequence data will be recovered. The reserved sequence ID length is set to 5, and currently there are sequence ID1, sequence ID2, sequence ID3, sequence ID4 and sequence ID5. New sequence data is written, and the sequence data corresponds to sequence ID6. When the cyclic overwrite function is turned on, the sequence data corresponding to sequence ID1 is deleted, and the sequence data corresponding to sequence ID2, sequence ID3, sequence ID4, sequence ID5 and sequence ID6 are retained.

[0083] The circular overwrite function reasonably arranges the storage location of sequence data with the same or similar sequence IDs on different sequence data streams in the sequence container. When the sequence data on the corresponding sequence ID is overwritten, the complete storage medium space is directly reclaimed, thereby balancing the wear of the medium while avoiding garbage collection operations, reducing the write amplification caused by garbage collection, reducing the wear on the storage medium, and extending the service life of the storage medium.

[0084] In some embodiments, a sequence data set is used to identify multiple sequence data indexes, which are used in a sequence data instruction to operate on multiple sequence data at the same time.

[0085] A sequence data index set is a combination of multiple sequence data indexes, or a sequence data index set is a combination of multiple sequence data stream IDs and sequence IDs, and is used to identify sequence data corresponding to sequence IDs on multiple sequence data streams.

[0086] In some embodiments, the sequence ID is represented by an interval as a set of all sequence IDs within the interval, and the sequence data index set is a combination of the sequence data stream ID and the sequence ID interval, and is used to identify all sequence data on the corresponding sequence data stream whose sequence ID is within the sequence ID interval.

[0087] In some embodiments, the sequence data index set is a combination of multiple sequence data stream IDs and sequence ID intervals, and is used to identify all sequence data of the multiple sequence data stream IDs within the sequence ID interval.

[0088] In some embodiments, the sequence container may be provided with a corresponding sequence data deletion function. In response to a sequence data deletion instruction, the sequence data corresponding to the sequence data index set in the target sequence container corresponding to the data deletion instruction is deleted to release the storage space of the target sequence container.

[0089] In an application scenario, the storage space of the sequence container can be set according to actual needs, and a corresponding sequence data deletion function can be set. For example, in response to the available storage space of the target sequence container being less than a threshold, a sequence data deletion instruction is generated, data is deleted based on the sequence data index, and the storage space of the target sequence container is released. For example, if the threshold is ten percent, then when the available storage space is less than 10% of the storage space of the sequence container, data is deleted based on the sequence data index to release the storage space of the target sequence container. Specifically, the storage device will delete the corresponding sequence data in a manner from old to new according to the sequence data index, and reclaim the storage space to store the newly written data. In some embodiments, the deleted sequence data should be data stored in the same batch. Based on this, a cyclic write function can be set for the corresponding sequence container, and when the available storage space of the target sequence container is less than the threshold, data is deleted based on the sequence data index to release the storage space of the target sequence container.

[0090] In an application scenario, a sequence container of a storage device allows data on a sequence data stream to be compressed. The sequence data compression instruction or the sequence data stream compression setting compresses the sequence data corresponding to the sequence data index set using a given compression algorithm. For example, in response to the sequence data compression instruction, the sequence data corresponding to the data index set in the sequence data compression instruction is compressed, and the compressed sequence data is used to replace the original sequence data.

[0091] In response to the sequence data stream being set to a compression mode, the sequence data written into the sequence data stream is compressed using a given compression algorithm and written into the storage medium after being compressed.

[0092] Before reading or calculating the compressed sequence data, the compressed sequence data is decompressed using a given decompression algorithm to obtain the sequence data before compression.

[0093] In some embodiments, the storage device performs a merge operation on the sequence data on the sequence data stream corresponding to the sequence data merge instruction in response to the sequence data merge instruction.

[0094] In an application scenario, in response to a sequence data merge instruction, a merge operation is performed on the data on the target sequence data stream of the target sequence container. The merge operation includes operations such as merging and / or reducing the data. Among them, within the timing index interval of the specified sequence data stream, according to the given timing index function, the timing data that meets the timing index function condition is deleted. Then, forward / backward merging is performed according to the sequence data index, thereby reducing the effective data size in the sequence container and releasing storage space. Among them, different timing data types support merging and the types of supported merging operations are different. The data compression function of the sequence data can be configured in advance according to the data type.

[0095] Combination Figure 4 To explain:

[0096] like Figure 4 In the example, the sequence ID corresponding to data 1 is 0, the sequence ID corresponding to data 2 is 1, the sequence ID corresponding to data 3 is 2, the sequence ID corresponding to data 4 is 3, the sequence ID corresponding to data 5 is 4, the sequence ID corresponding to data 6 is 5, and the sequence ID corresponding to data 7 is 6. In response to the data merge instruction, data 4, data 5, and data 6 need to be merged. Then, according to the data merge instruction, data 4 and data 6 are deleted according to the corresponding sequence IDs, and data 4, data 5, and data 6 are merged into data 5' according to the selected merge algorithm, thereby releasing the storage space corresponding to the two sequence IDs to store new sequence data.

[0097] In one application scenario, a sequence container can be configured with a delete-forbidden function. After the function is enabled, the data written in the sequence container cannot be deleted. For example, in response to a data protection instruction, a data protection operation is performed on the data on the target sequence data stream of the target sequence container, wherein the data protected cannot be deleted and / or modified.

[0098] However, when the circular write function is enabled, the space can be reclaimed to write new data when the available space of the sequence storage sequence container is about to be filled.

[0099] In response to the data protection instruction, data protection is enabled for the sequence data corresponding to the data index set in the data protection instruction. The sequence data corresponding to the data index set and the corresponding sequence data index cannot be deleted or changed, thereby ensuring the integrity and consistency of the data within the sequence data index interval.

[0100] In response to the sequence container and / or the sequence data stream being set as a data protection option, all sequence data on the sequence data stream corresponding to the data protection option and the corresponding sequence data index cannot be deleted or modified.

[0101] In some embodiments, in response to the data protection function of a sequence container or a sequence data stream being enabled, deletion and / or modification of sequence data already written to the protected sequence data stream is not allowed.

[0102] In some embodiments, in response to the data protection function of the sequence container or the sequence data stream being enabled, only sequence data with a sequence ID newer than the latest sequence ID in the current sequence data stream is allowed to be written into the protected sequence data stream.

[0103] In some embodiments, in response to the data protection function of a sequence container or a sequence data stream being enabled, only sequence data whose sequence ID is within the sequence ID protection interval is allowed to be written to all sequence data streams in the sequence container.

[0104] The sequence ID protection interval is determined by the length of the interval between the latest sequence ID in the sequence container and the sequence ID protection zone in the sequence container data protection function.

[0105] In some embodiments, in response to the data protection function of the sequence container or sequence data stream being enabled, the protected sequence data stream does not allow the deletion and / or modification of the already written sequence data. And in response to the data protection function of the sequence container or sequence data stream being enabled, only the sequence data whose sequence ID is newer than the latest sequence ID in the current sequence data stream is allowed to be written into the protected sequence data stream.

[0106] In some embodiments, in response to the data protection function of a sequence container or a sequence data stream being enabled, the protected sequence data stream does not allow deletion and / or modification of the written sequence data. And in response to the data protection function of a sequence container or a sequence data stream being enabled, only sequence data whose sequence ID is within the sequence ID protection interval is allowed to be written to all sequence data streams in the sequence container.

[0107] The sequence ID protection interval is determined by the length of the interval between the latest sequence ID in the sequence container and the sequence ID protection zone in the sequence container data protection function.

[0108] In some embodiments, in response to the data protection function of a sequence container or a sequence data stream being enabled, the protected sequence data stream does not allow deletion and / or modification of already written sequence data; and in response to the data protection function of a sequence container or a sequence data stream being enabled, only sequence data that is newer than the latest sequence ID in the current sequence data stream is allowed to be written into the protected sequence data stream; and in response to the data protection function of a sequence container or a sequence data stream being enabled, only sequence data whose sequence ID is within the sequence ID protection interval is allowed to be written into all sequence data streams in the sequence container.

[0109] The sequence ID protection interval is determined by the latest sequence ID in the sequence container and the sequence ID protection interval in the sequence container data protection function.

[0110] In some embodiments, when the data protection function of the sequence container or the sequence data stream is turned on, in response to turning on the sequence data stream protection function, a sequence signature of the corresponding sequence data is added to the signature field of the sequence data tag to protect the data integrity of the corresponding sequence data stream. Any third party can verify the integrity of the protected sequence data stream through the signature public key stored in the sequence container attribute.

[0111] In some embodiments, each sequence container with data protection function enabled has a sequence signature private key, which cannot be obtained externally. The sequence signature public key corresponding to the sequence signature private key is stored in the attributes of the sequence container, and is used to verify the data integrity of the protected sequence data stream. Among them, the sequence signature public key can be obtained in response to the execution of the instruction to obtain the sequence container attributes. Among them, the attributes of the sequence container include the sequence signature public key, related information, and the signature of the sequence signature public key and related information by the sequence memory security private key. The signature of the related information can be verified using the sequence memory security public key obtained from the sequence memory to prove the integrity of the sequence signature public key and related information.

[0112] In some embodiments, in response to the first verification instruction, the sequence memory security public key obtained by the sequence memory is used to verify the signature of the related information to prove the integrity of the sequence signature public key and the related information.

[0113] In response to the second verification instruction, the sequence data signature is used to verify the integrity of the current sequence ID, current sequence data, current sequence data label, and the sequence ID and signature of the previous sequence data; wherein the sequence data signature is obtained by signing the sequence ID and signature of the previous sequence data in the sequence data stream, the current sequence ID, the current sequence data, and the current sequence data label using the sequence signature private key.

[0114] Combination Figure 5 To explain:

[0115] The sequence data signature private key is used to sign the sequence ID and signature of the previous sequence data in the sequence data stream, the current sequence ID, the current sequence data, and the current sequence data tag to obtain the corresponding sequence data signature. The sequence data signature is saved in the signature segment of the sequence data tag. The sequence data signature is used to prove the integrity of the current sequence ID, the current sequence data, the current sequence data tag, and the sequence ID and signature of the previous sequence data. Figure 5As shown, sequence data 0 corresponds to sequence ID 0, sequence data tag tag0, and sequence data signature 0. Sequence data n corresponds to sequence ID n, sequence data tag tagn, and sequence data signature n. Sequence data n+1 corresponds to sequence ID n+1, sequence data tag tagn+1, and sequence data signature n+1.

[0116] Among them, the sequence data signature can be obtained using the following formula:

[0117] Signature n+1=Sign(Key_pri, sequence IDn, signature n, sequence IDn+1, sequence data n+1, tagn+1).

[0118] Signature 0 = Sign (Key_pri, 0, {0}, sequence ID 0, sequence data 0, tag 0). When the first sequence data on the sequence data stream is signed, the previous sequence ID is filled in with 0, and the previous sequence signature is all 0 data.

[0119] Among them, Sign represents the signature function, and Key_pri represents the sequence signature private key.

[0120] When verifying integrity, the integrity of the sequence data between any sequence data and any sequence data can be verified by using the sequence data signature corresponding to any sequence data in the sequence data stream to start the back-tracking verification in sequence, which can prove the integrity of the corresponding sequence data on the sequence data stream, including proving the integrity of the corresponding data index on the sequence data stream, proving the integrity of the sequence data and sequence data label, and proving that no new sequence data is inserted in the middle of the sequence data stream or that any sequence data is deleted or changed. Figure 5 Starting with sequence data n in the example, the integrity of sequence data from sequence data n-1 to sequence data 0 can be verified backtrackingly. Figure 5 Starting from sequence data n+1 in the sequence data, the integrity of sequence data from sequence data n to sequence data 0 can be verified backtrackingly. Since the signature of the latter sequence data is related to the previous sequence data, if there is sequence data modification, it can be verified in the above manner.

[0121] In some embodiments, when the sequence data stream protection function is turned on, the sequence ID and signature information of the latest data on all protected sequence data streams in the sequence container are signed to obtain a container signature, and the container signature is stored in the attributes of the sequence container. The container signature is used to ensure the integrity of all protected sequence data streams in the sequence storage. The sequence data stream protection function can be set according to the actual sequence data streams. For example, if the number of sequence data streams is 3, the sequence data stream protection function can be set for 2 of the sequence data streams. The sequence data stream protection function can also be set for 3 sequence data streams. The specific number of sequence data streams to be protected is set according to actual needs. Combined with Figure 6 To explain:

[0122] like Figure 6 As shown, the sequence container includes sequence data stream 1, sequence data stream 2, and sequence data stream 3. You can select sequence data streams to set the sequence data stream protection function according to actual needs, such as selecting sequence data stream 1, sequence data stream 2, and sequence data stream 3. Select sequence data stream 2 and sequence data stream 3. Select sequence data stream 1 and sequence data stream 3.

[0123] For example, select sequence data stream 1, sequence data stream 2 and sequence data stream 3, and when the sequence data stream protection function is turned on, sign the sequence ID and signature information of the latest data of sequence data stream 1, sequence data stream 2 and sequence data stream 3 in the sequence container to obtain the container signature. That is, sign the sequence ID and signature information of sequence data n+1 in sequence data stream 1, sequence data stream 2 and sequence data stream 3 in the sequence container. The container signature is stored in the attributes of the sequence container, and the container signature is used to ensure the integrity of all protected sequence data streams in the sequence storage.

[0124] The verification process is as follows:

[0125] First, use the container signature to verify whether the sequence ID of the latest data on all protected sequence data streams is consistent with the sequence ID in the container signature. When they are consistent, use the target sequence data signature corresponding to the latest sequence data on all protected sequence data streams to start the backtracking verification in sequence, verify the integrity of the latest sequence data to the oldest sequence data on each protected sequence data stream, and prove the integrity of the sequence data stream, including the integrity of all data indexes on the sequence data stream, the integrity of the sequence data and sequence data labels, and the proof that no new sequence data is inserted in the middle of the sequence data stream or that any sequence data is deleted or changed. Through this verification method, the integrity of the sequence data of all protected sequence data streams in each container can be verified.

[0126] Combination Figure 6 To explain:

[0127] For example, sequence data stream 1, sequence data stream 2, and sequence data stream 3 are protected, and the container signature is used to verify whether the sequence ID of the latest data of sequence data stream 1, sequence data stream 2, and sequence data stream 3 is consistent with the sequence ID in the container signature. When consistent, the target sequence data signature corresponding to the latest sequence data on all protected sequence data streams is used to start the backtracking verification in sequence, and the integrity of the latest sequence data to the oldest sequence data on each protected sequence data stream is verified, which can prove the integrity of the sequence data stream, including proving the integrity of all data indexes on the sequence data stream, proving the integrity of the sequence data and sequence data tags, and proving that no new sequence data is inserted in the middle of the sequence data stream or that any sequence data is deleted or changed. That is, when consistent, the sequence data signature corresponding to the latest sequence data (sequence data n+1) on sequence data stream 1, sequence data stream 2, and sequence data stream 3 is used to start the backtracking verification in sequence, and the integrity of the latest sequence data to the oldest sequence data on each protected sequence data stream is verified.

[0128] In some embodiments, the above-mentioned current sequence data tag can be used for filtering through tag operation. For example, if the corresponding sequence data includes reference frames and ordinary frames, 1 bit can be added to the tag of the reference frame to indicate whether it is a reference frame, and then {[stream1, stream2], 1000..20000, tag: bit1&1} can be used to indicate obtaining reference frames between 1000 and 20000 on stream1 and stream2.

[0129] The above current sequence data tag can be queried, involved in calculations, and accumulated. For example, you can use sum(stream1,100..20000,tag:byte1) to accumulate the first byte of the tag on the sequence data between 100 and 20000 on stream1 and return the accumulated value.

[0130] In some embodiments, in response to a sequence data encryption instruction or a sequence container and sequence data stream setting, a sequence data encryption function is enabled, and the corresponding sequence data is encrypted and stored to protect the sequence data stored on the storage medium from being obtained by a third party without the encryption key.

[0131] In response to the sequence data encryption instruction, the sequence data corresponding to the sequence data index set in the sequence data encryption instruction is encrypted and stored.

[0132] In response to a sequence container enabling a sequence data encryption function or a sequence data stream enabling the sequence data encryption function, the corresponding sequence data is encrypted and stored using a designated sequence encryption key; wherein the sequence encryption key is obtained by key generation or exchange.

[0133] Among them, the sequence data encryption instruction is used to encrypt and store the sequence data corresponding to the sequence data index set in the sequence data encryption instruction; the sequence data written on the sequence container or the sequence data stream is automatically encrypted and stored using the specified sequence encryption key; the sequence encryption key is obtained through key generation or exchange, and the sequence storage device or sequence storage system supports various secure key generation or exchange protocols.

[0134] In an application scenario, in response to an enhanced instruction, the sequence data stream corresponding to the enhanced instruction is encrypted. For example, the sequence container can enable the sequence data consistency enhancement function. After enabling this function, it is signed by the specified storage device private key, the sequence container private key, or the private key exchanged by other private key exchange protocols. The signature data and the sequence data are read together. The host side can use the signature public key to perform cryptographic verification on the data of the entire sequence data stream, and the entire data sequence is not tampered with, ensuring the integrity and consistency of the data.

[0135] In an application scenario, in response to an operation instruction on multiple sequence data streams, the multiple sequence data streams are obtained from a target sequence container, and operations corresponding to the operation instructions are performed on the multiple sequence data streams.

[0136] Multiple sequence data streams can be operated simultaneously in the sequence container of the storage device. Multiple sequence data stream operations include all sequence data stream operations, enumeration of sequence data streams, and matching of sequence data streams by excluding enumerated sequence data streams, so as to achieve writing, deleting, reading, querying and / or computing operations on data of multiple sequence data streams in a single operation in a single sequence container.

[0137] In some embodiments, the configuration information of the sequence container may include at least one of storage requirements, size, format, label, query method, and operation method.

[0138] The sequence data stream also has corresponding configuration information, wherein the configuration range of the sequence data stream configuration information is within the configuration range of the sequence container configuration information. For example, the size of the sequence data stream is less than or equal to the size of the sequence container. The query method of the sequence data stream is included in the query method of the sequence container. The operation method of the sequence data stream is included in the operation method of the sequence container. The format of the sequence data stream is included in the format of the sequence container.

[0139] The configuration information of the sequence data stream includes at least one of the storage requirements, size, format, label, query method and operation method of the data.

[0140] At least one sequence data tag is configured on the sequence data stream, and the configuration content includes the type, length, default value, and at least one of the tag operations that can be performed.

[0141] In response to all sequence data written or modified on the sequence data stream, and the sequence data stream is configured with a sequence data tag, the data of the configured sequence data tag is stored as a supplementary part of the sequence data index.

[0142] In response to an operation instruction on the sequence data stream, a rule operation is performed on at least one tag item in the sequence data tag, and when a result of the rule operation meets the requirements, an operation corresponding to the operation instruction is performed on the sequence data stream.

[0143] The label of the sequence data is provided when writing or modifying the sequence data.

[0144] Sequence data tags are modified and overwritten through special sequence tag modification instructions.

[0145] In response to the sequence data stream being configured with a sequence data tag, and the operation instruction requiring the use of a sequence data index set, the sequence data index set is filtered and adjusted using a tag filtering operation to obtain a filtered sequence data index set. Specifically, in response to the sequence data stream being configured with a sequence data tag, a tag filtering operation is added to the sequence data index set to filter the sequence data index set to obtain a result data index set that satisfies the tag filtering operation. The tag filtering operation is used for any instruction that uses a data index set to filter and adjust the data index set operated by the instruction.

[0146] The label operation instruction is used to perform corresponding operations on the labels of the corresponding sequence data in the sequence data index set and return the results of the operations.

[0147] The tag includes at least one tag item, and in response to an operation instruction on the sequence data stream, a rule operation is performed on at least one tag item, and when the result of the rule operation meets the requirements, the operation corresponding to the operation instruction is performed on the sequence data stream. For example, when operations such as writing, deleting, reading, querying or operating the sequence data stream are performed, by performing equal, greater than, less than, matching operations on the tag items, and combined operations such as AND, OR, XOR, etc. between the tag items, the sequence data stream will be executed with the corresponding operation only when the operation result is established.

[0148] See also Figure 7 , Figure 71 is a flow chart of another embodiment of the method for processing sequence data provided by the present application. Applied to a storage device, the storage device performs read and write operations with a sequence data stream. The storage device includes at least one sequence container, each sequence container is configured to store sequence data of a corresponding type. The method includes:

[0149] Step 71: Receive sequence data sent by the host.

[0150] Step 72: Determine a target sequence container corresponding to the type of sequence data.

[0151] Step 73: Obtain the sequence data index of the sequence data.

[0152] The sequence data index is obtained by combining the sequence ID and the sequence data stream ID.

[0153] Step 74: In response to the target sequence data index in the sequence data index being smaller than the preset index, the data corresponding to the target sequence data index is prohibited from being written.

[0154] The preset index is determined by the current maximum sequence data index and the sequence data index protection interval.

[0155] The preset index may be determined by subtracting the maximum value or the minimum value in the sequence data index protection interval from the maximum sequence data index currently written.

[0156] In some embodiments, the sequence data indexes are compared with the interval of sequence IDs. For example, if the sequence data index is composed of {sequence data stream ID, sequence ID}, then the sequence index space can be expressed as {*, sequence ID1, ..., sequence ID2}, which can identify the data with sequence IDs between sequence ID1 and sequence ID2 on all sequence data streams in the sequence container. Similarly, matching can also use the same expression method, such as {[sequence data stream 1, sequence data stream 2], sequence ID1, ..., sequence ID2} to identify all data with sequence IDs greater than sequence ID1, ..., sequence ID2 on sequence data stream 1 and sequence data stream 2.

[0157] The sequence data index protection interval can be used to protect the sequence data of the corresponding sequence interval to prevent the sequence data from being changed, that is, it can ensure that the sequence data that has been written cannot be tampered with in the sequence.

[0158] In an application scenario, a target sequence container includes at least one sequence data stream; in response to the configuration lock of the sequence data stream, a new sequence data stream cannot be added to the sequence container, and the configuration information of at least one sequence data stream cannot be changed. For example, the target sequence container includes sequence data stream D, sequence data stream E, and sequence data stream F. In response to the configuration lock of the sequence data stream in the target sequence container, a new sequence data stream cannot be added to the target sequence container, and the configuration information of multiple sequence data streams cannot be changed, that is, the target sequence container can only include sequence data stream D, sequence data stream E, and sequence data stream F.

[0159] In some embodiments, the above-mentioned sequence data index can be a time index. The time series data index can be a counter starting with 0 and not limited to time units such as picoseconds, microseconds, milliseconds, seconds or minutes, a counter that increments by write, various timestamps including Unix, ISO8601 or RFC2822, etc. At this time, the sequence container can store data generated by various IoT devices including videos, as well as new data edited, processed, and generated based on these data, to achieve efficient storage, query and calculation of data of related processing devices such as the Internet of Things. Among them, the sequence data index is based on a combination of sequence ID and sequence data stream ID. For example, {sequence data stream ID, sequence ID}, the sequence ID here is determined by the time when the sequence data is formed on the host side. That is, the sequence ID can be time.

[0160] The sequence index interval is {sequence data stream ID combination, sequence ID interval}. The sequence ID interval can include range(sequence ID1..duration), before(sequence ID1), between(sequence ID1..sequence ID2), after(sequence ID1), latest(duration) represents the latest sequence ID interval of the current container, oldest(duration) represents the earliest sequence ID range data of the current container, all(), enumeration combination: [sequence ID1, sequence ID2, ... sequence IDx].

[0161] The sequence data stream ID combination can be: all(), sequence data stream ID1..sequence data stream ID2. Since the sequence data stream is increasing in the sequence container, it can be represented by an interval; before(sequence data stream ID1), after(sequence data stream ID1), enumeration combination: [sequence data stream ID1, sequence data stream ID2,…, sequence data stream Idx].

[0162] In some embodiments, the sequence data includes audio and video data, wherein the sequence ID of the sequence container is a time offset sequence of the sequence data, and the sequence data on all sequence data streams in the sequence container use the same type of sequence ID.

[0163] The description information of the audio and video data is saved in the attributes of the sequence container. The host side decides how to access and use the audio and video data based on the description information in the attributes of the sequence container.

[0164] The audio or video stream data from the same source and / or the same encoding in the audio and video data are stored using the same sequence data stream, and the corresponding audio or video stream information is saved in the corresponding sequence data stream attributes. The host can decode the audio and video data on the corresponding sequence data stream according to the data description information on the corresponding sequence data stream.

[0165] Audio or video stream data from different sources or different encodings are stored using different sequence data streams.

[0166] Other sensor data, subtitles, interactive elements, processing of audio and video on other sequence data streams, artificial intelligence perception results and / or artificial intelligence decision results are each stored using a separate sequence data stream.

[0167] In some embodiments, in response to a counting operation on at least one sequence data stream, the number of data on a given sequence data index expression in the at least one sequence data stream is counted.

[0168] Among them, multiple data in the sequence data stream can be read, written, queried or operated through the sequence data index expression. The sequence data index expression can obtain all data before the target sequence data index, all data after the target sequence data index, all data in the target interval, and the latest data.

[0169] For example, the sequence data index expression may be before(index), where index represents the index subscript, and before(index) represents all data before the given sequence data index index.

[0170] For example, the sequence data index expression may be after(index), where after(index) represents all data after the given sequence data index is index.

[0171] For example, the sequence data index expression can be range(index, duration), where range(index, duration) means giving all data from the sequence data indexed as index to index+duration.

[0172] The sequence data index expression can be latest, which means obtaining the latest data.

[0173] In some embodiments, in response to a list operation on at least one sequence data stream, the number, size and / or label of data within a given index interval in the at least one sequence data stream is determined.

[0174] In some embodiments, the sequence data includes audio and video data, and the storage device responds to the transcoding instruction, and uses the codec circuit in the storage device to decode and encode the audio and video data of the first coding standard corresponding to the transcoding instruction to obtain the audio and video data of the second coding standard. In an application scenario, the sequence data stream data is an H.265 video frame, so the sequence data on the corresponding sequence data stream can be decoded using the standard H.265codec. Among them, the sequence data of the sequence data stream is a 4K H.265 video frame. In response to the transcoding instruction, the 4K H.265 video frame data stream is decoded by the codec circuit, and encoded into a 1080p H.265 video frame, and stored accordingly. For example, the 4K H.265 video frame is directly replaced by the H.265 video frame encoded into 1080p. For another example, a new sequence data stream is generated, and the 1080p H.265 video frame is stored according to the sequence ID of the 4K H.265 video frame. Or it is stored in another 1080p H265 video frame. In this way, the host side does not need to read data for decoding and encoding related operations, thereby improving the system working efficiency of the host side.

[0175] In some embodiments, the sequence data includes audio and video data, and the storage device responds to the processing instruction, uses the codec circuit in the storage device to decode the audio and video data corresponding to the processing instruction, and uses the calculation circuit in the storage device to calculate the decoded audio and video data according to the algorithm / model specified by the processing instruction to obtain the processing result. In an application scenario, the sequence data is an H.265 video frame, so the sequence data on the corresponding sequence data stream can be decoded using the standard H.265codec. The data of the sequence data is an H.265 video frame, responds to the processing instruction, uses the codec circuit to decode the H.265 video frame, and uses the calculation circuit to calculate the decoded video data using the formulated algorithm / model to obtain the processing result and store it. For example, the processing result is directly updated as the label data to the label of the corresponding video frame data. For another example, a new sequence data stream is generated, and the processing result is stored according to the sequence ID of the H.265 video frame. In this way, there is no need for the host to read the data, perform encoding and decoding and calculation, which improves the system work efficiency of the host.

[0176] See also Figure 8 , Figure 8 1 is a flow chart of another embodiment of the method for processing sequence data provided by the present application. The method is applied to a storage device, and the storage device performs read and write operations with a sequence data stream. The method includes:

[0177] Step 81: Receive blockchain data sent by the host.

[0178] Step 82: Use the height value of the blockchain data as the sequence ID, store the block data of the blockchain data in the form of a sequence data stream, and set the corresponding sequence data index.

[0179] In some embodiments, the blockchain data includes main chain data and forked chain data, and step 82 can use the height value of the main chain data as the sequence ID, use the first sequence data stream to store the main chain data, and set the corresponding sequence data index. Use the height value of the forked chain data as the sequence ID, use the second sequence data stream to store the forked chain data, and set the corresponding sequence data index.

[0180] In some embodiments, the blockchain data includes main chain data and side chain data, then step 82 can use the height value of the main chain data as the sequence ID, use the first sequence data stream to store the main chain data, and set the corresponding sequence data index. Use the height value of the side chain data as the sequence ID, use the second sequence data stream to store the side chain data, and set the corresponding sequence data index.

[0181] In some embodiments, the blockchain data includes main chain data, forked chain data, and side chain data. In step 82, the height value of the main chain data can be used as the sequence ID, the main chain data can be stored using the first sequence data stream, and the corresponding sequence data index can be set. The height value of the forked chain data can be used as the sequence ID, the forked chain data can be stored using the second sequence data stream, and the corresponding sequence data index can be set.

[0182] In some embodiments, the corresponding application data storage and service-related sequence container configurations, sequence container description information, and supported sequence data stream configuration and description information can be standardized into a sequence storage sequence container format according to different requirements of the host-side application. On the basis of standardization, multiple parties can develop a standard-compatible sequence storage sequence container format software library and a sequence data application running on the software library.

[0183] Specifically, corresponding standards are formulated by acquiring information of a sequence storage sequence container, including configuration of the sequence container, description information of the sequence container, configuration of all sequence data streams, description of the sequence data streams, and other information.

[0184] Based on the acquired sequence storage sequence container information, the standard-compatible sequence storage format software library and the sequence data application running on the software library can process all sequence containers that comply with the sequence storage sequence container format, even if there are great differences in the sequence container configuration, sequence container description information, data volume configuration, description and quantity of these sequence containers.

[0185] Here, the compatibility of the program needs to be described to the sequence data stream. For example, if the sequence data stream data is an H.265 video frame, then the data on the corresponding sequence data stream can be decoded using the standard H.265 codec. Corresponding to the decoding of various data of autonomous driving data, the standard application can be adapted to different autonomous driving sensors and the reading and processing of autonomous driving recorded data of different architectures.

[0186] See also Fig. 9 , Fig. 9 1 is a schematic diagram of a structure of an embodiment of a storage device provided by the present application. The storage device 90 includes a processor 91 and a storage unit 92 connected to the processor 91, the storage unit 92 is used to store a computer program and sequence data, and the processor 91 is used to execute the computer program to implement the following method on the storage device:

[0187] The storage device performs data processing operations in the form of a sequence data stream, and the method includes: the storage device receives sequence data sent by the host end; the storage device stores the sequence data in the form of a sequence data stream and sets a corresponding sequence data index; the storage device receives a data operation instruction for the sequence data sent by the host end, and performs corresponding processing on the corresponding sequence data based on the sequence data index.

[0188] It can be understood that the processor 91 is used to execute a computer program to implement the method of any of the above embodiments on the storage device.

[0189] In some embodiments, the storage device 90 further includes a communication interface module, and the storage device 90 is communicatively connected to the host end via the communication interface module.

[0190] See also Fig.10 , Fig.10 The storage system 100 includes a host 101 and a storage device 90 .

[0191] The storage device 90 is connected to the host terminal 101 for communication, and the storage device 90 can implement the following method:

[0192] The storage device performs data processing operations in the form of a sequence data stream, and the method includes: the storage device receives sequence data sent by the host end; the storage device stores the sequence data in the form of a sequence data stream and sets a corresponding sequence data index; the storage device receives a data operation instruction for the sequence data sent by the host end, and performs corresponding processing on the corresponding sequence data based on the sequence data index.

[0193] It can be understood that the storage device 90 can implement the method of any of the above embodiments.

[0194] In this embodiment, compared with the memory of the block storage interface used in the related art, such as SSD (Solid State Disk), UFS (Universal Flash Storage), SD card (Secure Digital Memory Card) and U disk, the sequence data is stored in the form of a data stream, and the corresponding data stream identifier is set, which can greatly simplify the host software stack on the host side, reduce the occupancy of the host processor by the host storage access, and improve the storage performance in practical applications; on the storage device side, various storage media can be organized more effectively, unnecessary garbage collection can be reduced, the wear of the storage medium can be reduced, the service life of the storage device can be increased, the use of over-allocated space and dynamic random access memory can be reduced, and the cost of the storage device can be reduced.

[0195] Furthermore, the sequence data stored according to the data stream can be read out simultaneously according to the data stream, or corresponding operations can be performed. Therefore, the sequence data is more complete and convenient for subsequent processing.

[0196] In the several embodiments provided in this application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementation described above is only illustrative, for example, the division of the modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed.

[0197] If the integrated units in the above other embodiments are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) or a processor (processor) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, RandomAccess Memory), disk or optical disk and other media that can store program codes.

[0198] The above description is only an implementation method of the present application, and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly used in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A method for processing sequence data, characterized in that: Applied to a storage device, the storage device performs data processing operations in the form of a serial data stream, the method comprising: The storage device receives sequence data sent by the host end; The storage device stores the sequence data in the form of a sequence data stream and sets a corresponding sequence data index; The storage device receives a data operation instruction for the sequence data sent by the host, and performs corresponding processing on the corresponding sequence data based on the sequence data index.

2. The method according to claim 1, characterized in that The storage device comprises at least one sequence container, each of the sequence containers stores one or more sequence data, and each of the sequence containers has a unique container ID; The sequence container sets properties of the sequence container when it is created and / or after it is created; wherein the properties of the sequence container store first description information of the stored sequence data, and the first description information is used to enable the host end to determine a method for accessing and using related sequence data through the first description information.

3. The method according to claim 2, characterized in that In response to deleting a sequence container in the storage device, deleting all sequence data stored in the sequence container; When deleting the sequence container, in response to the storage device being set to a data protection mode, deleting the sequence container is not allowed.

4. The method according to claim 2, characterized in that: There is at least one sequence data stream in each of the sequence containers, and each of the sequence data streams stores sequence data of the same type and has a unique sequence data stream ID; The storage device deletes all sequence data on the sequence data stream corresponding to the sequence data stream deletion instruction in response to the sequence data stream deletion instruction, wherein the sequence data stream can be set to be unable to be deleted; The sequence data stream sets the attributes of the sequence data stream after being created; the attributes of the sequence data stream include second description information of the data stored in the sequence data stream.

5. The method according to claim 2, characterized in that: The sequence container has a unique sequence space, the sequence space has a corresponding sequence ID, and the sequence ID is used to describe the position of sequence data in the sequence data stream in the sequence container in the sequence space; The new and old relationship of the sequence data in the sequence data stream is determined by the size of the sequence ID; The sequence data stream ID of the sequence data stream and the sequence ID constitute the sequence data index. In the sequence container, the sequence data index corresponds to unique sequence data, and other sequence data can only replace the original sequence data or be added to the original sequence data.

6. The method according to claim 5, characterized in that The sequence container supports a cyclic cover function, and the cyclic cover function is provided with a reserved sequence ID length; When the cyclic overwrite function is enabled, in response to new sequence data being written into the sequence container, sequence data outside the interval formed by the new sequence ID and the length of the reserved sequence ID is deleted.

7. The method according to claim 2, characterized in that Use a sequence data index set to identify multiple sequence data indexes, use them in sequence data instructions, and operate on multiple sequence data at the same time; The sequence data index set is a combination of multiple sequence data indexes, or the sequence data index set is a combination of multiple sequence data stream IDs and sequence IDs; or, The sequence data index set is a combination of a sequence ID interval and at least one sequence data stream ID, and is used to identify all sequence data of at least one sequence data stream within the sequence ID interval; wherein the sequence ID interval is used to represent the set of all sequence IDs within the interval.

8. The method according to claim 7, characterized in that In response to a sequence data deletion instruction, the sequence data corresponding to the sequence data index set in the target sequence container corresponding to the data deletion instruction is deleted.

9. The method according to claim 7, characterized in that: In response to a sequence data compression instruction, compress the sequence data corresponding to the data index set in the sequence data compression instruction, and use the compressed sequence data to replace the original sequence data; In response to the sequence data stream being set to a compression mode, compressing the sequence data written into the sequence data stream using a given compression algorithm; Before reading or calculating the compressed sequence data, the compressed sequence data is decompressed using a given decompression algorithm to obtain the sequence data before compression.

10. The method according to claim 7, characterized in that The storage device performs a merging operation on the sequence data on the sequence data stream corresponding to the sequence data merging instruction in response to the sequence data merging instruction.

11. The method according to claim 7, characterized in that In response to a data protection instruction, data protection is enabled for the sequence data corresponding to the data index set in the data protection instruction, so that the sequence data corresponding to the data index set and the corresponding sequence data index cannot be deleted or modified; In response to the sequence container and / or the sequence data stream being set as a data protection option, all sequence data on the sequence data stream corresponding to the data protection option and the corresponding sequence data index cannot be deleted or modified.

12. The method according to claim 2, characterized in that: In response to the data protection function of the sequence container or the sequence data stream being enabled, deletion and / or modification of the sequence data already written in the protected data stream is not allowed; and / or, in response to the data protection function of the sequence container or the sequence data stream being turned on, only sequence data whose sequence ID is newer than the latest sequence ID in the sequence data stream is allowed to be written into the protected sequence data stream; and / or, in response to the data protection function of the sequence container or the sequence data stream being enabled, only sequence data whose sequence ID is within the sequence ID protection interval is allowed to be written to all sequence data streams in the sequence container; The sequence ID protection interval is determined by the length of the interval between the latest sequence ID in the sequence container and the sequence ID protection zone in the sequence container data protection function.

13. The method according to claim 12, characterized in that When the data protection function of the sequence data stream is turned on, in response to turning on the sequence data stream protection function, a sequence signature of the corresponding sequence data is added to the signature field of the sequence data tag to protect the data integrity of the corresponding data stream.

14. The method according to claim 13, characterized in that Each sequence container with data protection function enabled has a sequence signature private key, which cannot be obtained externally. The sequence signature public key corresponding to the sequence signature private key is stored in the attributes of the sequence container and is used to verify the data integrity of the protected sequence data stream; wherein the attributes of the sequence container include the sequence signature public key, related information, and the signature of the sequence storage security private key on the sequence signature public key and related information; In response to the first verification instruction, verify the signature of the related information using the sequence memory security public key obtained by the sequence memory to prove the integrity of the sequence signature public key and the related information; In response to the second verification instruction, the sequence data signature is used to verify the integrity of the current sequence ID, the current sequence data, the current sequence data tag, and the sequence ID and signature of the previous sequence data; wherein the sequence data signature is obtained by signing the sequence ID and signature of the previous sequence data in the sequence data stream, the current sequence ID, the current sequence data, and the current sequence data tag using the sequence signature private key.

15. The method according to claim 13, characterized in that In response to enabling the sequence data stream protection function, the sequence ID and signature information of the latest sequence data on all protected sequence data streams in the sequence container are signed to obtain a container signature, and the container signature is stored in the attributes of the sequence container. The container signature is used to ensure the integrity of all protected sequence data streams in the sequence storage.

16. The method according to claim 7, characterized in that In response to the sequence data encryption instruction, encrypt and store the sequence data corresponding to the sequence data index set in the sequence data encryption instruction; In response to a sequence container enabling a sequence data encryption function or a sequence data stream enabling the sequence data encryption function, the corresponding sequence data is encrypted and stored using a designated sequence encryption key; wherein the sequence encryption key is obtained by key generation or exchange.

17. The method according to claim 7, characterized in that The configuration range of the configuration information of the sequence data stream is within the configuration range of the configuration information of the sequence container.

18. The method according to claim 17, characterized in that The configuration information of the sequence data stream includes at least one of data storage requirements, size, format, label, query method and operation method.

19. The method according to claim 18, characterized in that At least one sequence data tag is configured on the sequence data stream, and the configuration content includes the type, length, default value, and at least one of the tag operations that can be performed; In response to all sequence data written or modified on the sequence data stream, and the sequence data stream is configured with the sequence data tag, storing the data of the configured sequence data tag as a supplementary part of the sequence data index; In response to an operation instruction on the sequence data stream, performing a rule operation on at least one tag item in the sequence data tag, and when a result of the rule operation meets the requirement, performing an operation corresponding to the operation instruction on the sequence data stream; In response to the sequence data stream being configured with the sequence data tag and the operation instruction requiring the use of a sequence data index set, the sequence data index set is filtered and adjusted using a tag filtering operation to obtain a filtered sequence data index set.

20. The method according to claim 2, characterized in that The sequence container includes at least one sequence data stream; in response to locking the configuration of the sequence data stream, no new sequence data stream can be added to the sequence container, and configuration information of the at least one data stream cannot be modified.

21. The method according to claim 5, characterized in that The sequence ID is determined by the time when the sequence data is generated on the host side.

22. The method according to claim 21, characterized in that The sequence data includes audio and video data; description information of the audio and video data is saved in the attributes of the sequence container; The audio or video stream data from the same source and / or the same encoding in the audio and video data are stored using the same sequence data stream, and the corresponding audio or video stream information is saved in the corresponding sequence data stream attributes; The audio or video stream data from different sources or with different encodings are stored using different sequence data streams.

23. The method according to claim 4, characterized in that In response to a counting operation on at least one sequence data stream, counting the number of data on a given sequence data index expression in the at least one sequence data stream; Alternatively, in response to a list operation on at least one sequence data stream, the number, data size and / or label of data within a given index interval in the at least one sequence data stream is determined.

24. The method according to claim 1, characterized in that The sequence data includes audio and video data. The storage device responds to the transcoding instruction and uses the codec circuit in the storage device to decode and encode the audio and video data of the first coding standard corresponding to the transcoding instruction to obtain the audio and video data of the second coding standard.

25. The method according to claim 1, characterized in that The sequence data is blockchain data; The storage device stores the sequence data in the form of a sequence data stream and sets a corresponding sequence data index, including: The height value of the blockchain data is used as a sequence ID, the block data of the blockchain data is stored in the form of a data stream, and the corresponding sequence data index is set.

26. The method according to claim 25, characterized in that The blockchain data includes main chain data and forked chain data; The method of using the height value of the blockchain data as a sequence ID, storing the block data of the blockchain data in the form of a data stream, and setting the corresponding sequence data index includes: The height value of the main chain data is used as a sequence ID, and the main chain data is stored using a first data stream. The height value of the forked chain data is used as a sequence ID, and the forked chain data is stored using a second data stream.

27. The method according to claim 25, characterized in that The blockchain data includes main chain data and side chain data; The method of using the height value of the blockchain data as a sequence ID, storing the block data of the blockchain data in the form of a data stream, and setting the corresponding sequence data index includes: The height value of the main chain data is used as a sequence ID, and the main chain data is stored using a first data stream. The height value of the side chain data is used as a sequence ID, and the side chain data is stored using a second data stream.

28. A storage device, characterized in that: The storage device includes a processor, a storage unit, a memory and an interface unit. The storage unit is used to store computer programs and sequence data. The processor is used to execute the computer program to implement the method described in any one of claims 1 to 27 on the storage device. The interface unit is connected to the processor of the host side to complete data communication between the host side and the storage device.

29. A storage system, characterized in that: The storage system comprises: Host side; A storage device, wherein the storage device is communicatively connected to the host end, and the storage device is the storage device as claimed in claim 28.

Images