Object storage method, device and equipment based on time sequence and storage medium

Abstract

The invention relates to the technical field of data storage, and discloses a time sequence-based object storage method, device and equipment and a storage medium, and the method comprises the following steps: receiving a storage object uploaded by a client through an object storage system; acquiring flow information of the storage objects through a metadata storage component, and storing the storage objects in the object storage system according to the flow information and a time sequence; and when a user needs to query data, acquiring bucket information and time information input by the client through the metadata storage component, scanning the object storage system according to the bucket information and the time information, acquiring a storage object corresponding to the bucket information and the time information, and returning the storage object to the client. According to the method and the device, the flow information of the storage object is acquired by utilizing the metadata storage component, and the storage object is stored according to the time sequence according to the flow information, so that a user can acquire the storage object according to time traversal, and the query quantity of data is reduced.

Description

Technical Field

[0001] The present invention relates to the field of data storage technology, and in particular to a time-series-based object storage method, device, equipment and storage medium. Background Art

[0002] An object refers to a file and any metadata that describes the file. Amazon Simple Storage Service (Amazon S3) is an object storage service that stores data in buckets as objects. Buckets are containers for storing objects. When users store objects, they first need to create a bucket, then specify the bucket name and AWS region, and then upload the data as an object in Amazon S3 to the bucket to store massive files.

[0003] Amazon S3's traversal interface is sorted alphabetically. When users need to query uploaded data within a certain time period, the original query method needs to traverse all the data in the traversal interface, and then compare the time one by one before returning it to the user, resulting in a large amount of data query and unable to meet the user's needs for traversal by data upload time. For example, when a user needs to query all data uploaded between 2024-5-24 and 2024-5-30, it is necessary to traverse all the data in the traversal interface, and then compare the upload time of all the data to find the data from 2024-5-24 to 2024-5-30 before returning it to the user. If the bucket is large and data is continuously written, it is impossible to traverse the data within the user's selected time period. Summary of the invention

[0004] The present invention provides a time-series-based object storage method, device, equipment and storage medium to solve the technical problem that the traversal interface of the object storage service in the prior art is based on alphabetical sorting and cannot satisfy the user's traversal by data upload time.

[0005] In a first aspect, a time series-based object storage method is provided, including:

[0006] Receive the storage object uploaded by the client through the object storage system;

[0007] Acquire the flow information of the storage object through the metadata storage component, and store the storage object in the object storage system in time sequence according to the flow information;

[0008] When the user needs to query data, the metadata storage component obtains the bucket information and time information input by the client, scans the object storage system according to the bucket information and time information, obtains the storage object corresponding to the bucket information and time information, and returns the storage object to the client.

[0009] In a second aspect, a time series-based object storage device is provided, including:

[0010] Data receiving module: used to receive storage objects uploaded by the client through the object storage system;

[0011] Data storage module: used for obtaining the flow information of the storage object through the metadata storage component, and storing the storage object in the object storage system in time sequence according to the flow information;

[0012] Data query module: when the user needs to query data, it obtains the bucket information and time information input by the client through the metadata storage component, scans the object storage system according to the bucket information and time information, obtains the storage object corresponding to the bucket information and time information, and returns the storage object to the client.

[0013] In a third aspect, a computer device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned time-series-based object storage method when executing the computer program.

[0014] In a fourth aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above-mentioned time-series-based object storage method are implemented.

[0015] In the scheme implemented by the above-mentioned time-series-based object storage method, device, computer equipment and storage medium, a corresponding metadata storage component is set in the object storage system, and the metadata storage component is used to obtain the flow information of the storage object, and the storage object is stored in time sequence according to the flow information, so that when the user queries the data, the storage object within the set time range can be traversed according to the time, without traversing all the data, which greatly reduces the amount of data query, shortens the response time of data query, and improves the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0017] Figure 1 is an architecture diagram of a time-series-based object storage system in one embodiment of the present invention;

[0018] Figure 2 is a flow chart of a time-series-based object storage method in a first embodiment of the present invention;

[0019] Figure 3 is a flow chart of a time-series-based object storage method in a second embodiment of the present invention;

[0020] Figure 4 is a schematic diagram of a structure of a time-series-based object storage device in one embodiment of the present invention;

[0021] Figure 5 is a schematic diagram of a structure of a computer device in one embodiment of the present invention;

[0022] Figure 6 It is another structural schematic diagram of a computer device in one embodiment of the present invention. DETAILED DESCRIPTION

[0023] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0024] The time series-based object storage method provided by the embodiment of the present invention can be applied in the following aspects: Figure 1In the application environment, the client communicates with the server through the network. The server can receive the storage object uploaded by the client through the object storage system, obtain the flow information of the storage object through the metadata storage component, and store the storage object in the object storage system in time sequence according to the flow information. When the user needs to query data, the metadata storage component obtains the bucket information and time information input by the client, scans the object storage system according to the bucket information and time information, obtains the storage object corresponding to the bucket information and time information, and returns the storage object to the client. The present invention is applicable to object storage in the insurance field. For object storage in the insurance field, the flow information of the storage object can be obtained through the metadata storage component, and the storage object can be stored in the object storage system in time sequence according to the flow information; when the user needs to query data, the metadata storage component quickly obtains the storage object within the time range set by the user, which greatly improves the data query efficiency. Among them, the client can be but not limited to various personal computers, laptops, smart phones, tablet computers and portable wearable devices. The server can be implemented by an independent server or a server cluster composed of multiple servers. The present invention is described in detail below through specific embodiments.

[0025] See also Figure 2 , is a flow chart of a time-series-based object storage method in a first embodiment of the present invention. The time-series-based object storage method provided in the first embodiment of the present invention comprises the following steps:

[0026] S100: receiving a storage object uploaded by a client through an object storage system;

[0027] In this step, the storage object includes the file and any metadata describing the file. When the client uploads the storage object, it needs to create a storage bucket. Each storage bucket has its own bucket information (bucket), including name, usage analysis, monitoring alarm, domain name information, etc.

[0028] S110: Obtaining the flow information of the storage object through the metadata storage component, and storing the storage object in a time sequence in the object storage system according to the flow information;

[0029] In this step, in order to solve the problem that the existing object storage service cannot traverse the storage objects according to time, the present invention sets the corresponding metadata storage component TIKV in the object storage system. TiKV is a distributed transactional key-value database. As the distributed storage engine of TiDB, TiKV's distributed architecture provides a distributed transaction interface that meets ACID constraints, and ensures the consistency and high availability of multiple copies of data through the Raft protocol. Among them, ACID constraints are a theory proposed by database management systems to ensure the correctness of transactions, which includes four constraints: atomicity, consistency, isolation, and durability.

[0030] Specifically, the transaction information obtained through the metadata storage component includes bucket information (bucket), the write time of the storage object (time), and the unique identifier (key) corresponding to the storage object in the bucket. The storage format of the object stored according to the transaction information is: bucket "time" key.

[0031] It can be understood that the present invention uses the metadata storage component TIKV to obtain the flow information of the storage object, and stores the storage object in time sequence according to the flow information, so that the user can traverse the storage object according to time when querying data.

[0032] S120: When the user needs to query data, the metadata storage component Tikv obtains the bucket information and time information input by the client, scans the object storage system according to the bucket information and time information, obtains the storage object corresponding to the bucket information and time information, and returns the storage object to the client;

[0033] In this step, since the bucket information of each user's storage object is different, it is necessary to query based on bucket when querying objects. The time information is the time range that the user needs to query data. If the user needs to query storage objects within a certain time range, the time information corresponding to the time range needs to be entered. The time information includes the start time and the end time.

[0034] Specifically, the metadata storage component Tikv in the present invention uses a scan algorithm to scan the object storage system. The scan algorithm is an algorithm used in the disk scheduling process. The scan algorithm includes two different implementation methods: sequential scanning method and cyclic scanning method for data query. The scanning process of the Scan algorithm is specifically: taking the start time of the bucket and the time range as the start character, taking the end time of the bucket and the time range as the end character, based on the start character and the end character, using the Scan algorithm to traverse and scan the object storage system, and obtain all storage objects in the corresponding bucket from the start time to the end time.

[0035] It can be seen that in the above scheme, the time-series-based object storage method provided by the embodiment of the present invention sets a corresponding metadata storage component TIKV in the object storage system, uses the metadata storage component TIKV to obtain the flow information of the storage object, and stores the storage object in time sequence according to the flow information, so that when the user queries the data, he can traverse the time to obtain the storage objects within the set time range without traversing all the data, which greatly reduces the amount of data query, shortens the response time of data query, and improves the user experience.

[0036] See also Figure 3 , is a flow chart of a time-series-based object storage method in a second embodiment of the present invention. The time-series-based object storage method provided in the second embodiment of the present invention includes the following steps:

[0037] S200: receiving a storage object uploaded by a client through an object storage system;

[0038] In this step, the uploading process of the storage object is the same as S100 in the first embodiment, and will not be described again here to avoid redundancy.

[0039] S210: Obtaining the flow information of the storage object through the metadata storage component, the flow information includes bucket information (bucket), the writing time of the storage object (time) and the unique identifier (key) corresponding to the storage object in the bucket, and converting the writing time into UNIX format;

[0040] In this step, in order to facilitate the user to perform accurate data query within the corresponding time range according to the write time of the storage object, the present invention converts the write time (time) of the storage object into UNIX format after receiving the write time (time) of the storage object. For example, if the write time of the storage object is 2022-01-10 13:00:00, the write time is converted to UNIX format as 1641819600.

[0041] S220: storing the storage objects in the object storage system in time sequence according to the converted flow information;

[0042] In this step, the storage objects are stored in the object storage system in time sequence according to the converted transaction information. Specifically, assuming that the write time of the two storage objects is 2022-01-10 13:00:00 and 2022-01-10 13:00:01 respectively, then according to the converted transaction information, the two storage objects are stored in time sequence as: bucket "1641819600" key1, bucket "1641819601" key2.

[0043] S230: When the user needs to query data, the metadata storage component Tikv obtains the bucket information and time information input by the client, and converts the time information into UNIX format;

[0044] In this step, since the time information of the storage object is in UNIX format, after receiving the time information input by the client, it is also necessary to convert the time information into UNIX format.

[0045] S240: Scan the object storage system according to the bucket information and the converted time information, obtain the storage object corresponding to the bucket information and the time information, and return the storage object to the client;

[0046] In this step, the scanning process of the object storage system is the same as S120 in the first embodiment, and will not be described again here to avoid redundancy.

[0047] It can be understood that the order of execution of the steps in the above embodiment does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.

[0048] In one embodiment, a time-series based object storage device is provided, and the time-series based object storage device corresponds one-to-one to the time-series based object storage method in the above embodiment. Figure 4 As shown, the time-series-based object storage device includes a data receiving module 101, a first format conversion module 102, a data storage module 103, a second format conversion module 104 and a data query module 105. The functional modules are described in detail as follows:

[0049] Data receiving module 101: used to receive storage objects uploaded by clients through the object storage system; wherein the storage objects include files and any metadata describing the files. When the client uploads the storage objects, it needs to create storage buckets, each of which has its own bucket information (bucket), including name, usage analysis, monitoring alarms, domain name information, etc.

[0050] The first format conversion module 102: is used to obtain the flow information of the storage object through the metadata storage component, and the flow information includes bucket information (bucket), the write time (time) of the storage object and the unique identifier (key) corresponding to the storage object in the bucket, and convert the write time into UNIX format; wherein, in order to solve the problem that the existing object storage service cannot traverse the storage object according to time, the present invention sets a corresponding metadata storage component TIKV in the object storage system. TiKV is a distributed transactional key-value database. As the distributed storage engine of TiDB, TiKV's distributed architecture provides a distributed transaction interface that meets ACID constraints, and ensures the consistency and high availability of multiple copies of data through the Raft protocol. Among them, ACID constraints are a theory proposed by the database management system to ensure the correctness of transactions, which includes four constraints: atomicity, consistency, isolation, and durability). In order to facilitate users to perform accurate data query within the corresponding time range according to the write time of the storage object, the present invention converts the write time (time) of the storage object into UNIX format after receiving the write time (time) of the storage object. For example, if the write time of the storage object is 2022-01-1013:00:00, the write time is converted to UNIX format as 1641819600.

[0051] Data storage module 103: used to store the storage objects in the object storage system in time sequence according to the converted transaction information; wherein, assuming that the write times of the two storage objects are 2022-01-1013:00:00 and 2022-01-1013:00:01 respectively, then the two storage objects are stored in time sequence according to the converted transaction information as: bucket "1641819600" key1, bucket "1641819601" key2.

[0052] It can be understood that the present invention uses the metadata storage component TIKV to obtain the flow information of the storage object, and stores the storage object in time sequence according to the flow information, so that the user can traverse the storage object according to time when querying data.

[0053] The second format conversion module 104 is used to obtain the bucket information and time information input by the client through the metadata storage component when the user needs to query data, and convert the time information into UNIX format; wherein, since the bucket information of each user storage object is different, it is necessary to query based on the bucket when querying the object. The time information is the time range for which the user needs to query the data. If the user needs to query the storage object within a certain time range, it is necessary to enter the time information corresponding to the time range, and the time information includes the start time and the end time. Since the time information of the storage object is in UNIX format, after receiving the time information input by the client, it is also necessary to convert the time information into UNIX format.

[0054] Data query module 105: used to scan the object storage system according to the bucket information and time information, obtain the storage object corresponding to the bucket information and time information, and return the storage object to the client; wherein, the data query module 105 uses the scan algorithm to scan the object storage system. The scan algorithm is an algorithm used in the disk scheduling process. The scan algorithm includes two different implementation methods: sequential scanning method and cyclic scanning method for data query. The scanning process of the Scan algorithm is specifically as follows: taking the start time of the bucket and the time range as the start character, taking the end time of the bucket and the time range as the end character, based on the start character and the end character, using the Scan algorithm to traverse and scan the object storage system, and obtain all storage objects in the corresponding bucket from the start time to the end time. For example, if the time range that the user needs to query is: 2022-01-10 13:00:00 to 2022-01-10 13:00:01, the start time 2022-01-10 13:00:00 and the end time 2022-01-10 13:00:01 are converted to UNIX format, that is, 1641819600 and 1641819601 respectively, and then bucket|1641819600| is used as the start character and bucket|1641819601| is used as the end character. The object storage system is traversed according to the start character and the end character to obtain all the keys in the bucket within the time range from 2022-01-10 13:00:00 to 2022-01-10 13:00:01.

[0055] The time-series-based object storage device provided by the present invention sets a corresponding metadata storage component TIKV in the object storage system, uses the metadata storage component TIKV to obtain the flow information of the storage object, and stores the storage object in time sequence according to the flow information, so that when querying data, the user can traverse the time to obtain the storage objects within the set time range without traversing all the data, which greatly reduces the amount of data query, shortens the response time of data query, and improves the user experience.

[0056] For the specific definition of the time-series-based object storage device, please refer to the definition of the time-series-based object storage method above, which will not be repeated here. Each module in the above-mentioned time-series-based object storage device can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0057] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 5 As shown. The computer device includes a processor, a memory, a network interface and a database connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile and / or volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external client via a network connection. When the computer program is executed by the processor, it implements the functions or steps on the server side of a time-based object storage method.

[0058] In one embodiment, a computer device is provided. The computer device may be a client, and its internal structure diagram may be as follows: Figure 6 As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected via a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external server via a network connection. When the computer program is executed by the processor, it implements the functions or steps of the client side of a time-based object storage method.

[0059] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the following steps when executing the computer program:

[0060] Receive the storage object uploaded by the client through the object storage system;

[0061] Acquire the flow information of the storage object through the metadata storage component, and store the storage object in the object storage system in time sequence according to the flow information;

[0062] When the user needs to query data, the metadata storage component obtains the bucket information and time information input by the client, scans the object storage system according to the bucket information and time information, obtains the storage object corresponding to the bucket information and time information, and returns the storage object to the client.

[0063] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0064] Receive the storage object uploaded by the client through the object storage system;

[0065] Acquire the flow information of the storage object through the metadata storage component, and store the storage object in the object storage system in time sequence according to the flow information;

[0066] When the user needs to query data, the metadata storage component obtains the bucket information and time information input by the client, scans the object storage system according to the bucket information and time information, obtains the storage object corresponding to the bucket information and time information, and returns the storage object to the client.

[0067] It should be noted that the above functions or steps that can be implemented by the computer-readable storage medium or computer device can refer to the relevant descriptions on the server side and the client side in the aforementioned method embodiment. To avoid repetition, they will not be described one by one here.

[0068] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other storage medium used in the embodiments provided by the present invention can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0069] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0070] The embodiments described above are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the protection scope of the present invention.

Claims

1. A time series-based object storage method, characterized in that: include: Receive the storage object uploaded by the client through the object storage system; Acquire the flow information of the storage object through the metadata storage component, and store the storage object in the object storage system in time sequence according to the flow information; When the user needs to query data, the metadata storage component obtains the bucket information and time information input by the client, scans the object storage system according to the bucket information and time information, obtains the storage object corresponding to the bucket information and time information, and returns the storage object to the client.

2. The time series-based object storage method according to claim 1, characterized in that: The flow information obtained through the metadata storage component includes bucket information, write time of the storage object, and a unique identifier corresponding to the storage object in the bucket.

3. The time series-based object storage method according to claim 2, characterized in that: After obtaining the flow information of the storage object through the metadata storage component, the method further includes: The write time of the storage object is converted into UNIX format.

4. The time series-based object storage method according to claim 3, characterized in that: The storage format of storing the storage object in the object storage system in time sequence according to the flow information is: bucket "time" key, where bucket is the bucket information, time is the write time in UNIX format, and key is the unique identifier corresponding to the storage object in the bucket.

5. The time series-based object storage method according to claim 4, characterized in that: After obtaining the bucket information and time information input by the client through the metadata storage component, the method further includes: The time information is converted into UNIX format; wherein the time information is the time range of the user's query data, including the start time and the end time.

6. The time-series-based object storage method according to claim 5, characterized in that: The scanning the object storage system according to the bucket information and the time information to obtain the storage object corresponding to the bucket information and the time information includes: The bucket information and the start time are used as the start characters, and the bucket information and the end time are used as the end characters. Based on the start characters and the end characters, the Scan algorithm is used to traverse and scan the object storage system to obtain all storage objects in the corresponding bucket from the start time to the end time.

7. The time series-based object storage method according to any one of claims 1 to 6, characterized in that: The metadata storage component is TIKV.

8. A time-series-based object storage device, characterized in that: include: Data receiving module: used to receive storage objects uploaded by the client through the object storage system; Data storage module: used for obtaining the flow information of the storage object through the metadata storage component, and storing the storage object in the object storage system in time sequence according to the flow information; Data query module: when the user needs to query data, it obtains the bucket information and time information input by the client through the metadata storage component, scans the object storage system according to the bucket information and time information, obtains the storage object corresponding to the bucket information and time information, and returns the storage object to the client.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the time-series-based object storage method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the time-series-based object storage method according to any one of claims 1 to 7 are implemented.

Images