Data storage method and data storage device
By using the unique identification and timestamp of data in the Internet of Vehicles field to determine multi-level storage paths, data hierarchical storage and data descriptions are realized, the problems of high cost of massive data storage and low query efficiency in traditional solutions are solved, and efficient storage and The balance of query.
Patent Information
- Application Number
- CN202411162819.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-23
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2044-08-23
AI Technical Summary
Traditional data storage solutions are difficult to meet the real-time query and long-term storage needs of massive data in the field of Internet of Vehicles, and are expensive.
By obtaining the unique identification and timestamp of the data, determining the multi-level storage path, realizing data hierarchical storage, and saving data descriptions in the data storage file, ensuring data migration and controllable storage costs.
It realizes efficient storage and query efficiency of massive data, reduces storage costs, improves data migration, and balances storage costs and query efficiency by dynamically adjusting directory structure and file state conversion.
Smart Images

Figure CN118672519B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of storage technology, and in particular to a data storage method and a data storage device. Background Art
[0002] With the rapid development of Internet of Vehicles technology, the amount of data generated by vehicles has exploded. Vehicles will report tens of thousands of different signal data of various components to cloud servers. These data are not only used for real-time indicator analysis and short-term, medium-term and long-term data statistics, but are also often needed for personalized queries of historical data. Due to the complexity of vehicle components, large signal volume, and the retransmission mechanism of data transmission, traditional data storage solutions are either costly or unable to meet the needs of real-time query and long-term storage. Therefore, how to achieve efficient storage of Internet of Vehicles data while ensuring that storage costs are controllable has become a problem that needs to be solved urgently. Summary of the invention
[0003] In view of this, an embodiment of the present application provides a data storage method and a data storage device to solve the problems of high cost and inconvenient query of massive data in the prior art.
[0004] A first aspect of an embodiment of the present application provides a data storage method, including:
[0005] Obtain the data to be stored, and determine the unique identifier and timestamp of the data to be stored;
[0006] Determine a first storage path based on the unique identifier and timestamp of the data to be stored, the first storage path includes at least a first-level directory, a second-level directory, and a target i-th data storage file, the target i-th data storage file is a data storage file in the i-level directory, and the target i-th data storage file includes data description and time information, i is a positive integer greater than 1;
[0007] In response to determining that the target i-th data storage file satisfies a preset storage condition, storing the data to be stored in the target i-th data storage file;
[0008] The preset storage condition includes that the time information of the target i-th data storage file does not satisfy the time splitting condition, and the data capacity of the target i-th data storage file does not satisfy the capacity splitting condition.
[0009] According to a second aspect of the present application, there is provided a data storage device, including:
[0010] An acquisition module is configured to acquire the data to be stored and determine a unique identifier and a timestamp of the data to be stored;
[0011] a determination module configured to determine a first storage path based on a unique identifier and a timestamp of the data to be stored, the first storage path comprising at least a first-level directory, a second-level directory, and a target i-th data storage file, the target i-th data storage file being a data storage file in the i-level directory, and the target i-th data storage file comprising a data description and time information, where i is a positive integer greater than 1;
[0012] A storage module, configured to store the data to be stored in the target i-th data storage file in response to determining that the target i-th data storage file satisfies a preset storage condition;
[0013] The preset storage condition includes that the time information of the target i-th data storage file does not satisfy the time splitting condition, and the data capacity of the target i-th data storage file does not satisfy the capacity splitting condition.
[0014] Compared with the prior art, the embodiments of the present application have the following beneficial effects: the embodiments of the present application determine a first storage path for the data to be stored by acquiring a unique identifier and a timestamp in the data to be stored, the first storage path comprising at least a primary directory, a secondary directory and a target i-th data storage file, and when it is determined that the time information of the determined target i-th data storage file does not satisfy the time splitting condition and the data capacity of the target i-th data storage file does not satisfy the capacity splitting condition, the data to be stored is stored in the target i-th data storage file, thereby realizing hierarchical storage of data, and in addition to the stored data, a data description is additionally saved in the data storage file, so that the stored data has high portability, thereby realizing efficient storage of the data to be stored with controllable storage costs.
[0015] Furthermore, the device embodiment also provides a file storage and directory expansion method for each level of directories in multi-level data storage, that is, data in each directory at the second level and below is preferentially stored in the form of files. When the total capacity of files in the directory exceeds a preset capacity threshold, or meets other conditions specified by other users, the directory is split into a next-level directory group, and each directory in the next-level directory group continues to store the files. The level of the data storage system, the number of directories in the directory group, the number of files in the directory, and the conditions for directory splitting can all be dynamically adjusted, thereby providing a clear and concise hierarchical storage structure, ensuring data writing efficiency and saving storage space, while also speeding up data updates and queries.
[0016] Furthermore, the data to be stored is saved in a temporary state file, which can be converted into a stable state file when a first conversion condition is met, and the stable state file can be converted into a dormant state file when a second conversion condition is met, and the dormant state file can also be awakened. That is, the state of the storage file can be converted between different states, which reduces the data storage cost and achieves a balance between storage cost and query efficiency.
[0017] Furthermore, the data storage method of the present application can achieve read-write separation regardless of data storage or file status conversion of stored data. All operations ensure particle nature and do not affect other operations, thereby increasing the robustness of the data storage system. BRIEF DESCRIPTION OF THE DRAWINGS
[0018] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0019] Figure 1 It is a flow chart of a data storage method provided in an embodiment of the present application.
[0020] Figure 2 It is a schematic diagram of the hierarchical structure in the hierarchical storage solution provided in an embodiment of the present application.
[0021] Figure 3 It is a flowchart of a method for determining a first storage path based on a unique identifier and a timestamp of data to be stored when i is equal to 2 provided in an embodiment of the present application.
[0022] Figure 4 It is a flowchart of a method for determining a first storage path based on a unique identifier and a timestamp of data to be stored when i is greater than 2 provided in an embodiment of the present application.
[0023] Figure 5 It is a flow chart of another data storage method provided in an embodiment of the present application.
[0024] Figure 6 It is a flow chart of another data storage method provided in an embodiment of the present application.
[0025] Figure 7 It is a schematic diagram of the storage method of different status files provided in the embodiment of the present application.
[0026] Figure 8 It is a flowchart of a method for converting a target temporary state file into a target stable state file provided in an embodiment of the present application.
[0027] Fig. 9 It is a flowchart of a method for writing data in a target temporary state file into a first stable state file provided in an embodiment of the present application.
[0028] Fig.10 It is a flowchart of a method for converting a target stable state file into a target dormant state file provided in an embodiment of the present application.
[0029] Fig.11 It is a schematic diagram of converting a target stable state file into a target dormant state file provided by an embodiment of the present application.
[0030] Fig.12 It is a flowchart of a method for waking up a target dormant state file provided in an embodiment of the present application.
[0031] Fig.13 It is a flow chart of another data storage method provided in an embodiment of the present application.
[0032] Fig.14 It is a flowchart of a method for converting a temporary state file into a stable state file provided in an embodiment of the present application.
[0033] Fig.15 It is a schematic diagram of a data storage device provided in an embodiment of the present application.
[0034] Fig.16 It is a schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0035] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.
[0036] A data storage method and device according to an embodiment of the present application will be described in detail below with reference to the accompanying drawings.
[0037] As mentioned above, with the rapid development of Internet of Vehicles technology, the amount of data generated by vehicles has exploded. The vehicle will report tens of thousands of different signal data of various components to the cloud server. These data are not only used for real-time indicator analysis and short-term, medium-term and long-term data statistics, but are also often needed for personalized queries of historical data. Due to the complexity of the components of the vehicle computer, the large amount of signals, and the retransmission mechanism of data transmission, traditional data storage solutions are either costly or cannot meet the needs of real-time query and long-term storage. Therefore, how to achieve efficient storage of Internet of Vehicles data while ensuring that storage costs are controllable has become a problem that needs to be solved urgently.
[0038] In view of this, an embodiment of the present application provides a data storage method, which determines a first storage path of the data to be stored by obtaining a unique identifier and a timestamp in the data to be stored, and the first storage path includes at least a first-level directory, a second-level directory, and a target i-th data storage file. When it is determined that the time information of the determined target i-th data storage file does not meet the time splitting condition and the data capacity of the target i-th data storage file does not meet the capacity splitting condition, the data to be stored is stored in the target i-th data storage file, thereby realizing data hierarchical storage, and in addition to the stored data, the data storage file also saves data descriptions, so that the stored data has high portability, thereby realizing efficient storage of the data to be stored under controllable storage costs.
[0039] Figure 1 Schematic diagram of a data storage method provided by an embodiment of the present application. Figure 1 As shown, the method comprises the following steps:
[0040] In step S101, the data to be stored is obtained, and the unique identifier and timestamp of the data to be stored are determined.
[0041] In step S102, a first storage path is determined based on the unique identifier and timestamp of the data to be stored.
[0042] Among them, the first storage path includes at least a first-level directory, a second-level directory and a target i-th data storage file, the target i-th data storage file is a data storage file in the i-level directory, and the target i-th data storage file includes data description and time information, and i is a positive integer greater than 1.
[0043] In step S103 , in response to determining that the target i-th data storage file satisfies a preset storage condition, the data to be stored is stored in the target i-th data storage file.
[0044] The preset storage condition includes that the time information of the target i-th data storage file does not satisfy the time splitting condition, and the data capacity of the target i-th data storage file does not satisfy the capacity splitting condition.
[0045] In an embodiment of the present application, the data storage method can be executed by a data storage system. The data storage system can be installed on a terminal device or a server, and the server can be an independent server, a cluster server, or a cloud server. Furthermore, the data storage method in an embodiment of the present application can be used to store massive data such as Internet of Vehicles data.
[0046] In the embodiment of the present application, the data to be stored can be first obtained, and the data to be stored can include a unique identifier and a timestamp, wherein the unique identifier is used to represent the data to be stored, and each unique identifier is only mapped to a specific piece of data, so when updating and querying data, the unique identifier can be used to accurately locate the exact location of the data to be processed. The timestamp is used to mark the acquisition time of the data to be stored.
[0047] In the embodiment of the present application, the first storage path can be determined based on the unique identifier and timestamp of the data to be stored, and the first storage path includes at least a primary directory, a secondary directory, and a target i-th data storage file. That is to say, in the data storage system, multiple storage levels can be included, and each storage level also includes multiple directories. Each level, each directory, and each data storage file in the directory can be marked according to the data identifier and time information. Therefore, after obtaining the unique identifier and timestamp of the data to be stored, the corresponding first storage path can be determined based on the unique identifier and timestamp.
[0048] The target i-th data storage file includes data description and time information, wherein the time information may be the creation time of the data storage file, and the time information may be recorded in the data storage file in the form of a file name suffix, for example. Traditional data storage tools usually store the structure of data independently. For example, in the relational database management system MySQL, the table structure metadata is stored in the system tables; the metadata of the data warehouse tool Hive is stored in the independent service Hive Metastore. When querying data in MySQL, MySQL will first access the metadata (also called system tables or data dictionary) to obtain structural information about the table, and then retrieve the required data from the data file based on these metadata. This will increase query time and reduce query efficiency, and has low configuration flexibility and low portability.
[0049] In view of this, in the data storage method provided in the embodiment of the present application, the data description is directly saved in the data storage file, for example, saved in the beginning part of the data storage file, or saved in a specific column for a data storage file with column storage, and the data is saved in other parts of the data storage file. For massive data such as Internet of Vehicles data, since the data structure of each type of data, such as vehicle control data and vehicle operation data, usually does not change frequently after being determined, the data description in the data storage file usually does not change. Each time a new data storage file is created, the data description can be copied from the previous data storage file or the data storage file in the upper directory.
[0050] Furthermore, if the data structure of a certain type of data changes, or a new type of data structure is added, it is only necessary to uniformly modify the data descriptions in all files in all directories under the first-level directory of the type of data, or to add a new first-level directory to the data storage system and write the data descriptions of all files under the first-level directory as the data description of the newly added type of data.
[0051] In the embodiment of the present application, data description can be implemented in a variety of ways, such as using a description method in json format, a description method in xml format, or a description method in properties format. An example of data description implemented using a description method in json format can be:
[0052] {
[0053] "type": "record", / / data type
[0054] "name": "taable_name", / / data name
[0055] "status": "stable", / / data status
[0056] "fields": [ / / fields
[0057] {
[0058] "name": "_insert_time", / / Field name
[0059] "type": "timestamp-millis" / / Field type
[0060] },
[0061] {
[0062] "name": "_lase_select_time", / / Field name
[0063] "type": "timestamp-millis" / / Field type
[0064] },
[0065] {
[0066] "name": "_since_select_duration", / / Field name
[0067] "type": "long" / / Field type
[0068] },
[0069] {
[0070] "name": "uuid", / / field name
[0071] "type": "string" / / Field type
[0072] },
[0073] {
[0074] "name": "name", / / Field name
[0075] "type": "string" / / Field type
[0076] },
[0077] {
[0078] "name": "age", / / Field name
[0079] "type": "int" / / Field type
[0080] } ]
[0082] }
[0083] In the embodiment of the present application, after determining the first storage path, it can be determined whether the target i-th data storage file in the first storage path meets the preset storage condition, and if so, the data to be stored is stored in the target i-th data storage file. The preset storage condition may be a condition pre-specified by the user, for example, the time information of the target i-th data storage file does not meet the time splitting condition, and the data capacity of the target i-th data storage file does not meet the capacity splitting condition.
[0084] Further, the time information of the target i-th data storage file does not meet the time splitting condition, for example, the difference between the current time and the creation time of the target i-th data storage file does not exceed the preset duration. The data capacity of the target i-th data storage file does not meet the capacity splitting condition, for example, the current data capacity in the target i-th data storage file does not exceed the preset capacity upper limit, or the sum of the current data capacity in the target i-th data storage file and the data capacity of the data to be stored does not exceed the preset capacity upper limit.
[0085] According to the technical solution provided in the embodiment of the present application, a first storage path of the data to be stored is determined by obtaining a unique identifier and a timestamp in the data to be stored, and the first storage path includes at least a first-level directory, a second-level directory, and a target i-th data storage file. When it is determined that the time information of the determined target i-th data storage file does not meet the time splitting condition and the data capacity of the target i-th data storage file does not meet the capacity splitting condition, the data to be stored is stored in the target i-th data storage file, thereby realizing hierarchical storage of data. In addition, in addition to the stored data, the data description is also additionally saved in the data storage file, so that the stored data has high portability, thereby realizing efficient storage of the data to be stored under controllable storage costs.
[0086] In an embodiment of the present application, the first-level directory can be located in the first layer of the data storage system, and the second-level directory is located in the second layer of the data storage system. Among them, the first layer includes a first directory structure description file and N1 first-level directories. The first directory structure description file is used to record the metadata of all the first-level directories in the first layer, and each first-level directory corresponds to a different data unique identifier. The second layer includes at least one second-level directory group, each second-level directory group corresponds to a first-level directory, and each second-level directory group includes a second directory structure description file and N2 second-level directories. The second directory structure description file is used to record the metadata of all the second-level directories in the second layer. Each second-level directory has the same data unique identifier as its corresponding first-level directory. Different second-level directories correspond to different time intervals. Any second-level directory includes N 31 a second data storage file, or includes N 32 There are three levels of directories, N1, N2, N 31 and N 32 All are positive integers.
[0087] Furthermore, in some embodiments, the first storage path also includes level 3 to level i directories. The level i directory is located in the i-th layer of the data storage system, and the i-th layer includes at least one level i directory group, each level i directory group corresponds to an i-1 level directory, and each level i directory group includes an i-th directory structure description file and N i There are i-level directories, and the i-th directory structure description file is used to record the metadata of all i-th data storage files and i-level directories in the i-th layer. Each i-level directory has the same data unique identifier as its corresponding i-1-level directory. Any i-level directory includes N i1 j-th data storage file, or including N i2 Three-level directories, N i1 and N i2 All are positive integers.
[0088] That is to say, the files in the data storage system can be stored in the storage path of the corresponding state according to the preset rules, and in order to improve the search efficiency of the specified data, the files themselves are processed by hierarchical storage when the data files are actually stored. At the same time, as mentioned above, the data description part of the technical solution provided by the embodiment of the present application is stored in the data storage file itself, so each level of data file can parse the data through the file itself. This structure enables any level of data file to obtain its data structure description in time when it is read.
[0089] Figure 2 Schematic diagram of the hierarchical structure of the hierarchical storage solution provided in the embodiment of the present application. Figure 2 As shown, the storage system needs to include a first layer and a second layer, wherein the first layer includes multiple first-level directories such as shard 1, shard 2, ..., shard n, and the second layer includes multiple second-level directory groups, each second-level directory group corresponds to a first-level directory and each second-level directory group includes multiple second-level directories such as time interval 1, time interval 2, ..., time interval m. The data to be stored is distributed to the second-level directory by the data storage system according to preset time intervals, such as every day, every hour, etc. In one example, if the preset time interval is set to every day, all data generated on the same day will be distributed to the same second-level directory, and the time interval formed by the time interval will be recorded in the second-level directory. The maximum number of first-level directories and second-level directories is determined according to the rules set by the user.
[0090] When new data needs to be stored, the unique identifier of the data can be first calculated by sharding, for example, using a hash algorithm to calculate a hash value, and then determining the first-level directory corresponding to the new data based on the hash value. Next, the timestamp of the new data is obtained, and the timestamp is used to match the second-level directories under the determined first-level directory. If the timestamp is within the time interval corresponding to a second-level directory, the new data can be written to the second-level directory.
[0091] Furthermore, for the files in the secondary directory, the data storage system can further distribute them according to their size, thereby creating more data storage levels such as the third layer and the fourth layer. The split capacity threshold of each level of directories in each layer can be pre-set, and the data is written to the data storage files in the directory of that level in priority in each layer. When the data capacity in the data storage file exceeds its split capacity threshold, a new subdirectory can be created for the directory of that level, and the data storage files stored therein can be split into multiple parts according to the set rules (such as the hash value of the file name), and the multiple parts obtained by splitting are stored in the newly created subdirectory. This is applied recursively until the preset maximum number of directory levels is reached.
[0092] like Figure 2As shown, if it is first determined based on the unique identification and timestamp information of the new data to be stored that the new data should be stored in the secondary directory "Time Interval 1", and the secondary directory "Time Interval 1" includes multiple data storage files such as file 1, file 2, ..., file x, then the new data can be directly saved as a new data storage file in the secondary directory "Time Interval 1".
[0093] In some cases, if the data storage system determines that the data storage file of the new data is saved in the secondary directory "Time Interval 1", the data capacity of the secondary directory will exceed its preset split capacity threshold. At this time, a subdirectory of the secondary directory "Time Interval 1" can be newly created, and a third-level directory group can be newly created in the third layer of the data storage system. The third-level directory group includes at least one third-level directory. File 1, file 2, ..., file x stored in the original secondary directory "Time Interval 1" are split according to the preset splitting rules and stored in the corresponding third-level directories, and the new data is also stored as a new data storage file in the corresponding third-level directory. The preset splitting rules can be specified by the user or determined by the system, and there is no restriction here.
[0094] On the other hand, if the second-level directory "Time Interval 1" itself is already a second-level directory that has been split into levels, that is, the second-level directory "Time Interval 1" includes multiple third-level directories such as third-level directory 1, third-level directory 2, ..., third-level directory y, then the third-level directory into which the new data should be stored can be determined according to the preset splitting rules. When there are more levels, the processing method is similar. Among them, n, m, x and y are all positive integers.
[0095] In an embodiment of the present application, at each layer of the data storage system, an independent directory structure description file is maintained to record the data storage situation of the layer. The directory structure description file contains metadata information of all files or directories at this level, such as file name, size, hash value, timestamp, etc. For split files, the directory structure description file should also contain the association information between them. Whenever a new file or directory is added to a certain level, the system will update the directory structure description file of that level and write the newly added metadata into the directory structure description file. At the same time, when a file or directory is deleted or modified, the system will also update the directory structure description file accordingly to maintain consistency with the actual storage situation.
[0096] Figure 3 1 is a flow chart of a method for determining a first storage path based on a unique identifier and a timestamp of data to be stored when i is equal to 2 provided in an embodiment of the present application. Figure 3 As shown, the method comprises the following steps:
[0097] In step S301, a target primary directory is determined in the first layer using the unique identifier of the data to be stored.
[0098] In step S302, a second-level directory under the target first-level directory whose time interval includes the timestamp of the data to be stored is determined as the target second-level directory.
[0099] The target secondary directory includes a second file structure description file and N 31 A second data storage file, a second file structure description file is used to describe N 31 A rule for creating a second data storage file.
[0100] In step S303, in the target secondary directory, a second data storage file that matches the unique identifier and timestamp of the data to be stored is determined as the target second data storage file.
[0101] In step S304, a storage path consisting of a target primary directory, a target secondary directory, and a target second data storage file is determined to be a first storage path.
[0102] In the embodiment of the present application, when i is equal to 2, when determining the first storage path based on the unique identifier and timestamp of the data to be stored, the unique identifier of the data to be stored can be used to first determine the target first-level directory in the first layer, and determine that under the target first-level directory, the second-level directory whose time interval includes the timestamp of the data to be stored is the target second-level directory. At this time, the determined target second-level directory does not include the third-level directory, but includes N 31 Furthermore, the target secondary directory also includes a second file structure description file, which is used to describe N 31 A rule for creating a second data storage file.
[0103] Next, the unique identifier and timestamp of the data to be stored can be used to match each second data storage file in the target secondary directory. For example, if the creation rule of the second data storage file records the use of the file name hash value to split the data storage file, the unique identifier of the data to be stored can be used to calculate the file name hash value, and the calculated file name hash value can be matched with the file name hash value of each second data storage file. If multiple second data storage files are matched, since each data storage file records the time information when it is created, for example, the creation time is saved as the file name suffix, the timestamp of the data to be stored can be further used to match the multiple matched second data storage files again to determine the unique target second data storage file.
[0104] Finally, the storage path consisting of the target primary directory, the target secondary directory, and the target second data storage file is determined as the first storage path, and the determination of the first storage path can be completed.
[0105] Figure 4 1 is a flow chart of a method for determining a first storage path based on a unique identifier and a timestamp of data to be stored when i is greater than 2 provided in an embodiment of the present application. Figure 4 As shown, the method comprises the following steps:
[0106] In step S401, a target primary directory is determined in the first layer using the unique identifier of the data to be stored.
[0107] In step S402, a second-level directory whose time interval includes the timestamp of the data to be stored under the target first-level directory is determined as the target second-level directory.
[0108] In step S403, in response to determining that the target secondary directory includes N 32 The number of third-level directories is N, based on the timestamp of the data to be stored or the remaining capacity of each third-level directory. 32 Determine the target third-level directory among the three-level directories.
[0109] In step S404, the previous step is iterated until the determined target i-level directory includes N i1 The i-th data storage file.
[0110] In step S405, in N i1 In the i-th data storage file, the i-th data storage file that matches the unique identifier and timestamp of the data to be stored is determined as the target i-th storage file.
[0111] In step S406, a storage path consisting of the target first-level directory to the target i-th level directory and the target i-th storage file is determined to be a first storage path.
[0112] In the embodiment of the present application, when i is greater than 2, when determining the first storage path based on the unique identifier and timestamp of the data to be stored, the target first-level directory can be determined in the first layer using the unique identifier of the data to be stored, and the second-level directory under the target first-level directory whose time interval includes the timestamp of the data to be stored is determined as the target second-level directory. At this time, the determined target second-level directory includes N 32 There are three levels of directories, the N 32 Each of the three-level directories can record time information, such as directory creation time. In one example, the third-level directory corresponding to the directory creation time closest to the timestamp of the data to be stored can be used as the target third-level directory. Alternatively, the remaining capacity of each third-level directory can be queried, and the third-level directory with the largest remaining capacity can be used as the target third-level directory. Alternatively, the time information can be combined with the remaining capacity, and the target third-level directory can be determined after comprehensive evaluation.
[0113] Next, step S403 may be iteratively executed until the determined target level i directory includes N i1 The i-th data is stored in the file. Then, in N i1 The i-th data storage file that matches the unique identifier and timestamp of the data to be stored is determined as the target i-th storage file. The specific implementation method of determining the target i-th storage file is as follows: Figure 3 The implementation methods for determining the target second storage file in the illustrated embodiments are substantially the same and will not be described in detail herein.
[0114] Finally, the storage path consisting of the target first-level directory to the target i-th level directory and the target i-th storage file is determined as the first storage path, and the determination of the first storage path can be completed.
[0115] Figure 5 is a flow chart of another data storage method provided by an embodiment of the present application. Figure 5 Steps S501 to S503 in the embodiment shown are Figure 1 Steps S101 to S103 in the illustrated embodiment are substantially the same and will not be described in detail herein. Figure 5 As shown, the method also includes the following steps:
[0116] In step S504, in response to the target i-th data storage file not satisfying the preset storage condition, a new i-th data storage file is created in the i-level directory.
[0117] In step S505, time information is added to the newly created i-th data storage file, data description is written into the newly created i-th data storage file, and data to be stored is stored in the newly created i-th data storage file.
[0118] In the embodiment of the present application, after determining the target i-th data storage file, if it is determined that the target i-th data storage file does not meet the preset storage conditions, for example, the capacity of the target i-th data storage file has reached the upper limit, or the file capacity of the target i-th data storage file will exceed the upper limit after the data to be stored is added, or the creation time of the target i-th data storage file has exceeded the preset update time, then a new i-th data storage file can be created in the i-level directory, and time information such as the creation time can be added to the newly created i-th data storage file, and the data description can be written into the newly created i-th data storage file, and finally the data to be stored can be stored in the newly created i-th data storage file. Among them, the data description can be obtained from the target i-th data storage file, or from other i-th data storage files, which is not limited here.
[0119] In an embodiment of the present application, the maximum number of layers of the data storage system, the maximum number of directory groups in each layer, the maximum number of directories in each directory group, the maximum number of data storage files in each directory, and the directory splitting conditions of each directory are updated according to a preset dynamic adjustment strategy. Among them, the preset dynamic adjustment strategy can be, in response to determining that the number of data storage files in the target directory is greater than a first quantity threshold, increasing one or more of the maximum number of directory groups at the layer where the target directory is located, the maximum number of directories in the directory group where the target directory is located, and the maximum number of data storage files in the target directory; or in response to determining that the number of data storage files in the target directory is less than a second quantity threshold, reducing one or more of the maximum number of directory groups at the layer where the target directory is located, the maximum number of directories in the directory group where the target directory is located, and the maximum number of data storage files in the target directory; or in response to determining that the remaining capacity of the storage space where the target directory is located is less than a preset capacity threshold, reducing one or more of the maximum number of directory groups at the layer where the target directory is located, the maximum number of directories in the directory group where the target directory is located, and the maximum number of data storage files in the target directory; or in response to determining that the performance of the storage device where the storage space where the target directory is located is less than a preset performance threshold, reducing one or more of the maximum number of directory groups at the layer where the target directory is located, the maximum number of directories in the directory group where the target directory is located, and the maximum number of data storage files in the target directory; or in response to receiving an updated target directory splitting rule, determining the directory splitting condition of the updated target directory based on the updated splitting rule.
[0120] That is to say, in order to meet the needs of different application scenarios, the data storage system also supports dynamic adjustment strategies. For example, when users find that there are too many or too few directories at a certain level, they can optimize data distribution by adjusting the maximum number of files / directories at each level. In addition, the data storage system can also dynamically adjust the threshold for file splitting and the maximum number of directory levels based on the performance and capacity of the storage device.
[0121] To ensure that the dynamic adjustment strategy does not affect data writing and query, it is necessary to confirm before dynamic adjustment that there are no ongoing write and file status conversion operations in the current secondary directory, and during the dynamic adjustment process, the acceptance of all write and file status conversion requests in the current secondary directory will be suspended. After the adjustment is completed, the write and status conversion requests that arrive during the adjustment period will be executed.
[0122] Figure 6 is a flow chart of another data storage method provided by an embodiment of the present application. Figure 6 Steps S601 to S602 in the embodiment shown are similar to Figure 1 Steps S101 to S102 in the illustrated embodiment are substantially the same and will not be described in detail herein. Figure 6 As shown, the method also includes the following steps:
[0123] In step S603, it is determined whether a preset dynamic adjustment strategy is currently being executed in the secondary directory in the first storage path.
[0124] In step S604, in response to determining that the preset dynamic adjustment strategy is currently being executed, the operation of waiting for the first time threshold and then querying again whether the preset dynamic adjustment strategy is currently being executed is repeatedly executed until it is determined that the preset dynamic adjustment strategy is not currently being executed.
[0125] In step S605 , in response to determining that the preset dynamic adjustment strategy is not currently being executed and the target i-th data storage file meets the preset storage condition, the data to be stored is stored in the target i-th data storage file.
[0126] The preset storage condition includes that the time information of the target i-th data storage file does not satisfy the time splitting condition, and the data capacity of the target i-th data storage file does not satisfy the capacity splitting condition.
[0127] In the embodiment of the present application, after determining the first storage path, it can be first determined whether the preset dynamic adjustment strategy is currently being executed in the secondary directory in the first storage path. If it is determined that the preset dynamic adjustment strategy is currently being executed in the secondary directory in the first storage path, the operation of waiting for the first time threshold and then querying again whether the preset dynamic adjustment strategy is currently being executed is repeatedly executed until it is determined that the preset dynamic adjustment strategy is not currently being executed.
[0128] Further, when it is determined that the preset dynamic adjustment strategy is not currently being executed and the target i-th data storage file meets the preset storage condition, the data to be stored may be stored in the target i-th data storage file to complete the storage of the data to be stored.
[0129] In an embodiment of the present application, data storage files include temporary state files, stable state files and dormant state files. Temporary state files and stable state files are stored in a first type of storage medium, and dormant state files are stored in a second type of storage medium; wherein the target i-th data storage file and the newly created i-th data storage file are both temporary state files.
[0130] In the embodiment of the present application, the storage of data files includes three states, namely, temporary state, stable state and dormant state. Among them, the temporary state file is used to store data written in real time. The temporary state data file is divided into multiple small files according to certain rules, for example: a new temporary state file is created every fixed time interval, or every fixed number of data items; or a new temporary state file is created every time the previous temporary state file reaches the specified storage capacity.
[0131] Each temporary file can use its creation timestamp as part of the file name, so that the data storage system can determine which temporary file the temporary data should be written to through the timestamp field in the file name. To ensure high throughput of data writing, temporary storage media usually use high-performance storage media.
[0132] Stable state files are generated based on the data in temporary state files according to certain rules. Although temporary state files can maintain a high write rate, they have no advantage in storage efficiency. Especially in the case of a lot of duplicate data such as Internet of Vehicles data, merging and compressing the same data can greatly save storage space and storage costs. For example, columnar storage and compression can greatly improve storage efficiency. In addition, the generation rules of temporary state files lead to an unlimited growth in the number of files. Regularly converting temporary state files to stable state can suppress this situation. As the main force during query, stable state files need to maintain good read and write performance. Therefore, stable state data files usually choose high-performance storage media.
[0133] A major feature of Internet of Vehicles data is that the older the data is, the less likely it is to be accessed. However, historical data still has value in preservation, and some analysis activities with a longer time span still need to be able to access historical data from a long time ago. However, the access frequency may be extremely low. Dormant files can save data with lower access frequency for a longer period of time at a lower cost and wake it up when needed. Therefore, dormant files can be stored on low-performance, low-cost storage media.
[0134] Figure 7 Schematic diagram of the storage method of different status files provided in the embodiment of the present application. Figure 7 As shown, temporary state files and stable state files can be stored in high-performance physical storage media, and dormant state files can be stored in low-performance physical storage media.
[0135] The entire storage space of the data storage system, including logically "one" storage space, may physically involve multiple storage media, so the data storage system also needs to maintain a mapping relationship from the logical storage space to the physical storage medium path.
[0136] For example, when storing the national standard 32960 data reported by the vehicle, a storage space named GB32960 is logically opened for this data, and physically, these storage spaces are mapped to two storage media, namely: a directory with a relative path of / data / GB32960 / on a high-performance solid state disk (SSD), and a directory with a relative path of / data / GB32960 / on a low-cost mechanical hard disk.
[0137] Figure 8 1 is a flow chart of a method for converting a target temporary state file into a target stable state file provided by an embodiment of the present application. Figure 8 As shown, the method comprises the following steps:
[0138] In step S801, in response to determining that the target temporary state file satisfies the first conversion condition, it is queried whether a preset dynamic adjustment strategy is currently being executed in the secondary directory where the target temporary state file is located.
[0139] Among them, the target temporary state file is any temporary state file, the target stable state file is the stable state file corresponding to the target temporary state file, and the first conversion condition includes that the difference between the current time and the creation time of the target temporary state file is greater than the preset merge time interval threshold, and / or the data capacity in the target temporary state file is greater than the preset capacity merge threshold.
[0140] In step S802, in response to determining that the preset dynamic adjustment strategy is currently being executed, the operation of waiting for a second time threshold and then querying again whether the preset dynamic adjustment strategy is currently being executed is repeatedly executed until it is determined that the preset dynamic adjustment strategy is not currently being executed.
[0141] In step S803, in response to determining that the preset dynamic adjustment strategy is not currently being executed, a new first stable state file is created in the lowest-level directory where the target temporary state file is located, and the data in the target temporary state file is written into the first stable state file to obtain the target stable state file.
[0142] As mentioned above, the stable state file is generated based on the data in the temporary state file according to certain rules. Therefore, the data storage system can periodically detect whether the first conversion condition for converting the temporary state file to the stable state file is currently met according to the preset rules. Among them, the first conversion condition can be, for example, that the difference between the current time and the creation time of the target temporary state file is greater than the preset merge time interval threshold, or that the data capacity in the target temporary state file is greater than the preset capacity merge threshold, or that both of the above conditions are met at the same time. Furthermore, the preset rule can be a preset detection period.
[0143] In one example, the data storage system can check at certain intervals whether the current conditions meet the first conversion condition for converting a temporary state file to a stable state file according to the rules set by the user. For example, the system checks once every 5 minutes. Assuming that the current temporary state file is limited to 10M, when the previous temporary state file reaches this capacity, the data writing of the current temporary state file is terminated and a new temporary state file is created. Then, when the conclusion that the current temporary state file has finished writing is obtained after checking every 5 minutes, the condition for the temporary state file to convert to a stable state is met.
[0144] In an embodiment of the present application, after determining that the target temporary state file satisfies the first conversion condition, it is necessary to first query whether the preset dynamic adjustment strategy is currently being executed in the secondary directory where the target temporary state file is located. If so, it is necessary to repeat the operation of waiting for the second time threshold and then query again whether the preset dynamic adjustment strategy is currently being executed until it is determined that the preset dynamic adjustment strategy is not currently being executed.
[0145] On the contrary, if it is determined that the operation of the preset dynamic adjustment policy is not currently being executed, a new first stable state file can be created in the lowest-level directory where the target temporary state file is located, and the data in the target temporary state file can be written into the first stable state file to obtain the target stable state file. In one example, if the target temporary state file is a data storage file stored in a fourth-level directory, when it is determined that the target temporary state file meets the first conversion condition and the operation of the preset dynamic adjustment policy is not being executed in the second-level directory where the fourth-level directory is located, a new first stable state file can be created in the fourth-level directory, and then the data in the target temporary state file can be format converted, data merged, etc., and finally the processed data can be written into the first stable state file, so that the conversion of the file from temporary state to stable state can be completed.
[0146] In the embodiment of the present application, in order to save storage space, if the amount of data in the target temporary file is not large, or the amount of data in the target temporary file is not large after conversion and merging, it can be merged and stored with the data in the stable state file obtained by the last conversion. Furthermore, since the data storage system of the embodiment of the present application can realize read-write separation, in order to avoid errors in simultaneous data queries during merge storage, the embodiment of the present application adopts the implementation method of creating a new stable state file instead of the traditional implementation method of appending data to the existing stable state file.
[0147] Fig. 9 1 is a flow chart of a method for writing data in a target temporary state file into a first stable state file provided by an embodiment of the present application. Fig. 9 As shown, the method comprises the following steps:
[0148] In step S901, the data description in the target temporary state file is written into the first stable state file.
[0149] In step S902, a second stable state file is obtained.
[0150] The second stable state file is the stable state file constructed last time in the lowest level directory where the target temporary state file is located.
[0151] In step S903, in response to determining that the capacity of the second stable state file reaches a preset split threshold, all data in the target temporary state file is written into the first stable state file.
[0152] In step S904, in response to determining that the capacity of the second stable state file does not reach the preset split threshold, all data of the second stable state file and all data in the target temporary state file are written into the first stable state file.
[0153] In the embodiment of the present application, when performing a data write operation on a first stable state file, a data description may be first obtained from a target temporary state file, and the data description may be written into the first stable state file. Next, a second stable state file may be obtained, and the second stable state file is a stable state file constructed last time in the lowest level directory where the target temporary state file is located. For example, in the example where the target temporary state file is a data storage file stored in a fourth-level directory, the second stable state file is a stable state file constructed last time in the fourth-level directory.
[0154] In an embodiment of the present application, after obtaining the second stable state file, it can be determined whether the capacity of the second stable state file reaches a preset split threshold. For example, it can be determined whether the current capacity in the second stable state file has reached the preset split threshold, or it can be determined whether the sum of the current capacity in the second stable file and the data capacity of the target temporary state file after data merging, conversion, etc. has reached the preset split threshold. If the preset split threshold is reached, it indicates that the data capacity in the second stable file is large enough, or the data capacity in the target temporary state file is large enough, so the target temporary state file can be converted into a stable state file separately. At this time, the data of the target temporary state file after data merging, conversion, etc. can be directly written into the first stable state file.
[0155] On the other hand, if it is determined that the capacity of the second stable state file has not reached the preset split threshold, and the utilization rate of storage resources has not been improved to reduce storage costs, the data in the first stable state file and the data in the target temporary state file after data merging, conversion, etc. can be written into the first stable state file together.
[0156] In the embodiment of the present application, after all the data in the target temporary state file, or all the data in the second stable state file and all the data in the target temporary state file are written into the first stable state file, it can also be determined whether the data writing operation is successfully executed. In the case where the capacity of the second stable state file reaches the preset splitting threshold, if it is determined that all the data in the target temporary state file has been successfully written into the first stable state file, the target temporary state file can be deleted. On the other hand, in the case where the capacity of the second stable state file does not reach the preset splitting threshold, if it is determined that all the data in the second stable state file and all the data in the target temporary state file have been successfully written into the first stable state file, the target temporary state file and the second stable state file can be deleted.
[0157] On the contrary, in either case, if it is determined that the data is not successfully written into the first stable state file, the first stable state file can be deleted, and the operation of converting the target temporary state file into the target stable state file can be re-executed. The re-execution can be performed immediately or after the next judgment cycle, which is not limited here.
[0158] Fig.10 1 is a flow chart of a method for converting a target stable state file into a target dormant state file provided by an embodiment of the present application. Fig.10 As shown, the method comprises the following steps:
[0159] In step S1001, query timestamp information and query time interval information of a target stable state file are obtained.
[0160] In step S1002, in response to determining that the target stable state file meets the second conversion condition based on the query timestamp information and the query time interval information, the target stable state file is copied to the dormant state file storage space to obtain the target dormant state file.
[0161] In step S1003, the data in the target stable state file is cleared, and the file storage path of the target dormant state file is written therein.
[0162] Among them, the second conversion condition includes that the time difference between the current time and the most recent query timestamp is greater than the third time threshold, and / or the query time interval is greater than the fourth time threshold, the target stable state file is any stable state file, and the target dormant state file is the dormant state file corresponding to the target stable state file.
[0163] In the embodiment of the present application, query timestamp information and query time interval information may be added to the stable state file, wherein each time the stable state file is queried, the query timestamp is updated, and the query time interval between the current query and the last query is recorded.
[0164] In the embodiment of the present application, when converting a stable state file to a dormant state file, the query timestamp information and query time interval information of the target stable state file may be first obtained, and then based on the query timestamp information and query time interval information, it is determined whether the target stable state file meets the second conversion condition. The second conversion condition may be that the time difference between the current time and the most recent query timestamp is greater than the third time threshold, or the query time interval is greater than the fourth time threshold, or the time difference between the current time and the most recent query timestamp is greater than the third time threshold and the query time interval is greater than the fourth time threshold.
[0165] In the embodiment of the present application, when it is determined that the target stable state file meets the second conversion condition, the target stable state file can be directly copied to the dormant state file storage space to obtain the target dormant state file. Fig.11 Schematic diagram of converting a target stable state file into a target dormant state file provided by an embodiment of the present application. Fig.11 As shown, the target stable state file is directly copied from the first storage medium to the second storage medium, and a target dormant state file with the same name as the target stable state file is obtained. The first storage medium is a high-performance storage medium for storing temporary state files and stable state files, and the second storage medium is a low-performance and low-cost storage medium for storing dormant state files.
[0166] In an embodiment of the present application, after the target stable state file is copied to the dormant state file storage space to obtain the target dormant state file, the data in the target stable state file can be cleared and the file storage path of the target dormant state file can be written therein. Therefore, when the target dormant state file is queried, it is first necessary to query whether the file exists in the high-performance stable state file storage medium, and then obtain it from the low-performance dormant state file storage medium after confirming its existence, thereby achieving a balance between storage cost and query speed.
[0167] That is to say, since the query of Internet of Vehicles data is usually focused on recent vehicle data, the data query method in the data storage system provided by the embodiment of the present application is mainly based on stable data files and supplemented by temporary data files. Combining the data of the two can provide users with fast data feedback.
[0168] Dormant data files refer to data files that have not been accessed for a long time or are accessed very infrequently. Storing them in high-performance storage media will waste performance, so they are transferred to low-cost storage media. Temporary data files usually exist for a short period of time. Therefore, there is no need to convert temporary files to dormant state. After a temporary file is converted to a stable state, it starts counting its state transitions to calculate when it triggers the transition to dormant state.
[0169] For example, a steady-state data file generated on 2023-05-01 will have a "query timestamp" and "query time interval" saved for it starting from the time it was generated. Each time the file is queried, the timestamp is updated and the time interval between the current query and the previous query is recorded.
[0170] Scan all data files at a fixed frequency. It should be noted that this type of scan only focuses on the "query timestamp" and "query time interval" information stored in the header / tail of the file, and does not need to read other data. It has high scanning efficiency and does not occupy too many resources. When the "query timestamp" and "query time interval" meet the user-defined rules, the corresponding file is transferred to the dormant physical storage path.
[0171] At the same time, the file information converted to dormant state will retain a file with the same name under the stable state file path for mapping to the dormant state file when querying. This is because the query efficiency of low-performance dormant state storage files is low, and directly specifying the file path can save scanning time.
[0172] Fig.12 1 is a flow chart of a method for waking up a target dormant state file provided in an embodiment of the present application. Fig.12 As shown, the method comprises the following steps:
[0173] In step S1201, in response to receiving a data query request, the storage space for storing temporary state files and stable state files is searched for data to be queried.
[0174] In step S1202, in response to obtaining a dormant storage path of the data to be queried in the target stable state file, the target dormant state file is acquired from the dormant state storage space using the dormant storage path.
[0175] In step S1203, the target dormant state file is copied to the target stable state file, and the query timestamp information and query time interval information for the target stable state file are reset to wake up the target dormant state file.
[0176] In the embodiment of the present application, when the data storage system receives a data query request, it can first query the data to be queried in the storage space for storing temporary state files and stable state files. If the dormant storage path of the data to be queried is obtained in the target stable state file, the dormant storage path can be used to obtain the target dormant state file from the dormant state storage space. Subsequently, the target dormant state file can be copied to the target stable state file, and the query timestamp information and query time interval information for the target stable state file can be reset to wake up the target dormant state file.
[0177] In other words, dormant data files do not mean they will no longer be queried. When a user submits a query request to the data query system, the data query system will first search for data in stable and temporary paths. In some cases, the query involves data files that have been converted to dormant state, and the dormant data files need to be awakened.
[0178] First, as mentioned above, when searching for a dormant file, the path of the dormant file in the dormant storage medium can be obtained from the stable file. Through this path, the dormant data file is moved to the high-performance storage medium storing the stable file, thus completing the awakening of the dormant data file.
[0179] At this time, the "query timestamp" and "query time interval" for the awakened dormant data file can also be reset to facilitate the subsequent entry of the stable state file into the dormant state again.
[0180] Fig.13 FIG. 1 is a flow chart of another data storage method provided in an embodiment of the present application. Fig.13 As shown, when performing data storage, the primary and secondary data paths of the target temporary file can be calculated first; then it is determined whether there is a hierarchical dynamic adjustment in progress under the current secondary path. If so, it is determined again after waiting for a period of time until the judgment result is no; next, the hierarchical path and file name of the target temporary file are calculated; then it is determined whether the conditions for creating a new temporary file are met. If not, the data is directly written in an appended manner. If so, a new temporary file is created, the data structure description is written in the temporary file, and the data is written in the temporary file in an appended manner, thereby completing the data storage and returning the storage result.
[0181] That is to say, in the data storage method provided by the embodiment of the present application, under the premise that the data storage system adopts a hierarchical mechanism, the primary directory and the secondary directory to which the data file is written are defined when the data container is created, so it is possible to directly calculate which temporary state file under which primary and secondary directories it should be written to according to the data situation. After obtaining the primary and secondary paths, query whether there is currently a hierarchical dynamic adjustment in progress. During the dynamic adjustment process, since the third-level and lower directories may change, which may affect the writing of files, it is necessary to avoid writing and dynamic adjustment at the same time. If there is a hierarchical dynamic adjustment in progress, wait for a period of time and query again until the dynamic adjustment is completed, and then continue to execute the subsequent process.
[0182] Next, based on the unique key and timestamp information carried by the data to be written, it is possible to locate which temporary file in which path the data is written to. When there are multiple temporary files with different timestamp suffixes, the target temporary file is specifically located based on the timestamp carried by the data.
[0183] When a temporary file has reached the conditions for creating a new file, such as the file size reaches a threshold, the file creation time reaches a specified time interval, etc., a new temporary file is created with the creation timestamp as the naming suffix. For the newly created temporary file, the data structure description content is written to the beginning of the data file, and the data is directly appended to the end of the file.
[0184] Fig.14 FIG. 1 is a flow chart of a method for converting a temporary state file into a stable state file provided by an embodiment of the present application. Fig.14As shown, before the conversion begins, it is possible to first determine whether the temporary state to stable state conditions are met. If not, wait for a period of time and then re-judge until the result is yes; then calculate and record the source and target file paths, where the source file is a temporary state file and the target file is a stable state file; next, determine whether there is currently a hierarchical dynamic adjustment in progress. If so, wait for a period of time and then re-judge until the result is no; then create a stable state file, read the data structure description and write it into the newly created stable state file, and determine whether the previous stable state file has reached the size limit. If so, read the temporary state file data and write it into the newly created stable state file; if not, read the data of the previous stable state file and write it together with the temporary state file data into the newly created stable state file; after writing, determine whether the conversion is successful. If successful, delete the temporary state file and the previous stable state file whose data is written into the newly created stable state file, complete the file state conversion and return the result; if it fails, delete the newly created stable state file and re-execute the file state conversion operation.
[0185] That is to say, when converting a temporary state file to a stable state file, you can first check whether the conversion conditions are met. In one example, you can check whether the current conditions meet the conversion conditions from the temporary state to the stable state at certain intervals according to the rules set by the user. For example, check once every 5 minutes. Assuming that the current temporary state file is limited to 10M, when the previous temporary state file reaches this size, end the data writing of the current temporary state file and create a new temporary state file. Then, when the check every 5 minutes concludes that the current temporary state file has ended writing, the conditions for the temporary state file to be converted to the stable state are met.
[0186] Next, create a conversion task and record the current state and target state. In one example, you can record all temporary file paths that need to be converted, the target stable file paths they are converted to, and the current state of the conversion. At this time, the conversion state value can be "to be started".
[0187] Then wait for the conversion to be executed. If there is a hierarchical dynamic adjustment in progress, the directory and file structure may be adjusted, so you need to wait for the dynamic adjustment to be completed before starting the conversion from temporary state to stable state. Wait for a while and query again until the dynamic adjustment is completed, and then start the conversion from temporary state to stable state.
[0188] Then you can start the conversion. Temporary files usually use row storage to ensure the efficiency of data writing. When converting it to a stable state, you can choose a column storage solution that saves more storage space, such as column storage. At the same time, column storage can also take into account the query efficiency when only a small number of fields need to be queried each time when there are many and complex fields in the Internet of Vehicles data. Read the data structure description from the temporary file (stored at the beginning of the temporary state file), read the data in sequence, convert it and store it in the target stable state file.
[0189] Finally, the conversion is completed or failed. After the conversion is successfully completed, the conversion status value can be set to success. At this time, all temporary files and previous stable files involved in the conversion can be deleted. If an exception occurs during the conversion, resulting in the failure of the conversion, the current conversion status recorded in the conversion task can be set to failure. At this time, the generated stable files can be deleted, the number of retries can be recorded, and an alarm can be issued when the number of retries reaches a certain threshold.
[0190] By adopting the technical solution of the embodiment of the present application, storage space can be saved as much as possible while ensuring data writing efficiency; it can quickly determine whether data exists in massive data and overwrite it to avoid data duplication, thereby realizing low-cost and rapid data update; data can be quickly located through hierarchical storage; data description and data are stored in the same path, and have good portability; the data storage directory hierarchy can be dynamically adjusted to speed up the query speed; dynamic conversion of storage file status can be realized, data storage costs can be reduced, and a balance between cost and query efficiency can be achieved; read-write separation can be realized, all operations guarantee granularity, and do not affect other operations, thereby increasing the robustness of the system.
[0191] All the above optional technical solutions can be arbitrarily combined to form optional embodiments of the present application, which will not be described one by one here.
[0192] The following is an embodiment of the device of the present application, which can be used to execute the embodiment of the method of the present application. For details not disclosed in the embodiment of the device of the present application, please refer to the embodiment of the method of the present application.
[0193] Fig.15 Schematic diagram of a data storage device provided in an embodiment of the present application. Fig.15 As shown, the device comprises:
[0194] The acquisition module 1501 is configured to acquire the data to be stored and determine the unique identifier and timestamp of the data to be stored.
[0195] Determination module 1502 is configured to determine a first storage path based on the unique identifier and timestamp of the data to be stored, the first storage path includes at least a first-level directory, a second-level directory, and a target i-th data storage file, the target i-th data storage file is a data storage file in the i-level directory, and the target i-th data storage file includes data description and time information, and i is a positive integer greater than 1.
[0196] The storage module 1503 is configured to store the data to be stored in the target i-th data storage file in response to determining that the target i-th data storage file meets the preset storage condition.
[0197] The preset storage condition includes that the time information of the target i-th data storage file does not satisfy the time splitting condition, and the data capacity of the target i-th data storage file does not satisfy the capacity splitting condition.
[0198] According to the technical solution provided in the embodiment of the present application, a first storage path of the data to be stored is determined by obtaining a unique identifier and a timestamp in the data to be stored, and the first storage path includes at least a first-level directory, a second-level directory, and a target i-th data storage file. When it is determined that the time information of the determined target i-th data storage file does not meet the time splitting condition and the data capacity of the target i-th data storage file does not meet the capacity splitting condition, the data to be stored is stored in the target i-th data storage file, thereby realizing hierarchical storage of data. In addition, in addition to the stored data, the data description is also additionally saved in the data storage file, so that the stored data has high portability, thereby realizing efficient storage of the data to be stored under controllable storage costs.
[0199] In the embodiment of the present application, the first-level directory is located in the first layer of the data storage system, and the second-level directory is located in the second layer of the data storage system; the first layer includes a first directory structure description file and N1 first-level directories, the first directory structure description file is used to record the metadata of all the first-level directories in the first layer, and each first-level directory corresponds to a different data unique identifier; the second layer includes at least one second-level directory group, each second-level directory group corresponds to a first-level directory, and each second-level directory group includes a second directory structure description file and N2 second-level directories, the second directory structure description file is used to record the metadata of all the second-level directories in the second layer, each second-level directory has the same data unique identifier as its corresponding first-level directory, different second-level directories correspond to different time intervals, and any second-level directory includes N 31 a second data storage file, or includes N 32 There are three levels of directories; among them, N1, N2, N 31 and N 32 All are positive integers.
[0200] In the embodiment of the present application, in response to i being equal to 2, determining the first storage path based on the unique identifier and timestamp of the data to be stored includes: using the unique identifier of the data to be stored to determine the target first-level directory in the first layer; determining that under the target first-level directory, the second-level directory whose time interval includes the timestamp of the data to be stored is the target second-level directory, and the target second-level directory includes a second file structure description file and N 31 A second data storage file, a second file structure description file is used to describe N 31A creation rule for a second data storage file; in the target secondary directory, determine that the second data storage file that matches the unique identifier and timestamp of the data to be stored is the target second data storage file; determine that the storage path composed of the target primary directory, the target secondary directory and the target second data storage file is the first storage path.
[0201] In the embodiment of the present application, in response to i being greater than 2, the first storage path further includes directories from level 3 to level i; the level i directory is located at the i-th layer of the data storage system, the i-th layer includes at least one level i directory group, each level i directory group corresponds to an i-1 level directory, and each level i directory group includes an i-th directory structure description file and N i i-level directories, the i-th directory structure description file is used to record the metadata of all i-th data storage files and i-level directories in the i-th layer; each i-level directory has the same data unique identifier as its corresponding i-1-level directory, and any i-level directory includes N i1 j-th data storage file, or including N i2 three-level directories; among them, N i1 and N i2 All are positive integers.
[0202] In the embodiment of the present application, in response to i being greater than 2, determining the first storage path based on the unique identifier and timestamp of the data to be stored includes: using the unique identifier of the data to be stored to determine the target first-level directory in the first layer; determining that the second-level directory under the target first-level directory, whose time interval includes the timestamp of the data to be stored, is the target second-level directory; in response to determining that the target second-level directory includes N 32 The number of third-level directories is N, based on the timestamp of the data to be stored or the remaining capacity of each third-level directory. 32 Determine the target third-level directory among the third-level directories; iterate the previous step until the determined target i-level directory includes N i1 The i-th data storage file; i1 In the i-th data storage file, the i-th data storage file that matches the unique identifier and timestamp of the data to be stored is determined as the target i-th storage file; and the storage path consisting of the target first-level directory to the target i-level directory and the target i-th storage file is determined as the first storage path.
[0203] In an embodiment of the present application, it also includes: in response to the target i-th data storage file not satisfying the preset storage condition, creating a new i-th data storage file in the i-level directory; adding time information to the newly created i-th data storage file, writing the data description into the newly created i-th data storage file, and storing the data to be stored in the newly created i-th data storage file.
[0204] In an embodiment of the present application, the maximum number of layers, the maximum number of directory groups in each layer, the maximum number of directories in each directory group, the maximum number of data storage files in each directory, and the directory splitting condition of each directory of the data storage system are updated according to a preset dynamic adjustment strategy; the preset dynamic adjustment strategy includes: in response to determining that the number of data storage files in the target directory is greater than a first quantity threshold, increasing one or more of the maximum number of directory groups of the layer where the target directory is located, the maximum number of directories in the directory group where the target directory is located, and the maximum number of data storage files in the target directory; or in response to determining that the number of data storage files in the target directory is less than a second quantity threshold, reducing the maximum number of directory groups of the layer where the target directory is located, the maximum number of directories in the directory group where the target directory is located, and the maximum number of data storage files in the target directory. One or more of the number of big data storage files; or in response to determining that the remaining capacity of the storage space where the target directory is located is less than a preset capacity threshold, reducing one or more of the maximum number of directory groups at the layer where the target directory is located, the maximum number of directories in the directory group where the target directory is located, and the maximum number of data storage files in the target directory; or in response to determining that the performance of the storage device where the storage space where the target directory is located is less than a preset performance threshold, reducing one or more of the maximum number of directory groups at the layer where the target directory is located, the maximum number of directories in the directory group where the target directory is located, and the maximum number of data storage files in the target directory; or in response to receiving an updated target directory splitting rule, determining the directory splitting condition of the updated target directory based on the updated splitting rule.
[0205] In an embodiment of the present application, before storing the data to be stored in each data storage file, it also includes: determining whether a preset dynamic adjustment strategy is currently being executed in the secondary directory in the first storage path; in response to determining that the preset dynamic adjustment strategy is currently being executed, repeatedly waiting for a first time threshold and then querying again whether the preset dynamic adjustment strategy is currently being executed until it is determined that the preset dynamic adjustment strategy is not currently being executed; in response to determining that the preset dynamic adjustment strategy is not currently being executed, storing the data to be stored in the corresponding data storage file.
[0206] In an embodiment of the present application, data storage files include temporary state files, stable state files and dormant state files. Temporary state files and stable state files are stored in a first type of storage medium, and dormant state files are stored in a second type of storage medium; wherein the target i-th data storage file and the newly created i-th data storage file are both temporary state files.
[0207] In an embodiment of the present application, the target temporary state file is converted into a target stable state file in the following manner: in response to determining that the target temporary state file satisfies a first conversion condition, querying whether a preset dynamic adjustment strategy is currently being executed in the secondary directory where the target temporary state file is located; wherein the first conversion condition includes that the difference between the current time and the creation time of the target temporary state file is greater than a preset merge time interval threshold, and / or the data capacity in the target temporary state file is greater than a preset capacity merge threshold; in response to determining that the preset dynamic adjustment strategy is currently being executed, repeatedly executing the operation of waiting for a second time threshold and then querying again whether the preset dynamic adjustment strategy is currently being executed until it is determined that the preset dynamic adjustment strategy is not currently being executed; in response to determining that the preset dynamic adjustment strategy is not currently being executed, creating a new first stable state file in the lowest-level directory where the target temporary state file is located, and writing the data in the target temporary state file into the first stable state file to obtain the target stable file.
[0208] In an embodiment of the present application, writing the data in the target temporary state file into the first stable state file includes: writing the data description in the target temporary state file into the first stable state file; obtaining a second stable state file, the second stable state file being the stable state file last constructed in the lowest-level directory where the target temporary state file is located; in response to determining that the capacity of the second stable state file reaches a preset split threshold, writing all the data in the target temporary state file into the first stable state file; in response to determining that the capacity of the second stable state file does not reach the preset split threshold, writing all the data in the second stable state file and all the data in the target temporary state file into the first stable state file.
[0209] In an embodiment of the present application, it also includes: determining whether the target data has been successfully written into the first stable state file, the target data including all the data in the target temporary state file, or all the data in the second stable state file and all the data in the target temporary state file; in response to determining that the target data has been successfully written into the first stable state file, deleting the target temporary state file, or deleting the target temporary state file and the second stable state file; in response to determining that the target data has not been successfully written into the first stable state file, deleting the first stable state file, and re-executing the operation of converting the target temporary state file into the target stable state file.
[0210] In an embodiment of the present application, a target stable state file is converted into a target dormant state file in the following manner: obtaining query timestamp information and query time interval information of the target stable state file; in response to determining that the target stable state file satisfies a second conversion condition based on the query timestamp information and the query time interval information, copying the target stable state file to a dormant state file storage space to obtain a target dormant state file; clearing the data in the target stable state file, and writing the file storage path of the target dormant state file therein; wherein the second conversion condition includes that the time difference between the current time and the most recent query timestamp is greater than a third time threshold, and / or the query time interval is greater than a fourth time threshold.
[0211] In an embodiment of the present application, the target dormant state file is awakened in the following manner: in response to receiving a data query request, querying the data to be queried in the storage space for storing temporary state files and stable state files; in response to querying the dormant storage path of the data to be queried in the target stable state file, using the dormant storage path to obtain the target dormant state file from the dormant state storage space; copying the target dormant state file to the target stable state file, and resetting the query timestamp information and query time interval information for the target stable state file, thereby realizing the awakening of the target dormant state file.
[0212] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.
[0213] Fig.16 Schematic diagram of an electronic device provided in an embodiment of the present application. Fig.16 As shown, the electronic device 16 of this embodiment includes: a processor 1601, a memory 1602, and a computer program 1603 stored in the memory 1602 and executable on the processor 1601. When the processor 1601 executes the computer program 1603, the steps in the above-mentioned method embodiments are implemented. Alternatively, when the processor 1601 executes the computer program 1603, the functions of the modules / units in the above-mentioned device embodiments are implemented.
[0214] The electronic device 16 may be a desktop computer, a notebook, a PDA, a cloud server, or other electronic device. The electronic device 16 may include, but is not limited to, a processor 1601 and a memory 1602. Those skilled in the art will appreciate that Fig.16 The electronic device 16 is merely an example and does not limit the electronic device 16 , and may include more or less components than those shown in the figure, or different components.
[0215] The processor 1601 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.
[0216] The memory 1602 may be an internal storage unit of the electronic device 16, for example, a hard disk or memory of the electronic device 16. The memory 1602 may also be an external storage device of the electronic device 16, for example, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card (FlashCard), etc. equipped on the electronic device 16. The memory 1602 may also include both an internal storage unit of the electronic device 16 and an external storage device. The memory 1602 is used to store computer programs and other programs and data required by the electronic device.
[0217] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units.
[0218] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. The computer program may include computer program code, which may be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc.
[0219] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.
Claims
1. A data storage method, characterized in that: include: Acquire the data to be stored, and determine the unique identifier and timestamp of the data to be stored; Determine a first storage path based on the unique identifier and timestamp of the data to be stored, wherein the first storage path includes at least a first-level directory, a second-level directory, and a target i-th data storage file, wherein the target i-th data storage file is a data storage file in the i-level directory, and the target i-th data storage file includes data description and time information, where i is a positive integer greater than 1; In response to determining that the target i-th data storage file satisfies a preset storage condition, storing the data to be stored in the target i-th data storage file; Wherein, the preset storage conditions include that the time information of the target i-th data storage file does not meet the time splitting condition, and the data capacity of the target i-th data storage file does not meet the capacity splitting condition; the first-level directory is located at the first layer of the data storage system, different first-level directories correspond to different data unique identifiers, and the i-level directory is located at the i-th layer of the data storage system; each layer of the data storage system is configured with a directory structure description file for recording metadata information of all directories or data storage files at this layer; In response to i being equal to 2, different level i directories correspond to different time intervals, and determining the first storage path based on the unique identifier and timestamp of the data to be stored includes: Determine the target primary directory in the first layer using the unique identifier of the data to be stored; Determine under the target primary directory, a secondary directory whose time interval includes the timestamp of the data to be stored as the target secondary directory; In the target secondary directory, determining a second data storage file that matches the unique identifier and timestamp of the data to be stored as the target second data storage file; A storage path consisting of the target primary directory, the target secondary directory, and the target second data storage file is determined to be the first storage path.
2. The method according to claim 1, characterized in that The first layer of the data storage system includes a first directory structure description file and N1 first-level directories. The second layer of the data storage system includes at least one second-level directory group. Each second-level directory group corresponds to a first-level directory. Each second-level directory group includes a second directory structure description file and N2 second-level directories. Each second-level directory has the same data unique identifier as its corresponding first-level directory. Any second-level directory includes N 31 a second data storage file, or includes N 32 A three-level directory; Among them, N1, N2, N 31 and N 32 All are positive integers.
3. The method according to claim 2, characterized in that The target secondary directory includes a second file structure description file and N 31 a second data storage file, wherein the second file structure description file is used to describe the N 31 A rule for creating a second data storage file.
4. The method according to claim 2, characterized in that: In response to i being greater than 2, the first storage path includes a first-level directory, a second-level directory, and third-level to i-level directories, and different second-level directories correspond to different time intervals; The i-th level directory is located in the i-th layer of the data storage system, the i-th layer includes at least one i-th level directory group, each i-th level directory group corresponds to an i-1-th level directory, and each i-th level directory group includes an i-th directory structure description file and N i an i-th directory structure description file for recording metadata of all i-th data storage files and i-th directory in the i-th layer; Each level i directory has the same data unique identifier as its corresponding level i-1 directory. i1 j-th data storage file, or including N i2 A three-level directory; The determining the first storage path based on the unique identifier and timestamp of the data to be stored includes: Determine a target primary directory in the first layer using the unique identifier of the data to be stored; Determine under the target primary directory, a secondary directory whose time interval includes the timestamp of the data to be stored as the target secondary directory; In response to determining that the target secondary directory includes N 32 three-level directories, based on the timestamp of the data to be stored or the remaining capacity of each three-level directory in the N 32 Determine the target third-level directory among the third-level directories; Iterate the previous step until the target i-level directory contains N i1 The i-th data storage file; In the N i1 Among the i-th data storage files, determine the i-th data storage file that matches the unique identifier and timestamp of the data to be stored as the target i-th storage file; Determine a storage path consisting of the target first-level directory to the target i-th level directory and the target i-th storage file as the first storage path; Among them, N i1 and N i2 All are positive integers.
5. The method according to claim 1, characterized in that The method further comprises: In response to the target i-th data storage file not satisfying a preset storage condition, creating a new i-th data storage file in the i-level directory; Time information is added to the newly created i-th data storage file, data description is written into the newly created i-th data storage file, and the data to be stored is stored in the newly created i-th data storage file.
6. The method according to any one of claims 1 to 5, characterized in that The maximum number of layers of the data storage system, the maximum number of directory groups in each layer, the maximum number of directories in each directory group, the maximum number of data storage files in each directory, and the directory splitting conditions of each directory are updated according to the preset dynamic adjustment strategy; The preset dynamic adjustment strategy includes: In response to determining that the number of data storage files in the target directory is greater than a first quantity threshold, increase one or more of the maximum number of directory groups at the layer where the target directory is located, the maximum number of directories in the directory group where the target directory is located, and the maximum number of data storage files in the target directory; or In response to determining that the number of data storage files in the target directory is less than a second number threshold, reducing one or more of the maximum number of directory groups at the layer where the target directory is located, the maximum number of directories in the directory group where the target directory is located, and the maximum number of data storage files in the target directory; or In response to determining that the remaining capacity of the storage space where the target directory is located is less than a preset capacity threshold, reducing one or more of the maximum number of directory groups at the layer where the target directory is located, the maximum number of directories in the directory group where the target directory is located, and the maximum number of data storage files in the target directory; or In response to determining that the performance of a storage device where the storage space where the target directory is located is less than a preset performance threshold, reducing one or more of the maximum number of directory groups at the layer where the target directory is located, the maximum number of directories in the directory group where the target directory is located, and the maximum number of data storage files in the target directory; or In response to receiving the updated target directory splitting rule, determining an updated directory splitting condition of the target directory based on the updated splitting rule.
7. The method according to claim 6, characterized in that Before storing the data to be stored in each data storage file, the method further includes: Determine whether a preset dynamic adjustment strategy is currently being executed in the secondary directory in the first storage path; In response to determining that the preset dynamic adjustment strategy is currently being executed, repeatedly executing the operation of waiting for the first time threshold and then querying again whether the preset dynamic adjustment strategy is currently being executed, until determining that the preset dynamic adjustment strategy is not currently being executed; In response to determining that the preset dynamic adjustment strategy is not currently being executed, the data to be stored is stored in a corresponding data storage file.
8. The method according to claim 6, characterized in that The data storage files include temporary state files, stable state files and dormant state files, the temporary state files and stable state files are stored in the first type of storage medium, and the dormant state files are stored in the second type of storage medium; The target i-th data storage file and the newly created i-th data storage file are both temporary files.
9. The method according to claim 8, characterized in that The target temporary state file is converted to the target stable state file in the following way: In response to determining that the target temporary state file satisfies the first conversion condition, querying whether a preset dynamic adjustment strategy is currently being executed in the secondary directory where the target temporary state file is located, the target temporary state file is any temporary state file, and the target stable state file is a stable state file corresponding to the target temporary state file; The first conversion condition includes that the difference between the current time and the creation time of the target temporary state file is greater than a preset merge time interval threshold, and / or the data capacity of the target temporary state file is greater than a preset capacity merge threshold; In response to determining that the preset dynamic adjustment strategy is currently being executed, repeatedly executing the operation of waiting for a second time threshold and then again querying whether the preset dynamic adjustment strategy is currently being executed, until determining that the preset dynamic adjustment strategy is not currently being executed; In response to determining that the operation of the preset dynamic adjustment policy is not currently being executed, a new first stable state file is created in the lowest-level directory where the target temporary state file is located, and the data in the target temporary state file is written into the first stable state file to obtain the target stable state file.
10. The method according to claim 9, characterized in that Writing the data in the target temporary state file into the first stable state file comprises: Writing the data description in the target temporary state file into the first stable state file; Obtain a second stable state file, where the second stable state file is a stable state file that was last constructed in the lowest level directory where the target temporary state file is located; In response to determining that the capacity of the second stable state file reaches a preset split threshold, writing all the data in the target temporary state file into the first stable state file; In response to determining that the capacity of the second stable state file does not reach a preset split threshold, all data of the second stable state file and all data in the target temporary state file are written into the first stable state file.
11. The method according to claim 10, characterized in that The method further comprises: Determine whether target data has been successfully written into the first stable state file, the target data including all data in the target temporary state file, or all data in the second stable state file and all data in the target temporary state file; In response to determining that the target data has been successfully written into the first stable state file, deleting the target temporary state file, or deleting the target temporary state file and the second stable state file; In response to determining that the target data is not successfully written into the first stable state file, the first stable state file is deleted, and the operation of converting the target temporary state file into the target stable state file is re-executed.
12. The method according to claim 8, characterized in that The target stable state file is converted to the target dormant state file in the following way: Obtain query timestamp information and query time interval information of the target stable state file; In response to determining that the target stable state file meets a second conversion condition based on the query timestamp information and the query time interval information, copying the target stable state file to a dormant state file storage space to obtain a target dormant state file; Clear the data in the target stable state file, and write the file storage path of the target dormant state file into it; Among them, the second conversion condition includes that the time difference between the current time and the most recent query timestamp is greater than a third time threshold, and / or the query time interval is greater than a fourth time threshold, the target stable state file is any stable state file, and the target dormant state file is a dormant state file corresponding to the target stable state file.
13. The method according to claim 12, characterized in that The target dormant state file is awakened in the following manner: In response to receiving the data query request, querying the data to be queried in the storage space for storing temporary state files and stable state files; In response to obtaining a dormant storage path of the data to be queried in the target stable state file, obtaining the target dormant state file from the dormant state storage space using the dormant storage path; The target dormant state file is copied to the target stable state file, and the query timestamp information and the query time interval information for the target stable state file are reset to realize the awakening of the target dormant state file.
14. A data storage device, characterized in that: include: An acquisition module is configured to acquire the data to be stored and determine a unique identifier and a timestamp of the data to be stored; a determination module configured to determine a first storage path based on a unique identifier and a timestamp of the data to be stored, wherein the first storage path includes at least a primary directory, a secondary directory, and a target i-th data storage file, wherein the target i-th data storage file is a data storage file in the i-th directory, and the target i-th data storage file includes a data description and time information, and i is a positive integer greater than 1; A storage module, configured to store the data to be stored in the target i-th data storage file in response to determining that the target i-th data storage file meets a preset storage condition; Wherein, the preset storage conditions include that the time information of the target i-th data storage file does not meet the time splitting condition, and the data capacity of the target i-th data storage file does not meet the capacity splitting condition; the first-level directory is located at the first layer of the data storage system, different first-level directories correspond to different data unique identifiers, and the i-level directory is located at the i-th layer of the data storage system; each layer of the data storage system is configured with a directory structure description file for recording metadata information of all directories or data storage files at this layer; In response to i being equal to 2, different level i directories correspond to different time intervals, and determining the first storage path based on the unique identifier and timestamp of the data to be stored includes: Determine the target primary directory in the first layer using the unique identifier of the data to be stored; Determine under the target primary directory, a secondary directory whose time interval includes the timestamp of the data to be stored as the target secondary directory; In the target secondary directory, determining a second data storage file that matches the unique identifier and timestamp of the data to be stored as the target second data storage file; A storage path consisting of the target primary directory, the target secondary directory, and the target second data storage file is determined to be the first storage path.
