A Multi-Level Classification Method for Large-Scale Underwater Acoustic Data Storage

Abstract

This invention discloses a multi-level classification method for large-scale underwater acoustic data storage, belonging to the field of computer storage and data processing technology. The main processing flow involves creating a series of weighted storage pools for the thermal and thermospheres at a monitoring center, caching logical identifiers of different weight levels. Each storage pool's bucket has a lifecycle; upon expiration, the data's value is reassessed, and data whose value decreases is migrated to the next level of storage pool. Finally, once the cache pool reaches a certain size, it is uniformly migrated to the next level. This invention effectively reduces hardware costs while improving the energy storage performance of energy storage devices to meet user needs when dealing with massive amounts of underwater acoustic data.

Description

Technical Field

[0001] This invention belongs to the field of computer storage and data processing technology, specifically relating to a method for storing large-scale underwater acoustic data in a multi-level classification system. Background Technology

[0002] With the rapid development of marine observation technology, especially the large-scale deployment of high-frequency sensors, continuous monitoring platforms, and underwater acoustic networks, the acquisition of underwater acoustic data has entered an era of high frequency, continuous operation, and networking, resulting in an explosive growth in data volume. Faced with such a massive data deluge, achieving efficient, economical, and intelligent storage management has become a key challenge in the field of marine information technology. Currently, to balance storage costs, access efficiency, and the need for long-term data preservation, the industry generally adopts a three-tiered data storage architecture based on fixed strategies: cold, warm, and hot. This architecture typically follows a passive, rule-driven management model: after data is generated, it is first placed in a high-performance hot cache pool to ensure low latency for real-time or near-real-time access; subsequently, the system scans the hot cache pool through scheduled tasks, assessing the popularity and making degradation decisions for data objects based on preset single-dimensional indicators (such as file duration or recent access frequency), migrating data considered "colder" to lower-cost warm or cold storage layers; finally, data entering the cold layer is often subjected to high-ratio compression or even archived to maximize storage space savings.

[0003] While this model alleviates the conflict between storage costs and access speed to some extent, its design philosophy stems from traditional IT storage management thinking. When dealing with data like ocean observation—data with strong domain characteristics, high complexity, and unevenly distributed value density—its inherent limitations become increasingly apparent. The method's operational logic is essentially static and reactive, relying on simple, predictable rules rather than an understanding of the data's intrinsic value and business context. Its core flaws are rooted in the following two aspects:

[0004] First, at the data value assessment level, the current model for determining data popularity is too mechanical and one-sided. Relying solely on superficial indicators such as the passage of time or access frequency fails to penetrate to the data's inherent business attributes, scientific value, and application potential. Ocean observation data contains a large subset of event-driven or periodic analysis data, such as occasional oceanic anomalous acoustic events or long-term baseline data required for specific scientific research missions. These may have extremely low access frequencies but possess extremely high scientific research or military application value. Current methods crudely equate such data with ordinary low-frequency data and downgrade it, essentially causing a "misjudgment of value." This forces data to be retrieved from slow-responding cold storage during subsequent critical mission analysis or emergency retrieval, severely slowing down analytical decision-making efficiency and contradicting the ocean observation's goal of real-time insight and rapid response.

[0005] Secondly, at the system resource scheduling level, its data migration mechanism lacks foresight and adaptive coordination capabilities. Migration operations are typically triggered by fixed schedules, resulting in a "flood-like" batch data transfer that does not consider the system's current real-time load, fluctuations in business I / O pressure, or the priority of the migration tasks themselves. When dealing with petabyte-scale massive amounts of data, this centralized, periodic migration behavior easily leads to intense resource competition with critical I / O operations of the front-end business, causing storage bandwidth and IOPS to be periodically squeezed. The direct consequence is that storage service performance exhibits regular and predictable fluctuations, which not only affects the stability of real-time data writing but also interferes with the user experience of online data querying and analysis, making the overall system performance appear rough and unreliable, and unable to meet the high requirements of modern marine observation applications for continuous and stable service quality.

[0006] In summary, the existing three-tier storage architecture and its fixed management strategies have revealed deep-seated contradictions in addressing the demands of intelligent and efficient marine big data storage management, namely, rigid models and insufficient system collaboration. Therefore, a new large-scale underwater acoustic data storage management strategy is urgently needed, capable of more accurately identifying data value and intelligently collaborating with business systems. Summary of the Invention

[0007] In view of this, the purpose of the present invention is to provide a multi-level classification method for large-scale underwater acoustic data storage to solve the above-mentioned technical problems.

[0008] To achieve the above objectives, the present invention provides the following technical solution:

[0009] A multi-level classification method for large-scale underwater acoustic data storage, the database comprising a thermosphere physical storage pool, a temperature sphere physical storage pool, and a cold sphere physical storage pool, including:

[0010] S1. Receive data request, calculate the initial weight of the current object based on the metadata of the data request, cache the actual data corresponding to the current object in the hot layer physical storage pool, and store the metadata of the current object in the metadata center.

[0011] S2. Based on the initial weights, the logical identifiers of the actual data are allocated to the designated storage buckets of the logical storage pools corresponding to the weight ranges, and a lifecycle timer is configured for each storage bucket. Among them, multiple logical storage pools are set up in both the thermal layer and the warm layer physical storage pools, and each logical storage pool corresponds to a set of weight ranges. Each logical storage pool includes multiple storage buckets, and the generation of each storage bucket is related to the allocation time of the logical identifier.

[0012] S3. When the lifecycle of the storage bucket expires, the initial weights corresponding to all logical identifiers in the storage bucket are re-evaluated according to the time decay coefficient to obtain new initial weights; and logical identifiers whose new initial weights are lower than the lower limit of the weight range of the current logical storage pool are migrated to adjacent logical storage pools with a lower lower limit of the weight range.

[0013] S4. Monitor the final logical storage pool of the thermal layer physical storage pool. When the number of accumulated logical identifiers or the corresponding total amount of data reaches the preset migration threshold, trigger a batch migration task to migrate all the corresponding actual data from the thermal layer physical storage pool to the temperature layer physical storage pool in batches.

[0014] S5. For the actual data in the temperature layer physical storage pool, repeat steps S2-S3 until the number of logical identifiers or the corresponding total amount of data accumulated in the final logical storage pool of the temperature layer physical storage pool reaches the preset second migration threshold, and then migrate to the cold layer physical storage pool for compression and archiving.

[0015] Furthermore, a multi-level classification method for large-scale underwater acoustic data storage also includes:

[0016] In step S1, if the current object is determined to be a small file based on the data request, the actual data corresponding to the current object is directly cached in the hot layer physical storage pool until the number of small files in the hot layer physical storage pool reaches a preset third threshold. Then, an offline merge request is initiated to merge the actual data of all small files, and the actual data of the merged large file is allocated to the merged file cache pool. The metadata of the merged large file is also stored in the metadata center. Each of the hot layer and the warm layer physical storage pools has a separate merged file cache pool and a separate metadata center. When a layer migration operation occurs, the metadata of all actual data that is migrated is migrated simultaneously.

[0017] The initial weight of the merged large file is obtained by calculating the weighted average of the initial weights of all small files in the metadata, and its logical identifier is assigned to the designated storage bucket of the logical storage pool corresponding to the weight range in step S2.

[0018] In step S4, when the migration of a large file begins, the actual data corresponding to the large file is migrated from the merged file cache pool of the thermal layer physical storage pool to the merged file cache pool of the thermal layer physical storage pool.

[0019] Furthermore, a multi-level classification method for large-scale underwater acoustic data storage also includes:

[0020] In step S2, when one or more logical identifiers of actual data are assigned to the logical storage pool of the corresponding weight range, a storage bucket with the corresponding timestamp is automatically generated, and logical identifiers with the same creation time as the corresponding timestamp are assigned to the corresponding storage bucket; wherein, the storage bucket is named according to its generation timestamp;

[0021] The logical identifiers inside the storage bucket are organized into a balanced binary tree based on their corresponding initial weights. When any node is detected in step S3 to be less than the lower limit of the weight range of the logical storage pool in which the current storage bucket is located, all child nodes under that node are migrated to the adjacent logical storage pool with a lower weight range.

[0022] Furthermore, in step S3, when a logical identifier is migrated, its migration completion time is used as the creation time, and a storage bucket with the corresponding timestamp is automatically generated in the logical storage pool after the migration. Logical identifiers with the same creation time and corresponding timestamp are assigned to the corresponding storage bucket.

[0023] Furthermore, a multi-level classification method for large-scale underwater acoustic data storage also includes: in steps 4 and 5, there are multiple final logical storage pools for the thermal and thermosphere physical storage pools, and only one is activated at a time;

[0024] When the number of logical identifiers or the corresponding total amount of data accumulated in the currently enabled final logical storage pool reaches a preset migration threshold, the final logical storage pool will no longer provide services, and a new and usable final logical storage pool will be enabled to receive logical identifiers that have migrated from the adjacent logical storage pool.

[0025] Once the logical identifier in the final logical storage pool that is no longer providing services has completed the layer migration task, the final logical storage pool will be remarked as available.

[0026] Furthermore, a multi-level classification method for large-scale underwater acoustic data storage also includes: in step S5, when any actual data is migrated to the cold layer physical storage pool for compression and archiving, its corresponding metadata information is deleted in the metadata center.

[0027] Furthermore, a multi-level classification method for large-scale underwater acoustic data storage also includes: when performing a search operation, reading the location information of the data to be searched through the metadata center, and searching from the corresponding logical storage pool, and the searched data will re-enter the thermal layer physical storage pool.

[0028] Furthermore, a multi-level classification method for large-scale underwater acoustic data storage also includes: when performing a search operation, if the search is for a small file, the location information of the corresponding large file is read through the metadata center, the corresponding large file is found in the corresponding logical storage pool, and the corresponding small file is found based on the offset reference information of the small file carried by the metadata of the large file.

[0029] Furthermore, the initial weight calculation method and weight re-estimation method in steps S1 and S3 are as follows:

[0030]

[0031] In the formula, Indicates the initial weights. These represent the corresponding values ​​for file size, access frequency, service type, and storage location, respectively. The corresponding numerical value representing the file creation time; where, and All values ​​are unitless values ​​after normalization. This represents the new initial weights after weight reassessment. This represents the time decay factor, which is related to the lifecycle of the storage bucket.

[0032] The beneficial effects of this invention are as follows:

[0033] 1. Value-driven refined management: Through a multi-dimensional dynamic weight model, the value of data is accurately characterized and adjusted in real time, ensuring that storage resources are allocated according to the true value of the data;

[0034] 2. Seamless business migration: The unique two-level architecture of "logical ID migration" and "physical data batch migration" decouples high-frequency value judgments from low-frequency data movement, completely eliminating the performance impact of large-scale data migration on front-end business.

[0035] 3. Significant improvement in small file storage performance: The "caching first, merging later, and lazy indexing" processing paradigm transforms random metadata access to massive small files into sequential access and memory index lookup for large files, fundamentally solving the metadata bottleneck problem; at the same time, it prevents too many small files from wasting a lot of storage space and causing a decrease in cluster stability.

[0036] Other advantages, objectives, and features of the invention will be set forth in the following description and will be apparent to those skilled in the art in some respects, or may be learned by practice of the invention. The objects and other advantages of the invention may be realized and obtained by means of the structures particularly pointed out in the written description and the accompanying drawings.

[0037] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0038] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0039] Figure 1 This is a flowchart of a method for storing large-scale underwater acoustic data with multi-level classification, as described in an embodiment of the present invention.

[0040] Figure 2 This is a schematic diagram of the storage cache distribution of a multi-level classification large-scale underwater acoustic data storage method in an embodiment of the present invention;

[0041] Figure 3 This is another storage cache distribution diagram of a multi-level classification method for large-scale underwater acoustic data storage in an embodiment of the present invention. Detailed Implementation

[0042] The preferred embodiments of the present invention will be described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.

[0043] like Figure 1 As shown, this invention proposes a multi-level classification method for large-scale underwater acoustic data storage, comprising a thermal layer physical storage pool, a temperature layer physical storage pool, and a cold layer physical storage pool, including:

[0044] S1. Receive data request, calculate the initial weight of the current object based on the metadata of the data request, cache the actual data corresponding to the current object in the hot layer physical storage pool, and store the metadata of the current object in the metadata center.

[0045] S2. Based on the initial weights, the logical identifiers of the actual data are allocated to the designated storage buckets of the logical storage pools corresponding to the weight ranges, and a lifecycle timer is configured for each storage bucket. Among them, multiple logical storage pools are set up in both the thermal layer and the warm layer physical storage pools, and each logical storage pool corresponds to a set of weight ranges. Each logical storage pool includes multiple storage buckets, and the generation of each storage bucket is related to the allocation time of the logical identifier.

[0046] S3. When the lifecycle of the storage bucket expires, the initial weights corresponding to all logical identifiers in the storage bucket are re-evaluated according to the time decay coefficient to obtain new initial weights; and logical identifiers whose new initial weights are lower than the lower limit of the weight range of the current logical storage pool are migrated to adjacent logical storage pools with a lower lower limit of the weight range.

[0047] S4. Monitor the final logical storage pool of the thermal layer physical storage pool. When the number of accumulated logical identifiers or the corresponding total amount of data reaches the preset migration threshold, trigger a batch migration task to migrate all the corresponding actual data from the thermal layer physical storage pool to the temperature layer physical storage pool in batches.

[0048] S5. For the actual data in the temperature layer physical storage pool, repeat steps S2-S3 until the number of logical identifiers or the corresponding total amount of data accumulated in the final logical storage pool of the temperature layer physical storage pool reaches the preset second migration threshold, and then migrate to the cold layer physical storage pool for compression and archiving.

[0049] Furthermore, in step S1, if the current object is determined to be a small file based on the data request, the actual data corresponding to the current object is directly cached in the hot layer physical storage pool until the number of small files in the hot layer physical storage pool reaches a preset third threshold. Then, an offline merge request is initiated to merge the actual data of all small files and the actual data of the merged large file is allocated to the merged file cache pool.

[0050] The preferred process for merging small files is as follows:

[0051] Let the set of small files to be merged be Each of the small files The size is The dynamic weights before the merger are ;

[0052] Merged data generation: The data in the merged large file MF is concatenated sequentially.

[0053]

[0054] In the formula, This indicates the sequential concatenation operation of bytes.

[0055] Index structure generation: Generates an index Index(MF), which is a set of triples:

[0056]

[0057] in ,and ;

[0058] In the formula, Indicates the index item. This represents the original unique logical identifier of the small file. This represents the starting byte offset of the i-th small file within the large file MF. The generation of the index structure facilitates subsequent searches for small files.

[0059] The metadata of large files being merged is stored in the metadata center; each of the hot and warm physical storage pools has a separate merged file cache pool and a separate metadata center. When a layer migration operation occurs, the metadata of all the actual data being migrated is migrated simultaneously.

[0060] The initial weight of the merged large file is obtained by weighted averaging the initial weights of all small files in the metadata. The preferred initial weight is:

[0061]

[0062] And assign its logical identifier to the designated storage bucket of the logical storage pool corresponding to the weight range in step S2;

[0063] In step S4, when the migration of a large file begins, the actual data corresponding to the large file is migrated from the merged file cache pool of the thermal layer physical storage pool to the merged file cache pool of the thermal layer physical storage pool.

[0064] Furthermore, in step S2, when one or more logical identifiers of actual data are assigned to the logical storage pool of the corresponding weight range, a storage bucket with the corresponding timestamp is automatically generated, and logical identifiers with the same creation time as the corresponding timestamp are assigned to the corresponding storage bucket; wherein, the storage bucket is named according to its generation timestamp;

[0065] The logical identifiers inside the storage bucket are organized into a balanced binary tree based on their corresponding initial weights. When any node is detected in step S3 to be less than the lower limit of the weight range of the logical storage pool in which the current storage bucket is located, all child nodes under that node are migrated to the adjacent logical storage pool with a lower weight range.

[0066] Furthermore, in step S3, when a logical identifier is migrated, its migration completion time is used as the creation time, and a storage bucket with the corresponding timestamp is automatically generated in the logical storage pool after the migration. Logical identifiers with the same creation time and corresponding timestamp are assigned to the corresponding storage bucket.

[0067] Furthermore, in steps 4 and 5, there are multiple final logical storage pools for the thermal and warm layer physical storage pools, and only one is activated at a time.

[0068] When the number of logical identifiers or the corresponding total amount of data accumulated in the currently enabled final logical storage pool reaches a preset migration threshold, the final logical storage pool will no longer provide services, and a new and usable final logical storage pool will be enabled to receive logical identifiers that have migrated from the adjacent logical storage pool.

[0069] Once the logical identifier in the final logical storage pool that is no longer providing services has completed the layer migration task, the final logical storage pool will be remarked as available.

[0070] Furthermore, in step S5, when any actual data is migrated to the cold layer physical storage pool for compression and archiving, its corresponding metadata information is deleted in the metadata center.

[0071] Furthermore, during a search operation, the location information of the data to be searched is read through the metadata center, and the search is performed from the corresponding logical storage pool. The searched data will then re-enter the hot layer physical storage pool.

[0072] Furthermore, when performing a search operation, if the search target is a small file, the location information of the corresponding large file is read from the metadata center, the corresponding large file is found in the corresponding logical storage pool, and the corresponding small file is found based on the offset reference information of the small file carried in the metadata of the large file.

[0073] For reading small files, a lazy index lookup algorithm is preferred, especially when reading merged small files. hour:

[0074] Let the merged large file to which it belongs be MF, and the in-memory index cache be a global dictionary cache:

[0075] Cache lookup: If Then the index Otherwise, load from persistent storage (i.e., the corresponding physical storage pool). To memory, and make , .

[0076] In the formula, Cache represents a global, in-memory dictionary (key-value pair) structure used to cache the loaded merged file index; This indicates that the set of all keys in the cache is returned, which is a list of merged file IDs of the cached indexes.

[0077] Offset positioning: Find triples in index Idx This includes determining the offset using offset reference information.

[0078] Data readout: from the physical location of MF Start by reading the offset. , length is Continuous byte data;

[0079] This process can be formalized as the function ReadSubFile:

[0080]

[0081] in, LoadIndex encapsulates the above caching logic;

[0082] In the formula, This represents a function that encapsulates the index loading logic. If it's already in the cache, simply return the cached index. Otherwise, read from persistent storage. Store it in the cache, then return it; Indicating in the index Find the item with key ID(st) and return its corresponding value. ; This indicates that the read operation will begin from the physical location specified by pointer Ptr(MF) (i.e., the starting address of the large file MF), with an offset of [value missing]. , length is Continuous byte data.

[0083] Furthermore, the initial weight calculation method and weight re-estimation method in steps S1 and S3 are as follows:

[0084]

[0085] In the formula, Indicates the initial weights. These represent the corresponding values ​​for file size, access frequency, service type, and storage location, respectively. The corresponding numerical value representing the file creation time; where, and All values ​​are unitless values ​​after normalization. This represents the new initial weights after weight reassessment. This represents the time decay factor, which is related to the lifecycle of the storage bucket.

[0086] To better assist those skilled in the art in understanding the present application, an overall embodiment is provided below to further illustrate the present invention.

[0087] This application primarily uses a custom weighting factor to stratify data. The technical challenges are threefold: first, providing different service levels to different users, with a focus on better data read performance for key users; second, after refining the hierarchical management, migrating data between storage pools (logical storage pools) at a certain level inevitably leads to a significant decrease in responsiveness; and third, the need for rapid processing and merging of small files within massive datasets, as a large number of small files wastes significant storage space and reduces cluster stability. The overall solution can be divided into three layers: hot, cold, and warm. The main process involves creating a series of weighted storage pools for the hot and warm layers in the monitoring center, caching file IDs of different weight levels. Each storage pool bucket has a lifecycle; upon expiration, the data's value is reassessed, and data whose value decreases is migrated to the next lower-level storage pool. Finally, once the cache pool reaches a certain size, all data is migrated to the next level.

[0088] In addition, this solution is divided into three layers—hot, warm, and cold—based on the physical media capabilities of the underlying object storage. It uses unified Keystone authentication to support heterogeneous object storage clusters—Swift and Ceph—and OpenResty for cross-cluster forwarding. The overall deployment architecture is as follows:

[0089] OpenResty serves as the unified entry point, intercepting all requests, calculating object weights, adding them to the header, and forwarding them to the monitoring center and physical cache pool. The physical cache pool can be composed of underlying Swif or Ceph clusters. The monitoring center includes a metadata center that manages objects and iterates object IDs within the logical storage pool. Keystone provides a unified account authentication service for the backend cluster. The physical cache pool caches the actual physical files, while the monitoring center creates logical storage pools, iterates over file IDs, and ultimately maps the virtual storage pools to the actual memory space of the nodes. The metadata center stores file-related metadata such as size, creation date, weight, name, and, for merged files, the merged file name and the offset of smaller files.

[0090] The specific rules and concepts involved are explained below:

[0091] Step 1: Caching files, hot layer weight calculation and migration. After a user's request enters OneMact, the weight is automatically calculated (as mentioned above). As shown, each file receives an initial weight (which is part of the file's metadata) before entering the cache pool, and is then forwarded to the monitoring center and physical cache pool. The hot-layer cache pool first caches the actual files; this cache pool consists of an Otten SWIF or Ceph object storage cluster. Files entering the monitoring center are placed in buckets within their respective weighted storage pools (logical storage pools: a storage pool is a group of logical storage pools with weight ranges, storing objects (i.e., target files) with specific weights). Buckets are named with timestamps and have lifecycle timers. When a bucket's lifecycle expires, its weight is proportionally reduced over time. Files with IDs lower than the logical storage pool's ID (logical identifier: the object's label; during migration in the storage pool, the actual migrated label is the object tag. It contains basic file metadata information, object-name, and weight) are moved forward; otherwise, they remain unchanged. If OpenResty determines the file is small based on the request headers, it caches it directly in the physical cache pool and notifies the monitoring center. Once the monitoring center detects that the number of small files in the cache pool has reached a certain scale, it initiates an offline merge request. The merged data consists of the small file's data and meta-data. The merged metadata is stored in the metadata center. The weight of the merged large file is the weighted average of the weights of all small files and is placed in a dedicated merged file cache pool. Then, the large file's ID is placed in the logical bucket of the corresponding weighted logical storage pool. Throughout the entire file ID iteration process, the actual file data remains in the high-speed cache pool; only the file ID is iterated. Because the file IDs in the bucket are generally ordered, low-weight file IDs can be quickly migrated to the next pool.

[0092] It's worth noting that the bucket creation and deletion rules in this solution are as follows: buckets for each specific business department are created based on time, and the bucket name is the creation timestamp. Files created at the corresponding timestamp fall into the corresponding bucket. When a bucket's lifecycle expires, the weight of all files within it is automatically reduced proportionally over time. The file IDs within the bucket are organized in a balanced binary tree based on their weights. When a node is detected to be smaller than the range of this pool, all child nodes under that node are migrated to the next pool. During the entire migration process, only the file IDs are migrated as a whole within the logical storage pool. The data within the bucket remains ordered, the migration iteration speed is fast, and the impact on system performance is minimal.

[0093] Step 2: Migrate to the next level. Based on the version-id, select the final cache pool for the hot layer (located in the last logical cache pool of the layer; after entering this cache pool, file IDs will not be migrated again. When the number of file IDs within it reaches a certain level, the actual files will begin migrating to the next level's physical cache pool). When a certain number is reached, the monitoring center initiates the migration process. This final cache pool stops service and begins migrating data from the hot layer's high-speed cache pool to the warm layer's low-speed cache pool, updating the file metadata. The physical media for the warm layer uses NVMe SSDs.

[0094] Simultaneously, once the number of files in the final cache pool reaches a certain threshold, service ceases, and a new pool is used. At this point, the pool version changes, and the pool with the previous weight selects an available final cache pool based on the final cache pool's version_id. After the flooding process is complete, the pool becomes available again. Only during flooding is the file data actually copied from the high-speed cache pool to the low-speed cache pool. A storage cache distribution diagram is shown below. Figure 2 and Figure 3 As shown.

[0095] It is worth noting that version identification (version-id) and atomic metadata updates help ensure state consistency and service availability during data migration.

[0096] Step 3: Archiving. If the data weight continues to decrease, after entering the final cache pool of the warm layer, a compression and archiving operation will be performed, and the corresponding metadata information will be deleted. The physical medium of the cold layer is HDD.

[0097] Step 4: Search. If searching for a related object, the object's location information will first be read from the metadata center. If it cannot be found in the hot layer, it will search in the metadata pool of the warm layer, and then re-enter the hot cache pool. If searching for a small file, the large file information will be found first in the metadata center, and then the location of the large file will be found according to the previous steps. The small file will then be found using the offset information.

[0098] Finally, it should be noted that the above preferred embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail through the above preferred embodiments, those skilled in the art should understand that various changes can be made to it in form and detail without departing from the scope defined by the claims of the present invention.

Claims

1. A method for large-scale underwater acoustic data storage with multi-level classification, comprising a thermal layer physical storage pool, a temperature layer physical storage pool, and a cold layer physical storage pool, characterized in that: S1. Receive data request, calculate the initial weight of the current object based on the metadata of the data request, cache the actual data corresponding to the current object in the hot layer physical storage pool, and store the metadata of the current object in the metadata center. S2. Based on the initial weight, the logical identifier of the actual data is allocated to a designated storage bucket in the logical storage pool corresponding to the weight range, and a lifecycle timer is configured for the storage bucket; wherein, multiple logical storage pools are set in both the thermal layer and the warm layer physical storage pool, and each logical storage pool corresponds to a set of weight ranges; and each logical storage pool includes multiple storage buckets, and the generation of each storage bucket is related to the allocation time of the logical identifier; S3. When the lifecycle of the storage bucket expires, the initial weights corresponding to all logical identifiers in the storage bucket are re-evaluated according to the time decay coefficient to obtain new initial weights; and logical identifiers whose new initial weights are lower than the lower limit of the weight range of the current logical storage pool are migrated to adjacent logical storage pools with a lower lower limit of the weight range. S4. Monitor the final logical storage pool of the thermal layer physical storage pool. When the number of accumulated logical identifiers or the corresponding total amount of data in it reaches the preset migration threshold, trigger a batch migration task to migrate all the corresponding actual data from the thermal layer physical storage pool to the thermal layer physical storage pool in batches. The final logical storage pool is the logical storage pool with the lowest lower limit of the weight range. S5. For the actual data in the temperature layer physical storage pool, repeat steps S2-S3 until the number of logical identifiers or the corresponding total amount of data accumulated in the final logical storage pool of the temperature layer physical storage pool reaches the preset second migration threshold, and then migrate to the cold layer physical storage pool for compression and archiving.

2. The method for large-scale underwater acoustic data storage with multi-level classification according to claim 1, characterized in that, Also includes: In step S1, if the current object is determined to be a small file based on the data request, the actual data corresponding to the current object is directly cached in the hot layer physical storage pool until the number of small files in the hot layer physical storage pool reaches a preset third threshold. Then, an offline merge request is initiated to merge the actual data of all small files, and the actual data of the merged large file is allocated to the merged file cache pool. The metadata of the merged large file is also stored in the metadata center. Each of the hot layer and the warm layer physical storage pools has a separate merged file cache pool and a separate metadata center. When a layer migration operation occurs, the metadata of all actual data that is migrated is migrated simultaneously. The initial weight of the merged large file is obtained by calculating the weighted average of the initial weights of all small files in the metadata, and its logical identifier is assigned to the designated storage bucket of the logical storage pool corresponding to the weight range in step S2. In step S4, when the migration of a large file begins, the actual data corresponding to the large file is migrated from the merged file cache pool of the thermal layer physical storage pool to the merged file cache pool of the thermal layer physical storage pool.

3. The method for large-scale underwater acoustic data storage with multi-level classification according to claim 1, characterized in that, Also includes: In step S2, when one or more logical identifiers of actual data are assigned to the logical storage pool of the corresponding weight range, a storage bucket with the corresponding timestamp is automatically generated, and logical identifiers with the same creation time as the corresponding timestamp are assigned to the corresponding storage bucket; wherein, the storage bucket is named according to its generation timestamp; The logical identifiers inside the storage bucket are organized into a balanced binary tree based on their corresponding initial weights. When any node is detected in step S3 to be less than the lower limit of the weight range of the logical storage pool in which the current storage bucket is located, all child nodes under that node are migrated to the adjacent logical storage pool with a lower weight range.

4. A method for storing large-scale underwater acoustic data in a multi-level classification system according to any one of claims 1 or 3, characterized in that, In step S3, when a logical identifier is migrated, its migration completion time is used as the creation time. A storage bucket with the corresponding timestamp is automatically generated in the logical storage pool after the migration. Logical identifiers with the same creation time and corresponding timestamp are assigned to the corresponding storage bucket.

5. The method for large-scale underwater acoustic data storage with multi-level classification according to claim 1, characterized in that, It also includes: in steps S4 and S5, there are multiple final logical storage pools for the thermal layer and the temperature layer physical storage pools, and only one is activated at a time; When the number of logical identifiers or the corresponding total amount of data accumulated in the currently enabled final logical storage pool reaches a preset migration threshold, the final logical storage pool will no longer provide services, and a new and usable final logical storage pool will be enabled to receive logical identifiers that have migrated from the adjacent logical storage pool. Once the logical identifier in the final logical storage pool that is no longer providing services has completed the layer migration task, the final logical storage pool will be remarked as available.

6. The method for large-scale underwater acoustic data storage with multi-level classification according to claim 1, characterized in that, Also includes: In step S5, when any actual data is migrated to the cold layer physical storage pool for compression and archiving, its corresponding metadata information is deleted in the metadata center.

7. The method for large-scale underwater acoustic data storage with multi-level classification according to claim 1, characterized in that, Also includes: When performing a lookup operation, the location information of the data to be searched is read through the metadata center, and the search is performed from the corresponding logical storage pool. The searched data will then re-enter the hot layer physical storage pool.

8. A method for storing large-scale underwater acoustic data in a multi-level classification system according to claim 2, characterized in that, Also includes: When performing a search operation, if the search target is a small file, the location information of the corresponding large file is read from the metadata center, the corresponding large file is found in the corresponding logical storage pool, and the corresponding small file is found based on the offset reference information of the small file carried in the metadata of the large file.

9. The method for large-scale underwater acoustic data storage with multi-level classification according to claim 1, characterized in that, The initial weight calculation method and weight re-estimation method in steps S1 and S3 are as follows: In the formula, Indicates the initial weights. These represent the corresponding values ​​for file size, access frequency, service type, and storage location, respectively. The corresponding numerical value representing the file creation time; where, and All values ​​are unitless values ​​after normalization. This represents the new initial weights after weight reassessment. This represents the time decay factor, which is related to the lifecycle of the storage bucket.

Images