Marine laboratory multi-mode data storage and efficient query method

Abstract

The invention provides a navigation laboratory multi-mode data storage and efficient query method, and relates to the technical field of data storage and application, and the method comprises the following steps: adopting mixed storage combining a database and a file system; realizing recording of the database by utilizing distributed transaction management; establishing a combined index for the structured data in the database, and constructing a content index for the unstructured file in the file system by adopting Lucene; introducing a Redis cache technology, and performing cache replacement on the data by setting an LFU elimination algorithm in combination with a data thermodynamic value and a first-in first-out rule; loading the data to a cache step by step based on a historical access log and a data priority rule, and realizing quick duplicate judgment of the memory data through a Bloom filter; and querying an intermediate result according to the cache, and shunting the intermediate result at the same time. According to the method, the multi-mode data storage efficiency is optimized through a hybrid storage architecture and a loop log mechanism, and the consistency of cross-medium data is guaranteed by means of distributed transaction management.

Description

Technical Field

[0001] This invention relates to the technical field of data storage and application, and more particularly to a method for multi-mode data storage and efficient querying in a maritime laboratory. Background Technology

[0002] With the rapid development of fields such as marine engineering, marine environmental monitoring, and ship navigation technology, the research scenarios in marine laboratories are becoming increasingly complex, involving a diverse range of data types. These include structured data such as ship navigation parameters (e.g., speed, heading, draft), marine environmental data (e.g., water temperature, salinity, ocean current speed), and experimental measurement data, as well as unstructured documents such as experimental reports, design drawings, observation images, and technical documents (e.g., images, Word documents, PDF files). This type of multimodal data is characterized by its massive volume, high generation frequency, and diverse access requirements: on the one hand, daily laboratory research requires frequent access to high-frequency data such as personnel information, equipment parameters, and basic configurations; on the other hand, scientific research analysis requires joint queries of multi-condition structured data and unstructured document content, while demanding that data storage possess reliability, security, and traceability.

[0003] However, existing data storage and retrieval solutions for marine laboratories have the following technical shortcomings, making it difficult to meet practical application needs: Traditional solutions often employ a single storage model (pure database or pure file system): When storing unstructured files in a pure database, storage overhead surges and file read / write efficiency is low; when storing structured data in a pure file system, efficient field retrieval and relational queries are impossible. Some solutions attempt hybrid storage combining databases and file systems, but fail to address transaction consistency issues across storage media—for example, if a database record is deleted, the associated unstructured file may not be deleted synchronously, leading to data redundancy; or if a file write fails and the database has already generated associated records, inconsistencies may arise. Furthermore, maritime laboratory data often involves distributed deployment scenarios (cooperative storage between shore-based servers and shipboard acquisition nodes). Existing solutions lack transaction management mechanisms for distributed environments, making them susceptible to data loss or synchronization anomalies due to network interruptions or node failures. Simultaneously, database log files often grow indefinitely due to a lack of reasonable storage strategies, consuming excessive storage resources, and unstructured file paths and filenames are often stored in plaintext, posing a risk of unauthorized access.

[0004] Meanwhile, existing caching mechanisms often employ simple algorithms without dynamically adjusting based on access frequency and priority differences. High-frequency accessed basic data lacks targeted caching optimization, while low-frequency data occupies cache resources for extended periods, resulting in low cache hit rates and delayed query responses. The lack of a tiered cache loading strategy leads to the indiscriminate loading of all data at system startup, consuming significant memory and slowing down startup. Furthermore, the absence of an effective deduplication mechanism for cached data easily leads to duplicate caching, further wasting memory resources. Additionally, the lack of dynamic expiration times for intermediate cached results means that frequently updated data is prone to inconsistencies in query results due to cache expiration delays, while low-frequency updated data incurs excessive overhead from repeated loading due to short expiration times. Summary of the Invention

[0005] In view of the technical problems mentioned in the background section, a method for multi-mode data storage and efficient querying in a nautical laboratory is provided.

[0006] The technical means employed in this invention are as follows: A method for multi-modal data storage and efficient querying in a nautical laboratory includes the following steps: Step 1: A hybrid storage system combining a database and a file system is adopted; the database is structured by combining log files and data files; the log files use a circular storage mechanism to store uploaded unstructured files in the file system; Step 2: Implement database records using distributed transaction management; the distributed transaction management adopts either two-phase commit or the TCC protocol. Step 3: Create a composite index for the structured data in the database, and use Lucene to build a content index for the unstructured files in the file system; Step 4: Introduce Redis caching technology; set up the LFU eviction algorithm combined with data heat value and first-in-first-out rules for cache replacement; the data heat value is dynamically adjusted according to the number of calls and time intervals; Step 5: Load data into the cache step by step based on historical access logs and data priority rules, and use a Bloom filter to quickly determine duplicates in memory data; Step 6: Query intermediate results from the cache and perform split processing on the intermediate results; the intermediate results include: structured data, missing data, and unstructured data; for structured data queries, cache is prioritized; for missing data, the database is retrieved using a composite index; for unstructured data, the Lucene index is used for quick location.

[0007] Furthermore, the priority rules include: data priority P0, data priority P1, and data priority P2; The data priority P0 includes: homepage data, user information, global configuration, and data with the highest access frequency; the data priority P1 includes: core business data, popular data, and data required for the day; the data priority P2 includes: potential demand data.

[0008] Furthermore, the Bloom filter creates an array of preset length, performs hash calculation on the data and takes the modulo to obtain the index, and uses the array index status to indicate whether the data has been stored in the cache. The length of the array is dynamically calculated based on memory.

[0009] Furthermore, the unstructured files include: images, WORD documents, and PDF files.

[0010] Furthermore, in step 4, the LFU algorithm is used to remove the least frequently used data from the queue. When new data arrives, it is compared with the existing data in the queue, and the most recent usage time and the data's heat value are recorded. When the set memory size is exceeded, an algorithm combining the first-in-first-out principle and data heat value is used to replace the data in the cache.

[0011] Furthermore, the first-in-first-out principle prioritizes replacing the data that was stored earlier.

[0012] Furthermore, the heat value of the data changes according to the number of times it is called and the time interval between multiple calls; if the number of calls increases and the call time increases, the time for the data to enter the cache is shortened; if the number of calls decreases and the call time decreases, the time for the data to enter the cache increases.

[0013] Furthermore, the Bloom filter enables fast deduplication of in-memory data by including the following steps: First, create an array of length M. Calculate the hash value of the characters using a hash function, and then perform a modulo operation on the hash value to obtain the data's index. Set the data's index to 1. When data enters the cache, first calculate the hash value and perform a modulo operation to obtain the index. Then, check if the corresponding position in the array is 1. If it is 1, the data already exists; otherwise, set the index to 1 and add the data to the cache.

[0014] Compared with the prior art, the present invention has the following advantages: This application optimizes the efficiency of multi-mode data storage through a hybrid storage architecture and a circular log mechanism, ensures cross-media data consistency through distributed transaction management, and achieves efficient combined queries of structured and unstructured data by combining composite indexes and Lucene indexes. Through Redis caching, LFU algorithm with dynamic heat value adaptation, and hierarchical loading strategy, it improves cache hit rate and query response speed, and Bloom filter effectively avoids cache redundancy. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 This is a schematic diagram of the overall process of the present invention. Detailed Implementation

[0017] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0018] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0019] like Figure 1 As shown, this invention provides a multi-mode data storage and efficient query method for maritime laboratories, applicable to the integrated storage and efficient retrieval of structured and unstructured files in maritime laboratories, including the following steps: Step 1: A hybrid storage system combining database and file system is adopted. The database stores structured data through a combination of log files and data files. The log files employ a circular storage mechanism, storing uploaded unstructured files in the file system. For structured data such as ship navigation parameters and marine environmental data, structured storage is achieved through a combination of database log files and data files. The log files only support append operations, recording the content and time sequence of all data additions, deletions, and modifications, and the circular storage mechanism ensures fixed log space usage and stable database performance. For unstructured files such as images, Word documents, and PDF files, they are stored in the file system, while the file path and filename are recorded in the corresponding table field of the database, realizing the association mapping between database records and file system files.

[0020] Step 2: Implement database record management using distributed transaction management; the distributed transaction management adopts either two-phase commit or TCC protocol. This application uses an open-source Java distributed transaction management framework, implementing distributed transaction control through a two-phase commit protocol or TCC protocol: when performing data addition, modification, or deletion operations, database operations and file system operations are included in the same distributed transaction; for example, when deleting an experimental data record in the database, the transaction framework will synchronously trigger the deletion operation of the associated unstructured file in the file system, ensuring the atomicity and consistency of transaction execution and avoiding data redundancy or inconsistency issues.

[0021] Step 3: Create composite indexes for the structured data in the database and build content indexes for the unstructured files in the file system using Lucene. For the structured data in the database, create composite indexes based on common query scenarios in the maritime laboratory (such as voyage number + parameter type, time + experimental project, etc.) to improve the efficiency of multi-condition joint retrieval.

[0022] For unstructured files in the file system, Lucene technology is used to parse the file content and build a content index to generate a corresponding index file. Through this two-layer index system, the combined query function of database field values ​​and file content keywords is realized, supporting the synchronous retrieval of related results from multiple dimensions such as structured data and unstructured files.

[0023] Step 4: Introduce Redis caching technology; set up an LFU (Least Frequently Used) eviction algorithm combined with data heat index and FIFO (First-In, First-Out) rules for cache replacement; the data heat index is dynamically adjusted based on the number of calls and time intervals. The system records the most recent usage time and number of calls of cached data in real time to calculate the data heat index. The heat index increases with the number of calls and the shortening of the call interval, and decreases with the number of calls and the lengthening of the interval; when the cache memory usage reaches a set threshold, the data with the lowest usage frequency is evicted first. If there are data with the same frequency, the FIFO rule (prioritizing the replacement of the earliest stored data) is used to avoid high-heat-index data being mistakenly evicted.

[0024] The intermediate results are cached with dynamic expiration times. The expiration time is shortened for frequently updated data (such as real-time navigation parameters) and extended for infrequently updated data (such as laboratory basic configurations) to ensure consistency between cached data and source data. At the same time, frequently accessed basic data that does not change much, such as personnel information and laboratory configurations, are set as permanent caches and updated regularly.

[0025] Step 5: Load data into the cache step by step based on historical access logs and data priority rules, and use a Bloom filter to quickly determine duplicate data in memory. In this application, the priority rules include: data priority P0, data priority P1, and data priority P2; The data priority P0 includes: homepage data, user information, global configuration, and data with the highest access frequency; the data priority P1 includes: core business data, popular data, and data required for the day; the data priority P2 includes: potential demand data.

[0026] Preferably, a Bloom filter is used to quickly identify duplicate data in memory. The array length M is dynamically calculated based on available memory, and a binary array of length M is created. The data to be loaded is hashed using a hash function, and the array index is obtained by taking the modulo of the hash value and set to 1. When new data enters the cache, the above hash modulo process is repeated. If the corresponding index is 1, it is determined that the data already exists, avoiding duplicate caching; if it is 0, it is set to 1 and added to the cache, saving memory space.

[0027] Step 6: Query intermediate results from the cache and perform split processing on the intermediate results; the intermediate results include: structured data, missing data, and unstructured data; for structured data queries, cache is prioritized; for missing data, the database is retrieved using a composite index; for unstructured data, the Lucene index is used for quick location.

[0028] This can be understood as follows: if the query retrieves structured data and the cache is hit, the result is returned directly from the cache, reducing database pressure. If the query retrieves structured data but misses the cache, an efficient retrieval is performed using a database composite index, and the retrieval result is returned to the user while simultaneously written to the cache for subsequent queries. If the query retrieves unstructured files, the target file in the file system is quickly located using Lucene content indexes, the result is returned after the retrieval is completed, and the file association information is cached in Redis; by reusing intermediate cached results, consistency of results across multiple queries is ensured, reducing redundant calculations and I / O overhead.

[0029] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0030] In the above embodiments of the present invention, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0031] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For instance, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0032] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0033] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0034] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0035] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for multi-mode data storage and efficient querying in a maritime laboratory, characterized in that, Includes the following steps: Step 1: A hybrid storage system combining a database and a file system is adopted; the database is structured by combining log files and data files; the log files use a circular storage mechanism to store uploaded unstructured files in the file system; Step 2: Implement database records using distributed transaction management; the distributed transaction management adopts either two-phase commit or the TCC protocol. Step 3: Create a composite index for the structured data in the database, and use Lucene to build a content index for the unstructured files in the file system; Step 4: Introduce Redis caching technology, and use the LFU eviction algorithm combined with data heat value and first-in-first-out rules to cache and replace data; the data heat value is dynamically adjusted according to the number of calls and time intervals. Step 5: Load data into the cache step by step based on historical access logs and data priority rules, and use a Bloom filter to quickly determine duplicates in memory data; Step 6: Query intermediate results based on the cache, and simultaneously perform splitting processing on the intermediate results; the intermediate results include: structured data, missing data, and unstructured data; Structured data queries are cached first, data that is not found is retrieved from the database using a composite index, and unstructured data is quickly located using a Lucene index.

2. The method for multi-mode data storage and efficient querying in a nautical laboratory according to claim 1, characterized in that, The priority rules include: data priority P0, data priority P1, and data priority P2; The data priority P0 includes: homepage data, user information, global configuration, and data with the highest access frequency; the data priority P1 includes: core business data, popular data, and data required for the day; the data priority P2 includes: potential demand data.

3. The method for multi-mode data storage and efficient querying in a nautical laboratory according to claim 1, characterized in that, The Bloom filter creates an array of preset length, performs hash calculations on the data, and takes the modulo to obtain the index. The array index status indicates whether the data has been stored in the cache. The length of the array is dynamically calculated based on memory.

4. The method for multi-mode data storage and efficient querying in a nautical laboratory according to claim 1, characterized in that, The unstructured files include: images, Word documents, and PDF files.

5. The method for multi-mode data storage and efficient querying in a nautical laboratory according to claim 1, characterized in that, In step 4, the LFU algorithm is used to remove the least frequently used data from the queue. When new data arrives, it is compared with the existing data in the queue, and the most recent usage time and the data's heat value are recorded. When the set memory size is exceeded, an algorithm combining the first-in-first-out principle and data heat value is used to replace the data in the cache.

6. The method for multi-mode data storage and efficient querying in a nautical laboratory according to claim 5, characterized in that, The first-in, first-out (FIFO) principle prioritizes replacing the data that was stored earlier.

7. The method for multi-mode data storage and efficient querying in a nautical laboratory according to claim 5, characterized in that, The heat value of the data changes based on the number of times it is called and the time interval between multiple calls; if the number of calls increases and the call time increases, the time for the data to enter the cache is shortened; if the number of calls decreases and the call time decreases, the time for the data to enter the cache increases.

8. The method for multi-mode data storage and efficient querying in a nautical laboratory according to claim 1, characterized in that, The Bloom filter enables fast deduplication of in-memory data by including the following steps: First, create an array of length M. Calculate the hash value of the characters using a hash function, and then perform a modulo operation on the hash value to obtain the data's index. Set the data's index to 1. When data enters the cache, first calculate the hash value and perform a modulo operation to obtain the index. Then, check if the corresponding position in the array is 1. If it is 1, the data already exists; otherwise, set the index to 1 and add the data to the cache.

Images