Key-value-based distributed data management method

Abstract

The present invention relates to a key-value-based distributed data management method. For any test data, based on the data type of the test data, a data hash table is configured in a terminal device to store the key-value pairs of the test data using the configured data hash table, wherein the data name of the test data is used as a key KEY, and based on the hash value after hashing the key KEY, the bucket position where the key-value pair of the test data is stored in the data hash table is determined; based on the size of the test data, the terminal device configures the test data to be stored in the terminal device or a distributed file system, and uses the storage location information after storage as a Value value, and stores the Value value in the bucket of the data hash table determined above. The present invention can effectively realize the management of heterogeneous data generated by a ship virtual test system, and improve the efficiency, scalability and reliability of data management.

Description

Technical Field

[0001] The present invention relates to a data management method and system, and in particular to a key-value-based distributed data management method. Background Art

[0002] The shipbuilding industry is a modern comprehensive industry and a strategic industry combining military and civilian sectors. It is of great significance to improving the country's comprehensive strength, accelerating the pace of marine development, and safeguarding national marine rights and interests. However, due to the high cost of testing and the changeable marine environment, ship research faces the problem of high costs.

[0003] To solve the above problems, China Numerical Pool has launched a ship virtual test system, which can effectively realize the automation and intelligence of virtual tests and meet the precision requirements of ship industry applications. However, when conducting virtual tests, a large amount of heterogeneous data will be generated, such as relational data, files, texts, images, videos, and animations. The storage and management of these data is a challenging problem for distributed systems.

[0004] Theoretically, both relational and non-relational databases can be used to manage the data generated by the China Numerical Water Tank Virtual Test System. However, due to the large number of data types and large data scale generated in ship virtual tests, relational databases that store data in tabular form are not suitable for storing such large unstructured data. In contrast, non-relational databases have greater advantages in storing and processing large unstructured data.

[0005] Currently, key-value databases have become an ideal choice due to their simplicity, scalability, and flexibility. However, designing a key-value database that can handle multiple data types and support distributed deployment with high availability and consistency remains challenging.

[0006] Many existing key-value databases, such as Redis, Cassandra, and Riak, have been developed to handle large amounts of data and provide high availability and fault tolerance. However, these databases are designed for homogeneous data types (such as strings, integers, and lists), and they do not support data types such as files, images, and videos. In order to process heterogeneous data types, key-value databases such as MongoDB and Couchbase have been developed. Although these databases support multiple data types such as relational data, files, text, and images, they are not designed for distributed deployment with multi-copy consistency and high availability, and do not support ACID constraints.

[0007] From the above description, it can be seen that when storing virtual test data in the ship virtual test system, neither the relational database nor the non-relational database that is widely used on the market can meet the unified storage of heterogeneous data. This is also the main technical problem that needs to be solved urgently in the existing distributed ship virtual test data management technology. Summary of the invention

[0008] The purpose of the present invention is to overcome the deficiencies in the prior art and provide a key-value-based distributed data management method, which can effectively manage heterogeneous data generated by a ship virtual test system and improve the efficiency, scalability and reliability of data management.

[0009] According to the technical solution provided by the present invention, a key-value-based distributed data management method is used to manage test data generated by a ship virtual test system, wherein the distributed data management method includes:

[0010] Providing a terminal device for managing the test data, and configuring a distributed file system adapted to connect to the terminal device;

[0011] For any test data, based on the data type of the test data, a data hash table corresponding to the data type is configured in the terminal device, so as to store the key-value pairs of the test data using the configured data hash table, wherein:

[0012] Using the data name of the test data as the key KEY, and based on the hash value after performing a hash operation on the key KEY, determining the bucket position where the key-value pair of the test data is stored in the data hash table;

[0013] Based on the size of the test data, the terminal device stores the test data configuration in the terminal device or a distributed file system, uses the storage location information after storage as the Value value, and stores the Value value in the bucket of the data hash table determined above.

[0014] Configure the data threshold t in the terminal device, where:

[0015] When the size of the test data is not greater than the data threshold t, the test data configuration is stored in the terminal device, and the Value value is the disk block in the terminal device + the offset address in the block;

[0016] When the size of the test data is greater than the data threshold t, the test data configuration is stored in the distributed file system, and the Value value is the file name or path information in the distributed file system.

[0017] For each user in the ship virtual test system, the data hash table corresponding to all test data under the user is configured as a hash table bucket, where:

[0018] The hash table bucket is in the terminal device, and the hash table bucket of each user is stored in a disk of the terminal device at a location sufficient to be read into the memory at one time.

[0019] Based on the load factor of each data hash table, the terminal device adjusts the size of the corresponding data hash table, where:

[0020] For any data hash table, when the load factor of the data hash table is not less than the load factor threshold, the terminal device expands the capacity of the data hash table;

[0021] When the load factor of the data hash table is too small, the terminal device reduces the capacity of the data hash table.

[0022] The data hash table is expanded based on a progressive rehash method, wherein the expansion method includes:

[0023] Create a new data hash table that is consistent with the current data hash table, apply for memory space for the new data hash table, and set the capacity expansion flag to 1, wherein the capacity of the new data hash table is greater than the capacity of the current data hash table;

[0024] Sequentially migrate some key-value pairs in the current data hash table to the new data hash table created above;

[0025] The new data hash table is used as the current data hash table, and the expansion flag is set to 0;

[0026] Migrate the remaining key-value pairs in the data hash table before expansion to the current data hash table until all key-value pairs are migrated to the current data hash table.

[0027] In the terminal device, a time wheel is maintained for each data hash table to clear key-value pairs that have not been used for a long time.

[0028] For test data of the same data type, when the hash values ​​of the test data are the same, the key-value pairs of all the test data are stored in the same bucket of the data hash table and stored through a linked list.

[0029] For any key-value pair to be queried, the query includes:

[0030] Calculate the hash value of the key KEY in the key-value pair to be queried, and determine the data hash table corresponding to the key-value pair to be queried and the bucket corresponding to the data hash table based on the calculated hash value;

[0031] In the determined bucket, traverse the linked list to determine the Value value that matches the key-value pair to be queried;

[0032] Based on the determined Value, test data is read.

[0033] For heterogeneous multimodal data of the same object in the ship virtual test system, the name of the current object is used as the key KEY, and a heterogeneous association hash table of the current object is configured, in which the test data of each modality contains the name of the current object;

[0034] For any test data in the heterogeneous multimodal data, a storage state is configured based on the size of the test data, and the key KEY of the test data is stored as the Value in the heterogeneous association hash table;

[0035] When querying and reading heterogeneous multimodal data of the current object, the heterogeneous associated hash table corresponding to the current object is searched based on the key of the current object, and all the keys of the multimodal data contained in the current object are searched based on the Value in the heterogeneous associated hash table;

[0036] Based on all the keys of the multimodal data contained in the current object, the Value stored corresponding to each key is searched, and the test data of the current key is read based on the Value.

[0037] In the terminal device, periodically performing a snapshot operation on the memory data of the terminal device to store the data in the memory into a temporary file of the terminal device;

[0038] After the snapshot is executed, the current temporary file is used as the current snapshot file.

[0039] The advantages of the present invention are as follows: the distributed file system is used in conjunction with the terminal device to achieve effective storage management of all test data, the data hash table is used to store and index key-value pairs, and the efficiency, scalability and reliability of data management are improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 A flow chart of an embodiment of data management according to the present invention.

[0041] Figure 2 The figure is a schematic diagram of an embodiment of the present invention when performing data query based on key-value pairs.

[0042] Figure 3 The present invention is a flowchart of an embodiment of storing key-value pairs using a data hash table.

[0043] Figure 4 A schematic diagram of an embodiment of a hash table bucket of user corner of the present invention.

[0044] Figure 5 It is a schematic diagram of an embodiment of the heterogeneous multimodal data fusion of the present invention. DETAILED DESCRIPTION

[0045] The present invention will be further described below in conjunction with specific drawings and embodiments.

[0046] In order to effectively manage heterogeneous data generated by a ship virtual test system, a key-value-based distributed data management method is provided. In one embodiment of the present invention, the distributed data management method is used to manage test data generated by a ship virtual test system, wherein the distributed data management method includes:

[0047] Providing a terminal device for managing the test data, and configuring a distributed file system adapted to connect to the terminal device;

[0048] For any test data, based on the data type of the test data, a data hash table corresponding to the data type is configured in the terminal device, so as to store the key-value pairs of the test data using the configured data hash table, wherein:

[0049] Using the data name of the test data as the key KEY, and based on the hash value after performing a hash operation on the key KEY, determining the bucket position where the key-value pair of the test data is stored in the data hash table;

[0050] Based on the size of the test data, the terminal device stores the test data configuration in the terminal device or a distributed file system, uses the storage location information after storage as the Value value, and stores the Value value in the bucket of the data hash table determined above.

[0051] Specifically, the ship virtual test system is the test system launched by the above-mentioned China Numerical Pool. From the above description, it can be seen that when using the ship virtual test system for testing, a large amount of test data will be generated, and a large amount of test data can form heterogeneous data.

[0052] In order to perform ship virtual tests and manage heterogeneous data, terminal devices are generally required. The ship virtual test system is installed in the terminal device. At this time, the terminal device forms a local device. The terminal device can generally use existing commonly used computer devices, such as computer devices that can meet the operation of the ship virtual test system. From the above description, it can be seen that when running the ship virtual test system, the types and sizes of test data generated vary. In order to meet the management of all test data, a distributed file system is also required. The distributed file system is connected to the terminal device. The distributed file system can adopt the existing commonly used form, which is based on the connection with the terminal device and the realization of distributed storage. A distributed file system is a file system that stores files on multiple nodes and accesses and manages data through a network. It is highly scalable and fault-tolerant, and can effectively handle large-scale data storage and access.

[0053] In one embodiment of the present invention, for any test data, based on the type of the test data, a data hash table is configured in the terminal device, that is, in the terminal device, the number of data hash tables corresponds to the number of data types generated by running the ship virtual test system. After the data hash table is configured, the data hash table is used to implement key-value pair storage of the test data. In specific implementation, the data hash table is used to implement the storage and indexing of key-value pairs, and the data hash table is used as the basic unit for storing data. Generally, the data hash table includes a number of buckets, and one bucket can be used to implement the storage of a key-value pair of a test data, or multiple key-value pairs of multiple test data.

[0054] As for the key-value pair, it is known to those skilled in the art that it includes KEY-Value. In one embodiment of the present invention, when the key-value pair is stored in the bucket in the data hash table, for each test data, the data name of the test data is used as the key KEY, and the key KEY is hashed, and the location of the key-value pair of the test data stored in the bucket in the data hash table is determined based on the hash value after the hash operation, wherein the stored data hash table is the data hash table corresponding to the data type of the test data.

[0055] In one embodiment of the present invention, the storage location of the test data is selected based on the size of the test data, that is, the test data is generally stored in the terminal device, or in the distributed file system. Generally, when the test data is large, it is selected to be stored in the distributed file system. After configuring the storage status of the test data, the storage location information after storage is used as the Value value, and the Value value is the Value of the above key-value pair. After obtaining the Value value, the key KEY-Value value of the test data is associated and stored in the above-determined bucket.

[0056] The above shows an embodiment of storing and managing test data. Figure 1 As shown; of course, the management of test data also includes other situations, which will be explained in detail below.

[0057] As can be seen from the above description, the bucket position where the test data is stored in the data hash table is determined by the hash value of the key KEY. Therefore, when the hash operation is performed on the key KEY of the test data, there will be the same hash value. In one embodiment of the present invention, for test data of the same data type, when the hash values ​​of the test data are the same, all key-value pairs of the test data are stored in the same bucket of the data hash table and stored through a linked list.

[0058] Generally, a head pointer of a linked list is stored in a bucket. Specifically, when multiple key-value pairs are stored in the same bucket, the key-value pairs are mapped to a bucket through a hash function to store conflicting key-value pairs. Figure 3 An embodiment of using a data hash table to store key-value pairs is shown in FIG. N That is, the buckets in the data hash table, bucket B1 to bucket B N There are multiple linked lists in each bucket, such as bucket B1, which has linked list A 11 ~ Link List A 1m , each linked list stores a key-value pair. The situations of other buckets can be referred to here, and examples will not be given one by one.

[0059] In one embodiment of the present invention, a data threshold t is configured in the terminal device, wherein:

[0060] When the size of the test data is not greater than the data threshold t, the test data configuration is stored in the terminal device, and the Value value is the disk block in the terminal device + the offset address in the block;

[0061] When the size of the test data is greater than the data threshold t, the test data configuration is stored in the distributed file system, and the Value value is the file name or path information in the distributed file system.

[0062] Specifically, the selection of the data threshold t is generally related to the test data type and data size distribution of the ship virtual test system. For example, the data threshold t can be 50M. When the size of the test data is not greater than 50M, the test data is stored in the terminal device, otherwise, the test data is stored in the distributed file system. When the data threshold t is other cases, please refer to the description here.

[0063] In practice, in disk storage, data is usually organized into fixed-size blocks, and each block has a unique address. The size of the disk block is determined by the operating system or file system used by the terminal device, and is usually the result of hardware architecture and performance requirements. The method of determining the disk block and the offset address within the block depends on the specific disk storage structure.

[0064] RAID is a data storage technology that combines multiple hard disks into a logical unit and improves data reliability and performance through data distribution and redundancy technology; at this time, if some hard disks are broken, it will not affect the overall data. It only needs to insert a new hard disk to restore it, and the work will not be affected during the recovery process. Therefore, in one embodiment of the present invention, a RAID disk array is configured in the terminal device, such as a RAID disk array that can adopt a RAID10 disk array form, and the RAID10 disk array is used to ensure that the storage occupancy is small, and it has high efficiency and reliability. The way to configure the RAID10 disk array and the specific form of the RAID10 disk array can be consistent with the existing ones, and will not be repeated here.

[0065] The determination method is explained below. The determination of the disk block may specifically include:

[0066] 1) Fixed size: Disk blocks are usually of fixed size, such as 4KB, 8KB or other sizes. The file system will divide files and data into blocks according to this size and use blocks to allocate and manage disk space.

[0067] 2) Allocation unit of the file system: A file system usually has an allocation unit, which is the smallest unit of allocatable space. One or more allocation units can form a disk block. The size of this allocation unit may be the size of the disk block or an integer multiple of it.

[0068] 3) Logical block number: The operating system or file system uses a logical block number to identify each disk block. The logical block number starts at 0 and increases incrementally. Each logical block number corresponds to a disk block.

[0069] Determination of the offset address within the block may specifically include:

[0070] 1) Byte address: Within a disk block, a byte address can be used to determine the offset address within the block. The byte address starts at 0 and indicates the specific byte position within the disk block.

[0071] 2) Bitmap offset: In some cases, the file system may use a bitmap to represent the usage of disk blocks. The offset address within the block can be determined by looking at the corresponding bit in the bitmap, which tells the system which blocks are used.

[0072] Example: Assume that the size of the disk block is 4KB, the file system uses the logical block number and the byte address as the offset address within the block.

[0073] Logical block number: 0, 1, 2, ...

[0074] Byte address: 0,1,2,...,4095

[0075] If you want to determine the offset address within the third disk block, you can calculate it as 3*4096, and get the offset address within the block as 12288, which points to the 12288th byte of the disk block.

[0076] The above describes the test data stored in the disk of the terminal device and the method that can be used to obtain the Value value. Of course, other methods can also be used, which can be selected according to needs to satisfy the requirement of storing the test data in the terminal device and obtaining the Value value.

[0077] When the test data is configured by the terminal device to be stored in a distributed file system, the commonly used technical means in this technical field can be used to determine the file name or path information where the test data is stored, and the commonly used technical means in this technical field can be used to obtain the corresponding Value stored in the distributed file system.

[0078] In one embodiment of the present invention, based on the load factor of each data hash table, the terminal device adjusts the size of the corresponding data hash table, wherein:

[0079] For any data hash table, when the load factor of the data hash table is not less than the load factor threshold, the terminal device expands the capacity of the data hash table;

[0080] When the load factor of the data hash table is too small, the terminal device reduces the capacity of the data hash table.

[0081] Specifically, for any data hash table, the load factor of the data hash table can be: lf = n / m, where n is the number of elements in the data hash table, m is the size of the data hash table, the number of elements n in the data hash table specifically refers to the number of buckets filled in the data hash table, and the size m of the data hash table specifically refers to the number of buckets in the data hash table. For any data hash table, after determining the load factor of the data hash table, the load factor can be compared with a load factor threshold. When it is not less than the load factor threshold, the data hash table needs to be expanded to meet the storage of key-value pairs using the data hash table.

[0082] Generally, the load factor threshold can be selected as 0.75. Of course, the load factor threshold can also be selected as other values, so as not to affect the storage of key-value pairs in the data hash table. When the data hash table is expanded, the size of the data hash table is increased, and the storage capacity of key-value pairs is improved. When the load factor of the data hash table is too small, the capacity of the data hash table needs to be reduced to reduce memory waste. For example, when the load factor of the data hash table is less than 0.2, the capacity of the data hash table can be reduced; of course, a load factor threshold that is too small can be selected as needed.

[0083] When expanding the data hash table, in one embodiment of the present invention, the data hash table is expanded based on a progressive rehash method, wherein the expansion method includes:

[0084] Create a new data hash table that is consistent with the current data hash table, apply for memory space for the new data hash table, and set the capacity expansion flag to 1, wherein the capacity of the new data hash table is greater than the capacity of the current data hash table;

[0085] Sequentially migrate some key-value pairs in the current data hash table to the new data hash table created above;

[0086] The new data hash table is used as the current data hash table, and the expansion flag is set to 0;

[0087] Migrate the remaining key-value pairs in the data hash table before expansion to the current data hash table until all key-value pairs are migrated to the current data hash table.

[0088] In specific implementation, when expanding the data hash table, the terminal device can pre-allocate a certain amount of memory space to avoid frequent memory allocation and fragmentation. The terminal device can use the commonly used methods in the technical field to implement the pre-allocation of memory space. Generally, the pre-allocated memory space can be twice the space occupied by the data hash table to be expanded. Of course, the size of the pre-allocated memory space can be selected according to needs.

[0089] Within the pre-allocated memory space, the terminal device applies for memory space for the newly created data hash table and sets the expansion flag to 1. Setting the expansion flag to 1 indicates that the current data hash table is being expanded. Generally, when the data hash table is expanded, the size of the newly created data hash table is twice the size of the data hash table to be expanded.

[0090] For the expanded data hash table, a portion of the key-value pairs can be migrated to the new data hash table in a sequential manner. This step will not affect the operation of the terminal device because the key-value pairs in the new data hash table will not be exposed to the terminal device. In addition, the number of sequentially migrated key-value pairs can be selected as needed, subject to not affecting the operation of the terminal device.

[0091] When the expansion flag is set to 0, it means that the expansion is completed. At this time, if the terminal device operates, the expanded data hash table and the new data hash table will be processed at the same time to ensure data consistency.

[0092] In the subsequent period, the data in the expanded data hash table will continue to be migrated to the current data hash table in the same way until all key-value pairs in the expanded data hash table are migrated to the current data hash table. During this period, the expanded data hash table and the current data hash table coexist in the terminal device. After that, the terminal device can delete the data hash table before the expansion, that is, the terminal device will delete the original data hash table.

[0093] An embodiment of expanding the capacity of the data hash table is given above. When the capacity of the data hash table needs to be reduced, the commonly used technical means in the technical field can be adopted. The specific method can be selected according to the needs so as to reduce the capacity of the data hash table to the required target.

[0094] In one embodiment of the present invention, in the terminal device, a time wheel for clearing key-value pairs that have not been used for a long time is maintained for each data hash table.

[0095] Specifically, the Time Wheel is a data structure used to process time-related events. It is usually used to implement the LRU (Least Recently Used) approximation algorithm to remove key-value pairs that have not been used for a long time, thereby reducing memory usage. In the data hash table, the purpose of using the Time Wheel is to maintain efficient storage and fast data search.

[0096] In specific implementation, by periodically triggering the time wheel and recording the access time of the key-value pair, the key-value pairs that have not been used for a long time can be cleaned up in time on the basis of approximate LRU, so as to maintain the high performance of the data hash table. In actual implementation, it may be necessary to adjust the size and triggering period of the time wheel according to specific needs, and the method and process of using the time wheel to clean up the key-value pairs that have not been used for a long time can be consistent with the existing ones.

[0097] In one embodiment of the present invention, for each data hash table, the terminal device maintains a metadata structure of the data hash table, and the metadata structure includes at least the size of the data hash table and the load factor. The metadata structure can facilitate the operation and management of the data hash table.

[0098] Specifically, "maintenance" refers to continuously updating and managing the metadata structure of the data hash table at runtime to ensure the performance and effectiveness of the operation of the data hash table. The metadata structure usually contains some important information about the hash table, such as size, load factor, etc.

[0099] The specific maintenance process includes the following aspects:

[0100] 1) Dynamic resizing: The size of the data hash table needs to be dynamically adjusted at runtime to adapt to changes in the amount of data, such as expanding or reducing the capacity of the data hash table as mentioned above.

[0101] 2) Load factor control: The load factor is an indicator used to measure the degree of filling of the data hash table. By maintaining the meta-information structure, the load factor can be monitored in real time and the dynamic adjustment of the hash table can be triggered when necessary.

[0102] 3) Performance monitoring and optimization: The metadata structure can contain indicators about the performance of the data hash table, such as the average linked list length, the longest linked list length, etc. By monitoring these indicators, potential performance bottlenecks can be discovered and corresponding optimization operations can be performed.

[0103] 4) Statistical information: The metadata structure can be used to record statistical information about the data hash table, such as the number of insertions, deletions, queries, etc. These statistical information are very helpful for understanding the usage and performance of the hash table.

[0104] 5) Hash function adjustment: To adapt to changes in data distribution, the hash function may need to be adjusted. The metadata structure can contain information about hash function performance and collisions to help optimize the selection or adjustment of the hash function.

[0105] 6) Exception handling: The metadata structure can also record some abnormal situations, such as insufficient memory, frequent hash conflicts, etc. By monitoring these abnormal situations, corresponding measures can be taken, such as appropriate rehashing, resizing, etc.

[0106] In general, by maintaining the metadata structure, the data hash table can better adapt to changes at runtime and improve its efficiency and stability. The method and process of maintaining the metadata structure of each data hash table in the terminal device can be consistent with the existing ones. The specific method of maintaining the metadata structure will not be illustrated here.

[0107] In one embodiment of the present invention, in a terminal device, a snapshot operation is periodically performed on the memory data of the terminal device to store the data in the memory into a temporary file of the terminal device;

[0108] After the snapshot is executed, the current temporary file is used as the current snapshot file.

[0109] Specifically, the terminal device can implement the snapshot operation of the memory data in an existing commonly used manner, and the period of executing the snapshot operation can be selected according to actual needs, such as 10 minutes. When an accident occurs, data recovery can be achieved through the snapshot file. In specific implementation, when executing the snapshot operation, when the data is stored in a temporary file, due to the limited read and write speed of the disk, the data is written to the temporary file in an appending manner.

[0110] The present invention can implement write append, wherein all executed write commands are written in a log file in a disk in an appended manner, and the log file saves all executed write commands. A data set can be reconstructed by replaying the commands in the log file. Write append is a way of writing data, which does not overwrite the original data when writing, but appends the new data to the end of the existing data. This writing method can reduce random write operations to the disk, improve write performance, and reduce the possibility of data being overwritten, which is beneficial to data recovery and repair, thereby enhancing the high availability of the system. In specific implementation, the executed write commands include write commands written to the terminal device and write commands written to the distributed file system.

[0111] In one embodiment of the present invention, for any key-value pair to be queried, the query includes:

[0112] Calculate the hash value of the key KEY in the key-value pair to be queried, and determine the data hash table corresponding to the key-value pair to be queried and the bucket corresponding to the data hash table based on the calculated hash value;

[0113] In the determined bucket, traverse the linked list to determine the Value value that matches the key-value pair to be queried;

[0114] Based on the determined Value, test data is read.

[0115] Figure 2 An embodiment of querying test data using key-value pairs is shown in the figure. In the figure, based on the key KEY of the key-value pair to be queried, the type of Value corresponding to the key KEY is determined according to the identifier, wherein the identifier is the file suffix of the file corresponding to the key KEY, and the type of the file is determined according to the file suffix. Therefore, it can be seen from the above description that the data hash table corresponding to the key-value pair to be queried can be determined, and then the Value value can be determined, and finally the test data can be read according to the key-value pair.

[0116] In one embodiment of the present invention, for each user in the ship virtual test system, the data hash table corresponding to all test data under the user is configured as a hash table bucket, wherein:

[0117] The hash table bucket is in the terminal device, and the hash table bucket of each user is stored in a disk of the terminal device at a location sufficient to be read into the memory at one time.

[0118] In the ship virtual test system, user registration can be realized. Therefore, multiple user information can be present in the terminal device and the ship virtual test system. In one embodiment of the present invention, for each user, the data hash table corresponding to all the test data of the user can be configured as a hash table bucket, wherein the hash table bucket is in the terminal device, and the hash table bucket of each user is stored in the disk of the terminal device at a location sufficient to be read into the memory at one time. Specifically, after configuring the data hash table corresponding to all the test data of the same user into a hash table bucket, each user has his own operating space, and adding and deleting users will be more convenient; in addition, it can also improve the convenience of maintenance and be safer.

[0119] After configuring the data hash table corresponding to all test data of a user as a hash table bucket, the user is used as the key KEY and hash operation is performed to obtain all the test data of the current user, such as Figure 4 As shown. From the above description, it can be seen that when a hash table bucket is equipped for each user, you only need to traverse the hash table bucket of the current user to obtain all the data of the user. In specific implementation, it is generally used when the user is about to delete or back up data.

[0120] In one embodiment of the present invention, for heterogeneous multimodal data of the same object in a ship virtual test system, the name of the current object is used as a key KEY, and a heterogeneous association hash table of the current object is configured, wherein the test data of each modality includes the name of the current object;

[0121] For any test data in the heterogeneous multimodal data, a storage state is configured based on the size of the test data, and the key KEY of the test data is stored as the Value in the heterogeneous association hash table;

[0122] When querying and reading heterogeneous multimodal data of the current object, the heterogeneous associated hash table corresponding to the current object is searched based on the key of the current object, and all the keys of the multimodal data contained in the current object are searched based on the Value in the heterogeneous associated hash table;

[0123] Based on all the keys of the multimodal data contained in the current object, the Value stored corresponding to each key is searched, and the test data of the current key is read based on the Value.

[0124] Specifically, heterogeneous multimodal data refers to the test results in the ship virtual test system, which often contain data in various forms such as text, pictures, animations, etc. If some data describe the same object / file from different perspectives, it is called heterogeneous multimodal data of the same object / file; that is, the heterogeneous multimodal data of the same object / file has multiple different data types, such as the text, pictures and / or animation data types mentioned above.

[0125] For heterogeneous multimodal data, the name of the current object is generally retained in the name of each test data. The specific way of including the current object name includes using the name of the current object as a prefix. For example, a prefix will be automatically added when the program is compiled. For example, for a picture of the ship Titans, a prefix of "Titans_" will be added when the name of the picture is stored. The specific implementation method of retaining the current object name in each test data can be selected according to needs.

[0126] In practical applications, there is a need to fuse heterogeneous multimodal data of the same object so as to quickly read all modal test data. In order to meet the need, in one embodiment of the present invention, the name of the current object can be used as the key KEY, and a heterogeneous association hash table is configured. The heterogeneous association hash table is the same as the above-mentioned data hash table, that is, it mainly implements the storage of key-value pairs, and the key KEY of the key-value pairs stored in the heterogeneous association hash table is the name of the current object.

[0127] For heterogeneous multimodal data of the same object, the same as the above-mentioned test data, the storage state needs to be configured according to the size of each modal test data. The storage state specifically refers to storage in the terminal device or in the distributed file system. The specific storage in the terminal device or the distributed file system can refer to the above description, which will not be repeated here. Of course, when each test data in the heterogeneous multimodal data is stored, there is also a data hash table for each type of test data. The use of the data hash table to implement the storage of test data can refer to the above description.

[0128] In one embodiment of the present invention, after the test data in the heterogeneous multimodal data is stored, the key KEY of the test data is used as the Value value in the heterogeneous association hash table, that is, the name of the test data is stored as the Value value in the heterogeneous association hash table. At this time, the key-value pair in the heterogeneous association hash table can be obtained. After the storage of the corresponding key-value pair is realized by using the heterogeneous association hash table, the heterogeneous multimodal data of the same object can be associated as class column data, so that the heterogeneous multimodal data of the same object are all on the same linked list, that is, when the class column data is associated, the key KEY of the heterogeneous multimodal data of the same object is stored on the same linked list for subsequent search.

[0129] When querying and reading the heterogeneous multimodal data of the current object, the heterogeneous association hash table corresponding to the current object is searched based on the key of the current object, that is, the corresponding heterogeneous association hash table can be found according to the name of the current object. From the above description, it can be seen that by reading the Value value in the heterogeneous association hash table, all the keys of the multimodal data contained in the current object can be quickly found. At this time, the name of the experimental data of the heterogeneous multimodal data of the current object can be known.

[0130] From the above management process of each test data, it can be known that based on all the keys of the multimodal data contained in the current object, the Value stored corresponding to each key is searched, and the test data of the current key is read based on the Value. Therefore, for the heterogeneous multimodal data of the same object, the key corresponding to all modal test data can be searched according to the name of the object, and then the key is sent to the corresponding data hash table to read the Value stored in the test data. That is, through the heterogeneous multimodal data of the same object, two hash operations can be performed to quickly find all the stored test data, such as Figure 5 shown.

[0131] The following is an example of how to read data based on two hash operations. Specifically:

[0132] The cargo ship Titans has heterogeneous multimodal data in storage formats such as text, pictures, and files. When you need to read all the picture format data of the cargo ship Titans, you have:

[0133] The first hash operation: find the heterogeneous association hash table (linked list) based on the cargo ship Titans, and find the key KEY of the cargo ship Titans' image data in the heterogeneous association hash table;

[0134] Second hash operation: Based on the key KEY obtained in the first search, the corresponding Value value is searched in the hash table of the image type, and the image data expressing the cargo ship Titans is read based on the searched Value value.

[0135] In the specific implementation, when managing the test data, all hash operations use the MurmurHash2 algorithm as the hash function to reduce memory usage. The method and process of hash operations using the MurmurHash2 algorithm can be consistent with the existing ones and will not be repeated here.

[0136] The above provides an embodiment of test data storage and key-value pair-based query when managing test data. When the test data needs to be modified, the test data can be modified by commonly used means in the technical field after the test data is queried, and the modification includes deletion of the test data.

[0137] The present invention also supports multi-copy consistency and high availability when managing test data, specifically:

[0138] Support for multi-copy consistency: Databases usually use multiple replicas to improve availability and fault tolerance. Each replica is a copy of the database and is distributed in different physical locations. If a replica fails, data can still be obtained from other replicas. In this multi-copy scenario, it should be ensured that all replicas can reach the same state within a certain period of time, which is achieved by coordinating data synchronization between replicas.

[0139] There are usually two ways to coordinate data synchronization between replicas:

[0140] 1) Master-Slave Replication: In master-slave replication, one database node (master node) is designated as the master database, responsible for receiving and processing write operations and recording these write operations in the log. Other nodes (slave nodes) maintain consistency by replicating the data of the master node. The master node sends the changes of the write operation to the slave node, and the slave node updates the data according to the operation of the master node. This method can realize asynchronous replication of data, and the data update of the master node will be propagated to the slave node with delay.

[0141] 2) Multi-Master Replication: In multi-master replication, multiple nodes are allowed to receive and process write operations, and any node can become a master node. Nodes replicate each other's write operations to maintain data consistency. This approach can improve system availability and load balancing capabilities, but it also increases the complexity of data conflicts and consistency maintenance.

[0142] In specific implementation, any of the above methods can be used to achieve data synchronization between multiple copies. The specific method can be selected according to needs, based on the one that can effectively support the consistency of multiple copies.

[0143] Regarding high availability: High availability means that the system can provide services at any time, even if some components or copies fail. Generally, multi-copy deployment is usually used together with high availability to ensure that it can continue to run even if a copy fails. In one embodiment of the present invention, the above-mentioned RAID disk array + write-append + distributed file system can achieve support for multi-copy consistency and high availability.

[0144] RAID disk arrays provide data redundancy and fault tolerance, write-append improves data writing performance and reliability, and distributed file systems implement distributed storage and access to data, together building a system architecture with high availability and reliability.

Claims

1. A key-value-based distributed data management method, characterized in that: Used for managing test data generated by a ship virtual test system, wherein the distributed data management method comprises: Providing a terminal device for managing the test data, and configuring a distributed file system adapted to connect to the terminal device; For any test data, based on the data type of the test data, a data hash table corresponding to the data type is configured in the terminal device, so as to store the key-value pairs of the test data using the configured data hash table, wherein: Using the data name of the test data as the key KEY, and determining the bucket position where the key-value pair of the test data is stored in the data hash table based on the hash value after performing a hash operation on the key KEY; Based on the size of the test data, the terminal device stores the test data configuration in the terminal device or in a distributed file system, and uses the storage location information after storage as the Value value, and stores the Value value in the bucket of the data hash table determined above; For test data of the same data type, when the hash values ​​of the test data are the same, the key-value pairs of all the test data are stored in the same bucket of the data hash table and stored through a linked list; For heterogeneous multimodal data of the same object in the ship virtual test system, the name of the current object is used as the key KEY, and a heterogeneous association hash table of the current object is configured, in which the test data of each modality contains the name of the current object; For any test data in the heterogeneous multimodal data, the storage state is configured based on the size of the test data, and the key KEY of the test data is stored as the Value in the heterogeneous associative hash table, so that the keys KEY of the heterogeneous multimodal data of the same object are stored on the same linked list; When querying and reading heterogeneous multimodal data of the current object, the heterogeneous associated hash table corresponding to the current object is searched based on the key of the current object, and all the keys of the multimodal data contained in the current object are searched based on the Value in the heterogeneous associated hash table; Based on all the keys of the multimodal data contained in the current object, find the Value stored corresponding to each key, and read the test data of the current key based on the Value; Based on the load factor of each data hash table, the terminal device adjusts the size of the corresponding data hash table, where: For any data hash table, when the load factor of the data hash table is not less than the load factor threshold, the terminal device expands the capacity of the data hash table; When the load factor of the data hash table is too small, the terminal device reduces the capacity of the data hash table; The data hash table is expanded based on a progressive rehash method, wherein the expansion method includes: Create a new data hash table that is consistent with the current data hash table, apply for memory space for the new data hash table, and set the expansion flag to 1, wherein the capacity of the new data hash table is greater than the capacity of the current data hash table; Sequentially migrate some key-value pairs in the current data hash table to the new data hash table created above; The new data hash table is used as the current data hash table, and the expansion flag is set to 0; Migrate the remaining key-value pairs in the data hash table before expansion to the current data hash table until all key-value pairs are migrated to the current data hash table.

2. The key-value-based distributed data management method according to claim 1 is characterized in that: Configure the data threshold t in the terminal device, where: When the size of the test data is not greater than the data threshold t, the test data configuration is stored in the terminal device, and the Value value is the disk block in the terminal device + the offset address in the block; When the size of the test data is greater than the data threshold t, the test data configuration is stored in the distributed file system, and the Value value is the file name or path information in the distributed file system.

3. The key-value-based distributed data management method according to claim 1, characterized in that: For each user in the ship virtual test system, the data hash table corresponding to all test data under the user is configured as a hash table bucket, where: The hash table bucket is in the terminal device, and the hash table bucket of each user is stored in a disk of the terminal device at a location sufficient to be read into the memory at one time.

4. The key-value-based distributed data management method according to claim 1, characterized in that: In the terminal device, a time wheel is maintained for each data hash table to clear key-value pairs that have not been used for a long time.

5. The key-value-based distributed data management method according to claim 1, characterized in that: For any key-value pair to be queried, the query includes: Calculate the hash value of the key KEY in the key-value pair to be queried, and determine the data hash table corresponding to the key-value pair to be queried and the bucket corresponding to the data hash table based on the calculated hash value; In the determined bucket, traverse the linked list to determine the Value value that matches the key-value pair to be queried; Based on the determined Value, test data is read.

6. The key-value based distributed data management method according to any one of claims 1 to 4, characterized in that: In the terminal device, periodically performing a snapshot operation on the memory data of the terminal device to store the data in the memory into a temporary file of the terminal device; After the snapshot is executed, the current temporary file is used as the current snapshot file.

Images