System based on cloud platform and used for separating storage and computation during ship virtual testing

Abstract

A system which is based on a cloud platform and is used for separating storage and computation during ship virtual testing, said system comprising: a client module, which is the outermost layer of the entire system and is used to directly interact with a user, and provide an API for the user and return a task execution result; a distributed query module, which is connected to the client module and is used to, after a task sent by the client module is received, parse the task, access a storage and computation scheduling module, acquire a result upon completion of the specific task, and send the result to the client module; a distributed storage module, which is connected to the distributed query module and is used to store and encrypt data in the entire system, and provide data support for the storage and computation scheduling module and the distributed query module; and the storage and computation scheduling module, which is connected to the distributed query module and the distributed storage module and is responsible for carrying out effective data caching and scheduling storage for the entire system.

Description

A cloud-based storage and computing separation system for ship virtual testing Technical Field

[0001] The present invention belongs to the field of propagation virtual test engineering, and in particular relates to a cloud platform-based ship virtual test storage and computing separation system. Background Art

[0002] Since the first step in advanced ship design and offshore operation platform construction is to use software to complete virtual experiments of various fluid mechanics solution methods, the effect of virtual experiments largely determines the technical performance and manufacturing efficiency of marine-related equipment.

[0003] However, due to the large amount of computation and high equipment requirements of fluid mechanics solution methods, ship virtual test cloud computing platforms usually use expensive mainframes or parallel clusters as equipment, and the computing platforms mostly directly store test-related data on the computing devices. A large amount of data occupies the storage space of the computing devices, which makes the calculation impossible.

[0004] Summary of the Invention

[0005] In order to solve the above-mentioned problems existing in the prior art, the present invention proposes a cloud platform-based ship virtual test storage and computing separation system to avoid using expensive parallel computing nodes for data storage, avoid making storage space a bottleneck of the platform, enable the platform to have good economic efficiency and scalability, and comprehensively improve data management efficiency.

[0006] To achieve the above objectives, the present invention provides the following solution: a cloud platform-based ship virtual test storage and computing separation system, comprising:

[0007] The client module is the outermost layer of the entire system, used to interact directly with users, provide APIs to users and display the results of task execution;

[0008] A distributed query module, connected to the client module, is used to perform task analysis after receiving the task sent by the client module, and then access the storage and calculation scheduling module to complete the specific task to obtain the result and send it to the client module;

[0009] A distributed storage module, connected to the distributed query module, is used to store and encrypt data in the entire system, thereby providing data support for the storage and computing scheduling module and the distributed query module;

[0010] The storage and computing scheduling module is connected to the distributed query module and the distributed query module respectively, and is responsible for performing effective data caching and scheduling the storage of the entire system.

[0011] Preferably, the client module includes a user login unit, a first API unit, and a first communication unit;

[0012] The user login unit is used for user identity authentication and user operation authority authentication;

[0013] The first API unit is used to provide an API interface to the upper layer and return query results;

[0014] The first communication unit is used to implement a TCP connection with the distributed query module.

[0015] Preferably, the user login unit includes a user authenticator, a rights authenticator, and an identity management unit;

[0016] The user authenticator is used to read the user's account password to authenticate the user's identity;

[0017] The authority authenticator is used to further authenticate the user's operation authority;

[0018] The identity management unit manages user rights in the form of allocating operation rights to user roles. When user rights are changed, the operation can be completed by simply changing the user role.

[0019] Preferably, the user identity includes a common user and an administrator user;

[0020] The ordinary user can only operate in the user interface, and the administrator user can jump to the program interface after passing the authentication and operate through SQL statements;

[0021] The user's operation permissions include read and write, read only, and write only.

[0022] Preferably, the distributed query module includes an SQL statement parsing unit, an optimization unit, a query execution unit, a second communication unit, and a second API unit;

[0023] The SQL statement parsing unit is used to receive tasks sent by the client module through the listening port, and obtain SQL statement text after processing the tasks, and parse the SQL statement text according to SQL grammar rules through the SQL parser to obtain a query plan;

[0024] The optimization unit is used to optimize the query plan to minimize the execution cost of the query plan;

[0025] The query execution unit is used to locate the optimized query plan in the distributed database and establish a corresponding connection to transmit and calculate data;

[0026] The second API unit is used to provide an API interface and return query results to the upper layer, and provide an API interface and input query plans to the lower layer;

[0027] The second communication unit is used to realize TCP connection with the distributed storage module and the storage and computing scheduling module.

[0028] Preferably, the SQL parser includes a lexical parser and a grammatical parser;

[0029] The optimization unit includes a physical optimizer and a logical optimizer;

[0030] The physical optimizer and the logical optimizer provide logical optimization functions and physical optimization functions respectively;

[0031] The logic optimization function is to use the rules of relational algebra and a series of rules at the logic level to perform lightweight processing on the query plan;

[0032] The physical optimization function is to select a physical query plan with the minimum execution cost;

[0033] The query execution unit locates the data shard corresponding to the value according to the key parsed from the SQL statement, establishes a corresponding connection through the second API unit and the second communication unit, and performs data transmission and calculation.

[0034] Preferably, the distributed storage module includes a meta-information management unit, a RocksDB storage engine unit, a Raft algorithm unit, a third communication unit, and a third API unit;

[0035] The meta-information management unit is used to perform cluster management on the distributed storage module, and periodically check the nodes and blocks in the replication cluster to balance data and load;

[0036] The RocksDB storage engine unit is used to provide the data required by the computing unit and store the computing results;

[0037] The Raft algorithm unit is used to maintain the consistency of multiple copies in the distributed storage module;

[0038] The third API unit is used to provide an API interface to the upper layer and return query results, and to provide an API interface to the lower layer and perform data transmission;

[0039] The third communication unit is used to realize TCP connection with the distributed query module and the storage and calculation scheduling module.

[0040] Preferably, the storage and calculation scheduling module includes a calculation unit, a cache management unit, a storage scheduling unit, a fourth communication unit, and a fourth API unit;

[0041] The computing unit is responsible for computing tasks;

[0042] The cache management unit is used to predict future scheduling tasks, cache data, exchange data, and manage the data used by the computing unit according to the cache strategy to solve the problem of frequent communication with the distributed storage module;

[0043] The storage scheduling unit is used to collect status information of nodes and blocks in the cluster of the entire system, and formulate a scheduling strategy for the data in the distributed storage module through the fourth API interface;

[0044] The fourth API unit is used to provide an API interface to the upper layer and return query results, and to provide an API interface to the lower layer and perform data transmission;

[0045] The fourth communication unit realizes TCP connection with the distributed storage module and the distributed query unit module.

[0046] Preferably, the computer node in the computing unit first accesses the cache management unit to obtain data, and if the data does not exist, initiates a request to the storage scheduling unit to schedule the corresponding data in the distributed storage module;

[0047] The data exchange operation of the cache management unit includes calling into cache and replacing cold data;

[0048] The calling into cache is to place the file in a storage area specially used for caching in the computing unit;

[0049] The cold data replacement is to remove some data from the cache pool through the cache scheduler when the cache pool is full and to bring in new data;

[0050] The storage scheduling unit manages the cluster through the storage scheduling node. There are heartbeat packets between the storage nodes. The storage scheduling node is used to check the status information of the storage node and formulate a scheduling strategy.

[0051] Preferably, the communication between the first communication unit, the second communication unit, the third communication unit and the fourth communication unit adopts an encrypted transmission mode, and the data is encrypted end-to-end. The data is decrypted when the client reads it and encrypted when the client writes it.

[0052] Compared with the prior art, the present invention has the following advantages and technical effects:

[0053] (1) Design a distributed, block-based, multi-backup storage system for ship virtual test data to reduce storage costs while ensuring good scalability and data robustness.

[0054] (2) Perform efficient storage scheduling in the distributed storage module to achieve balanced data distribution.

[0055] (3) According to the user's demand for cloud node simulation computing capabilities and efficient data access capabilities, an efficient distributed read-write engine is implemented based on the read-write separation technology of the master-slave structure.

[0056] (4) Manage the computing system and storage system separately and schedule them in a unified manner to improve the storage efficiency, query efficiency, and computing efficiency of the cloud computing platform. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] The accompanying drawings, which constitute part of this application, are intended to provide a further understanding of this application. The exemplary embodiments and descriptions of this application are intended to explain this application and do not constitute an improper limitation on this application. In the accompanying drawings:

[0058] FIG1 is a schematic diagram of the overall system structure of an embodiment of the present invention;

[0059] FIG2 is a schematic structural diagram of a client module according to an embodiment of the present invention;

[0060] FIG3 is a schematic diagram of the structure of a distributed query module according to an embodiment of the present invention;

[0061] FIG4 is a schematic structural diagram of a distributed storage module according to an embodiment of the present invention;

[0062] FIG5 is a schematic diagram of the structure of the storage and calculation scheduling module according to an embodiment of the present invention. DETAILED DESCRIPTION

[0063] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments in this application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0064] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0065] As shown in Figure 1, the present invention provides a cloud platform-based ship virtual test storage and computing separation system, which consists of a client module, a distributed query module, a distributed storage module, and a storage and computing scheduling module. The query, storage, scheduling and power supply of the storage and computing separation system are completed by them in collaboration.

[0066] (1) Client module

[0067] The client module consists of three submodules: user login module, API module, and communication module. The client module is the outermost layer of the entire system, interacting directly with the user, providing the user with APIs and displaying the results of task execution.

[0068] The user login module includes a user authenticator and a permission authenticator, which are responsible for user identity authentication and user operation permission authentication; the API module is responsible for providing the API interface to the upper layer and returning query results; the communication module realizes the TCP connection with the distributed query module. When the user calls the API, the client module first reads the user's account password through the user authenticator to authenticate the user's identity, and then further authenticates the user's operation permission through the permission authenticator. After completing the authentication, the task is generated according to the user input, and the task is sent to the distributed query module through the communication module after the task queue. After that, the API module waits for the corresponding return result in the result table, and returns it to the upper-level user after finding the result. If the task is not executed or the distributed query module crashes when the task arrives, when the distributed query module is restarted, the task in the task queue will be resent.

[0069] (2) Distributed query module

[0070] The distributed query module consists of five submodules: SQL statement parsing module, optimization module, query execution module, communication module, and API module. After receiving tasks from the client module, the distributed query module parses the tasks and then accesses the storage and computing scheduling module to complete the specific tasks and obtain the results for the client module.

[0071] The SQL statement parsing module receives tasks sent by the client module through the listening port, and obtains the SQL statement text after processing. The SQL parser will parse the SQL statement according to the SQL syntax rules, convert it into a query plan, and hand it over to the optimization module for further processing; the optimization module includes a physical optimizer and a logical optimizer to minimize the execution cost of the query plan; the query execution module locates the optimized query plan in the distributed database, establishes the corresponding connection, and performs data transmission and calculation. After that, the API module waits for the corresponding return result in the result table, and returns the result to the upper-level client after finding it; the API module is responsible for providing the API interface and returning query results to the upper layer, and providing the API interface and incoming query plans to the lower layer; the communication module implements TCP connection with the distributed storage module and the storage and computing scheduling module.

[0072] (3) Distributed storage module

[0073] The distributed storage module is responsible for storing and encrypting data throughout the system, providing data support for the storage and computing scheduling module and the distributed query module. The distributed storage module consists of five sub-modules: the metadata management module, the RocksDB storage engine module, the Raft algorithm module, the communication module, and the API module. The metadata management module performs cluster management for the distributed storage module, periodically checking the nodes and blocks in the replication cluster to balance data and load; the RocksDB storage engine module is responsible for providing the data required by the computing sub-module and storing the calculation results; the Raft algorithm module is responsible for maintaining the consistency of multiple copies in the distributed storage module; the API module is responsible for providing the API interface and returning query results to the upper layer, and providing the API interface and data transmission to the lower layer; the communication module implements TCP connections with the distributed query module and the storage and computing scheduling module.

[0074] (4) Storage and computing scheduling module

[0075] The storage and computing scheduling module is the core module of the entire system, responsible for effective data caching, saving network resources, improving data transmission efficiency, and scheduling the storage of the entire system. The storage and computing scheduling module consists of five sub-modules, including the computing module, cache management module, storage scheduling module, communication module, and API module. The storage scheduling module is responsible for collecting the status information of the nodes and blocks in the cluster of the entire system, and formulating scheduling strategies for the data in the distributed storage module through the API interface; the computing module is responsible for computing tasks; the cache management module is responsible for predicting future scheduling tasks, caching data, and solving problems caused by frequent communication with the distributed storage module; the API module is responsible for providing the API interface and returning query results to the upper layer, providing the API interface to the lower layer and performing data transmission; the communication module implements TCP connections with the distributed storage module and distributed query module.

[0076] The query, storage and scheduling functions of the cloud platform-based ship virtual test storage and computing separation system are completed by the client module, distributed query module, distributed storage module and storage and computing scheduling module.

[0077] The query function is accomplished collaboratively by the client module and the distributed query module. The client module provides an API interface to the upper layer, generates specific tasks, and sends them to the distributed query module through the communication module. Once the task reaches the distributed query module, it enters the task queue through the communication module, awaiting parsing by the SQL statement parsing submodule. After the query plan is executed, the results are serialized and packaged, and the query results are returned through the communication module. After receiving the query results, the client module organizes and returns them to the upper layer.

[0078] The storage function is performed independently by the distributed storage module. The distributed storage module is responsible for storing, encrypting, and managing the data generated in the entire system, providing an API interface to the upper layer, storing the data required by the scheduling module, and returning the data through the communication module.

[0079] The scheduling function is accomplished collaboratively by the distributed storage module and the storage and computing scheduling module. The storage scheduling module schedules the data in the distributed storage module through the communication module, transmitting the data used for calculation to the computing module and the cache management module. The cache management module manages the data used by the computing module according to the cache policy.

[0080] Example 1

[0081] As shown in Figure 1, the cloud platform-based ship virtual test storage and computing separation system consists of four main modules: client module, distributed query module, distributed storage module, and storage and computing scheduling module.

[0082] The entire storage and computing separation system provides data query, storage, and scheduling functions for the virtual test data of the numerical pool, all of which are completed by the collaboration of the above modules. After the user logs in through the client module, he submits a query request to the distributed query module. After the distributed query module parses the query request through the SQL statement parsing module, it sends the query request to the distributed storage module and the storage and computing scheduling module through the API module and the communication module. For query data requests, the corresponding data is directly found in the distributed storage module and the query results are returned; for calculation data requests, the required data is scheduled for calculation in the cache management module. If the data does not exist, a request is initiated to the distributed storage module through the storage scheduling module, the required data is scheduled, and calculation is performed. After the calculation, the query results are returned to the upper layer.

[0083] As shown in Figure 2, the client module consists of three submodules: the user login module, the API module, and the communication module. The user login module provides user identity authentication and operation permission authentication. The API module provides an interface for returning query results and accessing the distributed query module. The communication module provides TCP for communication with other modules.

[0084] There are two types of user identities: ordinary users and administrator users. Ordinary users can only operate in the user interface, while administrator users can jump to the program interface after passing authentication and operate through SQL statements.

[0085] The user's operation permissions are divided into three types: read and write, read only, and write only.

[0086] Further optimize the solution, manage user identities, and manage user permissions by assigning operation permissions to user roles. When user permissions change, the operation can be completed simply by changing the user role.

[0087] To further optimize the solution, the communication between modules in the communication module adopts encrypted transmission, and the data is encrypted end-to-end. The data is decrypted when the client reads it, and the data is encrypted when the client writes it.

[0088] As shown in Figure 3, the distributed query module consists of five submodules: the SQL statement parsing module, the optimization module, the query execution module, the communication module, and the API module. The SQL statement parsing module includes a lexical parser and a syntax parser to parse SQL statements. The optimization module provides logical and physical optimization functions. Logical optimization uses relational algebra and a series of rules at the logical level to make the query plan lightweight. Physical optimization selects the physical query plan with the lowest execution cost. The query execution module locates the data shard corresponding to the value based on the key parsed from the SQL statement, establishes corresponding connections through the API module and the communication module, and transmits data.

[0089] As shown in Figure 4, the distributed storage module consists of five submodules: the metadata management module, the RocksDB storage engine module, the Raft algorithm module, the communication module, and the API module. The metadata management module makes management decisions about data storage based on received heartbeat information: the storage area heartbeat and the block area heartbeat. The RocksDB storage engine module is an LSM-tree architecture engine with key-value pair storage and read / write capabilities, used to store Raft logs and user data. The Raft algorithm is a consensus algorithm that maintains consistency across multiple replicas of data in a distributed data storage system. The API module is responsible for providing API interfaces and returning query results to upper layers, as well as providing API interfaces and data transmission to lower layers. The communication module implements TCP connections with the distributed query module and the storage and computing scheduling module.

[0090] As shown in Figure 5, the storage and computing scheduling module consists of five submodules: the computing module, the cache management module, the storage scheduling module, the communication module, and the API module. The computer nodes in the computing module first access the cache management module to obtain data. If the data does not exist, they initiate a request to the storage scheduling module to schedule the corresponding data from the distributed storage module.

[0091] The cache management module mainly involves two data exchange operations: loading into the cache and cold data replacement. Loading into the cache places the file in the storage area dedicated to caching in the computing module. For cold data replacement, when the cache pool is full, the cache scheduler removes some data from the cache pool and loads new data in.

[0092] The storage scheduling module manages the cluster through the storage scheduling node. There are heartbeat packets between the storage nodes and the storage scheduling nodes. The storage scheduling nodes can check the status information of the storage nodes and formulate scheduling strategies.

[0093] This paper proposes a cloud-based, storage- and computing-separated system for ship virtual testing. Specifically, the system employs a distributed cluster on the storage side to partition and store data in multiple backups, and employs big data-related technologies to manage data, providing excellent compatibility with multimodal virtual test data. Furthermore, to meet the specific data security requirements of the test, a unified and strict data permission management and security mechanism is provided for both the storage and computing sides. From a holistic perspective, the system rationally schedules and optimizes the coordinated operation of storage and computing, enhancing the user's data query experience without compromising the speed of test completion.

[0094] The above are merely preferred embodiments of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. A cloud platform-based ship virtual test storage and computing separation system, characterized in that: include: The client module is the outermost layer of the overall system, used to interact directly with users, provide APIs to users and display the results of task execution; A distributed query module, connected to the client module, is used to perform a task analysis operation on the task after receiving the task sent by the client module, and then access the storage and calculation scheduling module to complete the specific task to obtain the result and send it to the client module; A distributed storage module, connected to the distributed query module, is used to store and encrypt data in the entire system, thereby providing data support for the storage and computing scheduling module and the distributed query module; The storage and computing scheduling module is connected to the distributed query module and the distributed query module respectively, and is responsible for effective data caching and scheduling the storage of the entire system.

2. The cloud platform-based ship virtual test storage and computing separation system according to claim 1 is characterized in that: The client module includes a user login unit, a first API unit, and a first communication unit; The user login unit is used for user identity authentication and user operation authority authentication; The first API unit is used to provide an API interface to the upper layer and return query results; The first communication unit is used to realize a TCP connection with a distributed query module.

3. The cloud platform-based ship virtual test storage and computing separation system according to claim 2 is characterized in that: The user login unit includes a user authenticator, a rights authenticator, and an identity management unit; The user authenticator is used to read the user's account password to authenticate the user's identity; The authority authenticator is used to further authenticate the user's operation authority; The identity management unit manages user rights in the form of allocating operation rights to user roles. When user rights are changed, the operation can be completed by simply changing the user role.

4. The cloud platform-based ship virtual test storage and computing separation system according to claim 3 is characterized in that: The user identities include common users and administrator users; The ordinary user can only operate in the user interface, and the administrator user can jump to the program interface after authentication and operate through SQL statements; The user's operation permissions include read and write, read only, and write only.

5. The cloud platform-based ship virtual test storage and computing separation system according to claim 1 is characterized in that: The distributed query module includes an SQL statement parsing unit, an optimization unit, a query execution unit, a second communication unit, and a second API unit; The SQL statement parsing unit is used to receive the task sent by the client module through the listening port, and obtain the SQL statement text after processing the task, and parse the SQL statement text according to the SQL grammar rules through the SQL parser to obtain the query plan; The optimization unit is used to optimize the query plan to minimize the execution cost of the query plan; The query execution unit is used to locate the optimized query plan in the distributed database, establish a corresponding connection, and perform data transmission and calculation; The second API unit is used to provide an API interface and return query results to the upper layer, and to provide an API interface and input query plans to the lower layer; The second communication unit is used to realize TCP connection with the distributed storage module and the storage and computing scheduling module.

6. The cloud platform-based ship virtual test storage and computing separation system according to claim 5 is characterized in that: The SQL parser includes a lexical parser and a grammatical parser; The optimization unit includes a physical optimizer and a logical optimizer; The physical optimizer and the logical optimizer provide a logical optimization function and a physical optimization function respectively; The logic optimization function is to use the rules of relational algebra and a series of rules at the logic level to perform lightweight processing on the query plan; The physical optimization function is to select a physical query plan with the minimum execution cost; The query execution unit locates the data shard corresponding to the value according to the key parsed from the SQL statement, establishes a corresponding connection through the second API unit and the second communication unit, and performs data transmission and calculation.

7. The cloud platform-based ship virtual test storage and computing separation system according to claim 1 is characterized in that: The distributed storage module includes a meta-information management unit, a RocksDB storage engine unit, a Raft algorithm unit, a third communication unit, and a third API unit; The meta-information management unit is used to perform cluster management on the distributed storage module, and periodically check the nodes and blocks in the replication cluster to balance data and load; The RocksDB storage engine unit is used to provide the data required by the computing unit and store the computing results; The Raft algorithm unit is used to maintain the consistency of multiple copies in the distributed storage module; The third API unit is used to provide an API interface to the upper layer and return query results, and to provide an API interface to the lower layer and perform data transmission; The third communication unit is used to realize the TCP connection between the distributed query module and the storage and computing scheduling module.

8. The cloud platform-based ship virtual test storage and computing separation system according to claim 1 is characterized in that: The storage and calculation scheduling module includes a computing unit, a cache management unit, a storage scheduling unit, a fourth communication unit, and a fourth API unit; The computing unit is responsible for computing tasks; The cache management unit is used to predict future scheduling tasks, cache data, exchange data, and manage the data used by the computing unit according to the cache strategy to solve the problem caused by frequent communication with the distributed storage module; The storage scheduling unit is used to collect the status information of the nodes and blocks of the cluster in the whole system, and formulate a scheduling strategy for the data in the distributed storage module through the fourth API interface; The fourth API unit is used to provide an API interface to the upper layer and return query results, and to provide an API interface to the lower layer and perform data transmission; The fourth communication unit realizes TCP connection with the distributed storage module and the distributed query unit module.

9. The cloud platform-based ship virtual test storage and computing separation system according to claim 8 is characterized in that: The computer node in the computing unit first accesses the cache management unit to obtain data. If the data does not exist, it initiates a request to the storage scheduling unit to schedule the corresponding data in the distributed storage module. The data exchange operation of the cache management unit includes calling into cache and replacing cold data; The calling into cache is to place the file in a storage area specially used for cache in the computing unit; The cold data replacement is to remove some data in the cache pool through the cache scheduler when the cache pool is full, and to bring in new data; The storage scheduling unit manages the cluster through the storage scheduling node. There are heartbeat packets between the storage nodes. The storage scheduling node is used to check the status information of the storage node and formulate a scheduling strategy.

10. The cloud platform-based ship virtual test storage and computing separation system according to claim 2 is characterized in that: The communication between the first communication unit, the second communication unit, the third communication unit and the fourth communication unit adopts an encrypted transmission mode, and end-to-end encryption is adopted for data. The data is decrypted when the client reads it, and the data is encrypted when the client writes it.

Images