A method and system for collecting, storing and reading multi-source heterogeneous data for supercomputing platforms

Abstract

The present invention discloses a method and system for collecting, storing and reading multi-source heterogeneous data for supercomputing platforms, the method comprising: collecting multi-source heterogeneous data of each supercomputing platform through a unified API interface, and distributing and storing the collected data according to different application scenarios, when a user reads multi-source heterogeneous supercomputing platform data, calling a prefabricated query statement library to construct query statements for different data sources, querying the distributed stored multi-source heterogeneous data and integrating them according to priority, when querying preloaded data, after each query and data integration is completed, dynamically adjusting the priority order of subsequent execution of each query statement and data integration according to the time used for the current round of data query and integration. The present invention collects supercomputing platform data through a unified API interface and stores the data in a distributed manner, which not only reduces the pressure on storage space, but also facilitates reading and integrating required supercomputing data according to application scenarios.

Description

Technical Field

[0001] The present invention relates to the technical field of high-performance computer resource management, and in particular to a method and system for collecting, storing and reading multi-source heterogeneous data for a supercomputing platform. Background Art

[0002] In the process of data transmission and management of supercomputing platforms, due to the needs of different application scenarios and historical reasons such as the long time span of platform construction, different supercomputing platforms often have different physical configurations and software versions. At the same time, each platform has also formulated personalized data collection strategies and uses different data structures to store platform data. However, when it is necessary to use data from multiple supercomputing platforms at the same time, the data differences between different supercomputing platforms bring great difficulties to data transmission and management. At present, two relatively mature solutions have been formed to address the data differences in multi-source heterogeneous data on supercomputing platforms: one is to synchronize heterogeneous data after multiple platforms independently complete data collection, and finally access the synchronized data; the other is to directly integrate the multi-source heterogeneous data of the platform, convert the multi-source heterogeneous data into wide tables in the form of a single user and store them on the big data platform, and finally access the data according to the actual application scenario. However, the existing technology still has the following shortcomings:

[0003] (1) Adopting a data synchronization solution: In the application context of managing multi-source heterogeneous data on supercomputing platforms, since the amount of data on each supercomputing platform is already very large, if a data synchronization solution is adopted, after multiple platforms complete multiple data synchronizations, the occupancy rate of the database storage space of multiple platforms will be multiplied.

[0004] (2) Data synchronization solution: Data synchronization is performed in the form of building tasks and task scheduling thread pool is used for task management. The synchronization types are divided into full synchronization and batch synchronization. The program operation is complex and the learning cost is high.

[0005] (3) Adopting a direct data fusion solution: Converting multi-source heterogeneous data into wide tables for storage in the form of a single user not only increases the storage occupancy of a single storage space, but also requires data extraction again when facing different data application scenarios, operates a large amount of redundant data, increases the operation process, and puts a lot of storage pressure on the database storage space and cache space. Summary of the invention

[0006] In view of the shortcomings of the above-mentioned scheme, the present invention provides a method and system for multi-source heterogeneous data collection, storage and reading for supercomputing platforms. Its main purpose is to solve the problems of large storage pressure on database storage space and cache space and high system learning cost caused by data differences during data transmission and management of supercomputing platforms.

[0007] The present invention discloses a method for collecting, storing and reading multi-source heterogeneous data for a supercomputing platform, which comprises:

[0008] The interfaces of all operation and maintenance processes of the supercomputing platform are encapsulated into a unified API interface. Through the unified API interface, a large amount of multi-source heterogeneous job running time data, node resource utilization, job running logs and other data of each supercomputing platform with different physical configurations and Slurm software versions are uniformly collected, and the collected data are distributedly stored according to different application scenarios. In different application scenarios, the data in the storage space is read according to user needs. When the user reads the multi-source heterogeneous supercomputing platform data, the pre-made query statement library is called to build query statements for different data sources. After querying the distributed stored multi-source heterogeneous data, the read multi-source heterogeneous data is integrated according to the priority. When querying the pre-loaded data, after each query and data integration is completed, the priority order of subsequent execution of each query statement and data integration is dynamically adjusted according to the time used for the current round of data query and integration.

[0009] Furthermore, when the data queried by the user is not preloaded data, the query statements will not be dynamically sorted, but the query statements will be executed in the original order and the data will be integrated; if only one item of data in one of the supercomputing platforms is queried when querying data, no integration is required.

[0010] Furthermore, according to the differences in various physical configurations of supercomputing platforms and Slurm software versions, various data collection strategies available for supercomputing platforms are sorted out. Users can directly or indirectly connect to the corresponding supercomputing platform through the unified interface encapsulated by the data storage and access layer, and collect metadata of the supercomputing platform after selecting the corresponding data collection strategy.

[0011] In different application scenarios, the data processing layer reads relevant data from the corresponding data storage space for relevant processing based on the user's operations on the application side; it can also directly send requests for data collection to the uniformly encapsulated API interface; among which, the relevant processing includes data analysis, data visualization, resource integration and data access.

[0012] Furthermore, the data processing layer uses a multi-level workflow to process the query request information, which includes a request identification layer that identifies the query request information, filters out invalid information by verifying the format and content of the request information, and sends the filtered query request information to the request parsing layer; the request parsing layer converts the query request information into a request body, and constructs a query statement based on the query object; the authentication and interception processing layer authenticates the request information and the query statement, and the request execution layer connects to the storage space based on the authenticated request information, executes the authenticated query statement to read the required supercomputing data and performs data integration on it, and then returns the request execution result.

[0013] Furthermore, constructing the query statement includes:

[0014] A query statement library is prefabricated in advance according to the usage scenario, and a query statement is generated according to the relevant information in the request body combined with the corresponding prefabricated query statement library; wherein the relevant information includes storage space, query type, and query field.

[0015] Furthermore, the authentication and interception processing layer authenticates the request information and query statements, including:

[0016] The authentication and interception processing layer authenticates the key information in the request information and the standardization of the query statement. If the authentication is passed, the request information and query statement will be sent to the request execution layer, otherwise the query instruction will be intercepted; among them, the key information includes the request initiator, the execution node and the connected database type; the standardization of the query statement is checked by an integrated inspection tool.

[0017] Furthermore, for supercomputing data that requires a preloading mechanism, the preloaded data is dynamically queried and integrated according to priority, namely:

[0018] Parse user requests and calculate the weighted scores of each user's request information based on the weights and score sets of each factor; sort the weighted scores of each user's request from high to low to obtain the order of obtaining each preloaded data; then call the multi-level workflow to process the query request according to the priority order and read the required supercomputing data; finally, convert the read data into standard data and benchmark data for data integration;

[0019] The time used for executing data query and data integration in the current round is recorded to adjust the priority order of subsequent execution of each request and realize dynamic sorting of priority queue; the integrated preloaded data will be cached for users to call at any time until all preloaded data has been obtained.

[0020] Furthermore, if the execution time of data query or data integration in the current round exceeds or does not reach a preset time threshold, the user requests in the subsequent priority queue with the same factors as the current round will be deducted or added points on the scores of the corresponding factors.

[0021] Furthermore, different application terminals are established according to different application scenarios, and users issue operation instructions on different application terminals to obtain the results returned by the data processing layer.

[0022] The present invention also discloses a system for collecting, storing and reading multi-source heterogeneous data for supercomputing platforms, which realizes any of the above methods for collecting, storing and reading multi-source heterogeneous data for supercomputing platforms, and includes a data source, a data storage and access layer, a data processing layer and an application layer; the data source includes multiple supercomputing platforms; the data storage and access layer is used to encapsulate the interfaces of all operation and maintenance processes of the supercomputing platform into a unified API interface, and uniformly collect multi-source heterogeneous data of each supercomputing platform with different physical configurations and Slurm software versions in the data source through the unified API interface, and classify the collected data according to different application scenarios. Distributed storage; the data processing layer is used to read data in the storage space according to user needs in different application scenarios. When users read multi-source heterogeneous supercomputing platform data, they call the pre-made query statement library to build query statements for different data sources, query the distributed multi-source heterogeneous data, and integrate the read multi-source heterogeneous data according to priority. When querying pre-loaded data, after each query and data integration is completed, the priority order of subsequent execution of each query statement and data integration is dynamically adjusted according to the time used for the current round of data query and integration; the application layer includes multiple application terminals, which are used by users to obtain the required results returned from the data processing layer.

[0023] Due to the adoption of the above technical solution, the present invention has the following advantages:

[0024] 1. The present invention uses a uniformly encapsulated API interface to collect data, and distributes the collected data according to different application scenarios. There is no need to establish synchronization tasks and perform task scheduling to adapt to the multi-source heterogeneous characteristics of supercomputing platform data. The operation process is simple and easy to learn and maintain.

[0025] 2. The present invention uses a uniformly packaged API interface to collect data, and can collect a large amount of multi-source heterogeneous supercomputing data on various supercomputing platforms with different physical configurations and Slurm software versions.

[0026] 3. The present invention manages the multi-source heterogeneous data of the supercomputing platform in a coordinated manner and accesses it on demand, thereby reducing data redundancy and avoiding the problem of multiple increases in the storage space occupied by each supercomputing platform caused by managing the supercomputing platform in a data synchronization manner.

[0027] 4. The present invention stores the collected data in a distributed manner, avoiding the problem of increased storage pressure on the database and cache space caused by using wide tables to store data, and also facilitates reading data according to different application scenarios.

[0028] 5. The present invention prefabricates a query statement library, which can quickly construct query statements according to user needs, thereby improving the query efficiency of multi-source heterogeneous data. At the same time, the constructed query statements are checked for standardization, thereby improving the effectiveness of the query statements.

[0029] 6. The present invention introduces a data preloading mechanism and can dynamically adjust the priority order of executing query statements and data integration when obtaining preloaded data, thereby improving the efficiency of data loading and enhancing the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the embodiments of the present invention. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0031] Figure 1 A schematic diagram of a system architecture for collecting, storing and reading multi-source heterogeneous data on a supercomputing platform according to an embodiment of the present invention;

[0032] Figure 2 A schematic diagram of data collection and storage according to an embodiment of the present invention;

[0033] Figure 3 A schematic diagram of a multi-level workflow and data integration according to an embodiment of the present invention;

[0034] Figure 4 This is a schematic diagram of dynamic query preloading data according to an embodiment of the present invention. DETAILED DESCRIPTION

[0035] The present invention is further described in conjunction with the accompanying drawings and embodiments, and the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field should fall within the scope of protection of the embodiments of the present invention.

[0036] The present invention provides an embodiment of a method for collecting, storing and reading multi-source heterogeneous data on a supercomputing platform, which includes:

[0037] The interfaces of all operation and maintenance processes of the supercomputing platform are encapsulated into a unified API interface, and the multi-source heterogeneous data of each supercomputing platform with different physical configurations and Slurm software versions are uniformly collected through the unified API interface, and the collected data are distributedly stored according to different application scenarios. In different application scenarios, the data in the storage space is read according to user needs. When the user reads the multi-source heterogeneous supercomputing platform data, the prefabricated query statement library is called to build query statements for different data sources, and the read multi-source heterogeneous data in distributed storage are queried and integrated according to the priority. When querying preloaded data, after each query and data integration are completed, the priority order of subsequent execution of each query statement and data integration is dynamically adjusted according to the time used for the current round of data query and integration. The present invention only performs dynamic priority sorting on the query statements of preloaded data, and the acquisition mechanism of preloaded data is implicit (that is, the user obtains A data in the operation interface, and the system may execute the preloading mechanism to obtain other data in addition to obtaining A data).

[0038] The present invention can unify the data collection strategies on various supercomputing platforms. By using a unified API interface, the differences in physical configuration and Slurm software versions of various supercomputing platforms can be overcome, the data collection process can be elevated to a coordinated level, and unified collection of data from multiple supercomputing platforms can be achieved. The present invention distributes the collected data according to application scenarios, and reads and integrates multi-source heterogeneous data when needed by users. There is no need to synchronize the collected and stored heterogeneous data, thus reducing the storage pressure on database storage space and cache space. The unified API interface is used to collect data from multiple supercomputing platforms (which can be set to timed collection), which is in line with logical cognition and does not require the design of data synchronization strategies and rules, thus simplifying configuration items and reducing the user's learning cost.

[0039] In one embodiment, based on the differences in various physical configurations of supercomputing platforms and Slurm software versions, various data collection strategies available for supercomputing platforms are sorted out. Users can directly or indirectly connect to the corresponding supercomputing platform through a unified interface encapsulated by the data storage and access layer, select the corresponding data collection strategy, and then collect the metadata of the supercomputing platform.

[0040] In one embodiment, in different application scenarios, the data processing layer reads relevant data from the corresponding data storage space for relevant processing based on the user's operations on the application side; it can also directly send a request for collecting data to a uniformly packaged API interface; wherein the relevant processing includes data analysis, data visualization, resource integration and data access.

[0041] In one embodiment, when the data queried by the user is not preloaded data, the query statement will not be dynamically sorted, but the query statement will be executed and the data will be integrated in the original order; if only one item of data in one of the supercomputing platforms is queried when querying data, no integration is required.

[0042] In one embodiment, the data processing layer uses a multi-level workflow to process the query request information, which includes a request identification layer to identify the query request information, filter out invalid information by verifying the format and content of the request information, and send the filtered query request information to the request parsing layer; the request parsing layer converts the query request information into a request body, and constructs a query statement based on the query object; the authentication and interception processing layer authenticates the request information and the query statement, and the request execution layer connects to the storage space based on the authenticated request information, executes the authenticated query statement to read the required supercomputing data and performs data integration, and then returns the request execution result.

[0043] In one embodiment, constructing a query statement includes:

[0044] A query statement library is prefabricated in advance according to the usage scenario, and a query statement is generated according to the relevant information in the request body combined with the corresponding prefabricated query statement library; wherein the relevant information includes storage space, query type, and query field.

[0045] In one embodiment, the authentication and interception processing layer authenticates the request information and query statements, including:

[0046] The authentication and interception processing layer authenticates the key information in the request information and the standardization of the query statement. If the authentication is passed, the request information and query statement will be sent to the request execution layer, otherwise the query instruction will be intercepted; among them, the key information includes the request initiator, the execution node and the connected database type; the standardization of the query statement is checked by an integrated inspection tool.

[0047] In one embodiment, for supercomputing data that needs to implement a preloading mechanism, the preloaded data is dynamically queried and integrated according to priority, that is:

[0048] Parse user requests and calculate the weighted scores of each user's request information based on the weights and score sets of each factor; sort the weighted scores of each user's request from high to low to obtain the order of obtaining each preloaded data; then call the multi-level workflow to process the query request according to the priority order and read the required supercomputing data; finally, convert the read data into standard data and benchmark data for data integration;

[0049] The time used for executing data query and data integration in the current round is recorded to adjust the priority order of subsequent execution of each request and realize dynamic sorting of priority queue; the integrated preloaded data will be cached for users to call at any time until all preloaded data has been obtained.

[0050] In one embodiment, if the execution time of data query or data integration in the current round exceeds or does not reach a preset time threshold, user requests in the subsequent priority queue that have the same factors as the current round will be deducted or added to the scores of the corresponding factors.

[0051] In one embodiment, different application terminals are established according to different application scenarios, and the user issues operation instructions on different application terminals to obtain the results returned by the data processing layer.

[0052] See also Figure 1 The present invention also provides an embodiment of a system for collecting, storing and reading multi-source heterogeneous data for a supercomputing platform, which implements a method for collecting, storing and reading multi-source heterogeneous data for a supercomputing platform as described in the above embodiment, and includes a data source, a data storage and access layer, a data processing layer and an application layer; the data source includes multiple supercomputing platforms; the data storage and access layer is used to encapsulate the interfaces of all operation and maintenance processes of the supercomputing platform into a unified API interface, and through the unified API interface, a large amount of multi-source heterogeneous job running time data, node resource utilization, job running logs and other data of each supercomputing platform with different physical configurations and Slurm software versions in the data source are uniformly collected, and according to The collected data is distributedly stored in different application scenarios; the data processing layer is used to read data in the storage space according to user needs in different application scenarios. When users read multi-source heterogeneous supercomputing platform data, they call the pre-made query statement library to build query statements for different data sources. After querying the distributed stored multi-source heterogeneous data, the read multi-source heterogeneous data is integrated according to priority. When querying pre-loaded data, after each query and data integration is completed, the priority order of subsequent execution of each query statement and data integration is dynamically adjusted according to the time used for the current round of data query and integration; the application layer includes multiple application terminals, which are used by users to obtain the required results returned from the data processing layer.

[0053] For ease of understanding, the present invention also provides the following more specific embodiments:

[0054] In one embodiment, according to the differences in the various physical configurations of the supercomputing platform and the Slurm software versions, various data collection strategies available on the supercomputing platform are sorted out. Users can directly or indirectly connect to the corresponding supercomputing platform through the unified interface encapsulated by the data storage and access layer, and select the corresponding data collection strategy to collect the metadata of the supercomputing platform. The data collection process diagram is shown in Figure 1. Figure 2 shown.

[0055] In one embodiment, the data storage and access layer encapsulates the interfaces of all operation and maintenance processes of the supercomputing platform into a unified RESTFul style API interface, so as to collect heterogeneous data of different supercomputing platforms with a unified API interface, and distribute the collected metadata of the supercomputing platform according to different application scenarios. For example, the annual budget machine time data and annual machine time data are stored in a MySQL database due to the need for structured storage and complex queries; based on the need for fast retrieval and statistical analysis of the used machine time data generated by different supercomputing platforms, Elasticsearch with distributed architecture and aggregate analysis features is selected for storage; system operation logs are stored through a dedicated log storage server; Swagger stores description files related to the interface, and updates the interface documents by maintaining the description files.

[0056] In one embodiment, in different application scenarios, the data processing layer reads relevant data from the corresponding data storage space to perform data analysis, data visualization, resource integration, etc. according to the user's operations on the application side, and can also directly access the uniformly encapsulated API interface to send requests for data collection.

[0057] A multi-level workflow is used to process the query request information, which includes a request identification layer, a request parsing layer, an authentication and interception processing layer, and a request execution layer. The request identification layer identifies the query request information and filters invalid information by verifying the format and content of the request information. The request parsing layer converts the request information into a request body and constructs a query statement based on the query object (the construction of the query statement can be based on the use scenario. The query statement is generated in advance based on the storage space, query type, query field and other information in the request body combined with the corresponding prefabricated query statement library). The authentication and interception processing layer authenticates the request information and query statement. The request initiator, execution node, connected database type and the standardization of the query statement in the request information must be authenticated (the standardization of the query statement is checked by an integrated inspection tool, such as MySQL statements can be checked by integrating the MySQL plug-in of SonarQube), otherwise the query instruction will be intercepted. The request execution layer can connect to the storage space based on the request information, execute the query statement to read the required supercomputing data and return the query results (for multi-source heterogeneous supercomputing data, data integration is required after reading the supercomputing data). The schematic diagram of multi-level workflow processing query request and data integration is as follows Figure 3 shown.

[0058] In one embodiment, a method for dynamically querying and integrating preloaded data according to priority is provided. This method is only effective for supercomputing data in the system that requires a preload mechanism. First, expert scoring and hierarchical analysis are used to pre-set the scoring criteria and weight sets of various factors such as the display level of the data on the application side, the query node status (available, unavailable), the query data level, and the complexity of the data processing program; then, the request information of each preloaded data is parsed and obtained, and the weighted score of each request information is calculated based on the weights of the above factors and the scoring set. The mathematical expression can be expressed as:

[0059]

[0060] Sort the weighted scores of each request from high to low to obtain the order of obtaining each preloaded data (the order of executing query statements and performing data integration); then call the multi-level workflow to process the query information and read the required supercomputing data according to the priority order; finally, convert the read data into standard data and benchmark data for data integration. Considering that the multi-source heterogeneous characteristics of supercomputing data have a great impact on the execution time of the data query and data integration process, the present invention records the time used to execute data query and data integration in the current round, which is used to adjust the priority order of subsequent executions of each request and realize dynamic sorting of priority queues (such as: if the current round of data query or data integration exceeds / is less than the preset time threshold, the user requests with the same factors as the current round in the subsequent priority queue will be deducted / added points in the scores of the corresponding factors). The integrated preloaded data will be cached for users to call at any time. The schematic diagram of dynamically querying preloaded data according to priority is shown in the figure. Figure 4 shown.

[0061] In one embodiment, different application terminals (such as visualization system, resource management system, operation and maintenance platform, etc.) are established according to different application scenarios, with the characteristics of multiple terminal calls of one set of data, flexible customization of application terminals is realized, and data display scenarios are enriched. Users can issue operation instructions on different application terminals and obtain specific results returned by the data processing layer.

[0062] It should be noted that the above embodiments can refer to each other.

[0063] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A method for collecting, storing and reading multi-source heterogeneous data for a supercomputing platform, characterized in that: include: The interfaces of all operation and maintenance processes of the supercomputing platform are encapsulated into a unified API interface. Through the unified API interface, multi-source heterogeneous data of various supercomputing platforms with different physical configurations and Slurm software versions are uniformly collected, and the collected data is distributedly stored according to different application scenarios. In different application scenarios, the data in the storage space is read according to user needs. When the user reads the multi-source heterogeneous supercomputing platform data, the pre-made query statement library is called to build query statements for different data sources. After querying the distributed stored multi-source heterogeneous data, the read multi-source heterogeneous data is integrated according to the priority. When querying pre-loaded data, after each query and data integration is completed, the priority order of subsequent execution of each query statement and data integration is dynamically adjusted according to the time used for the current round of data query and integration. For supercomputing data that requires a preloading mechanism, dynamically query and integrate preloaded data according to priority, namely: Parse user requests and calculate the weighted scores of each user's request information based on the weights and score sets of each factor; sort the weighted scores of each user's request from high to low to obtain the order in which each preloaded data is obtained; Then, the multi-level workflow is called to process the query request according to the priority order to read the required supercomputing data; finally, the read data is converted into standard data and benchmark data for data integration; The time used for executing data query and data integration in the current round is recorded to adjust the priority order of subsequent execution of each request and realize dynamic sorting of priority queue; the integrated preloaded data will be cached for users to call at any time until all preloaded data has been obtained.

2. The method for collecting, storing and reading multi-source heterogeneous data for a supercomputing platform according to claim 1, characterized in that: When the data queried by the user is not preloaded data, the query statement will not be dynamically sorted, but the query statement will be executed in the original order and the data will be integrated; if only one item of data in one of the supercomputing platforms is queried when querying data, no integration is required.

3. The method for collecting, storing and reading multi-source heterogeneous data for a supercomputing platform according to claim 1, characterized in that: According to the differences in the various physical configurations of supercomputing platforms and Slurm software versions, various data collection strategies available for supercomputing platforms are sorted out. Users can directly or indirectly connect to the corresponding supercomputing platform through the unified interface encapsulated by the data storage and access layer, and select the corresponding data collection strategy to collect the metadata of the supercomputing platform. In different application scenarios, the data processing layer reads relevant data from the corresponding data storage space and performs relevant processing according to the user's operation on the application side; You can also directly send a request to collect data to the uniformly packaged API interface; the related processing includes data analysis, data visualization, resource integration and data access.

4. The method for collecting, storing and reading multi-source heterogeneous data for a supercomputing platform according to claim 1, characterized in that: The data processing layer uses a multi-level workflow to process the query request information, including the request identification layer to identify the query request information, filter out invalid information by verifying the format and content of the request information, and send the filtered query request information to the request parsing layer; the request parsing layer converts the query request information into a request body and constructs a query statement based on the query object; The authentication and interception processing layer authenticates the request information and query statements. The request execution layer connects to the storage space based on the authenticated request information, executes the authenticated query statements to read the required supercomputing data, integrates the data, and then returns the request execution result.

5. The method for collecting, storing and reading multi-source heterogeneous data for a supercomputing platform according to claim 4 is characterized in that: The constructing of the query statement comprises: A query statement library is prefabricated in advance according to the usage scenario, and a query statement is generated according to the relevant information in the request body combined with the corresponding prefabricated query statement library; wherein the relevant information includes storage space, query type, and query field.

6. The method for collecting, storing and reading multi-source heterogeneous data for a supercomputing platform according to claim 4, characterized in that: The authentication and interception processing layer authenticates the request information and query statements, including: The authentication and interception processing layer authenticates the key information in the request information and the standardization of the query statement. If the authentication is passed, the request information and query statement will be sent to the request execution layer, otherwise the query instruction will be intercepted; among them, the key information includes the request initiator, the execution node and the connected database type; the standardization of the query statement is checked by an integrated inspection tool.

7. The method for collecting, storing and reading multi-source heterogeneous data for a supercomputing platform according to claim 1, characterized in that: If the execution time of data query or data integration in the current round exceeds or does not reach the preset time threshold, the scores of the corresponding factors of user requests in the subsequent priority queue with the same factors as the current round will be deducted or added.

8. The method for collecting, storing and reading multi-source heterogeneous data for a supercomputing platform according to claim 1, characterized in that: Different application terminals are established according to different application scenarios. Users issue operation instructions on different application terminals and obtain the results returned by the data processing layer.

9. A system for collecting, storing and reading multi-source heterogeneous data for a supercomputing platform, which implements the method for collecting, storing and reading multi-source heterogeneous data for a supercomputing platform as described in any one of claims 1 to 8, characterized in that: It includes data source, data storage and access layer, data processing layer and application layer; data source includes multiple supercomputing platforms; The data storage and access layer is used to encapsulate the interfaces of all operation and maintenance processes of the supercomputing platform into a unified API interface, and through the unified API interface, uniformly collect multi-source heterogeneous data from various supercomputing platforms with different physical configurations and Slurm software versions in the data source, and distribute the collected data according to different application scenarios; the data processing layer is used to read data in the storage space according to user needs in different application scenarios. When the user reads multi-source heterogeneous supercomputing platform data, the pre-made query statement library is called to build query statements for different data sources. After querying the distributed stored multi-source heterogeneous data, the read multi-source heterogeneous data is integrated according to the priority. When querying pre-loaded data, after each query and data integration is completed, the priority order of subsequent execution of each query statement and data integration is dynamically adjusted according to the time used for the current round of data query and integration; the application layer includes multiple application terminals, which are used by users to obtain the required results returned from the data processing layer.

Images