Resource node device and online scientific research platform for astronomical big data

Abstract

The invention provides an astronomical big data-oriented resource node device and an online scientific research platform, which are used in the technical field of astronomical science computers. The resource node device corresponds to a single astronomical data center, a shared computing resource area of the resource node device is used for storing mirror data of a plurality of astronomical application programs, and a shared data resource area is used for mounting astronomical data of the astronomical data center in a read-only mode; the private resource area is used for storing calculation results; the resource node device is configured to determine a target astronomical application program from the shared computing resource area and determine target astronomical data from the shared data resource area in response to a data computing request received from the management server; calling a target astronomical application program to calculate the target astronomical data to obtain a target calculation result; and storing the target calculation result to a root directory matched with the target user initiating the data request in the private resource area, and returning the target calculation result to the management server.

Description

Technical Field

[0001] The present disclosure relates to the fields of astronomical science and technology, computer technology, and specifically to the field of astronomical data processing technology, and in particular to an online scientific research platform for astronomical big data. Background Art

[0002] Astronomical large scientific facilities are the main source of astronomical data. With the rapid development of manufacturing technology and processes, the astronomical scientific data obtained by astronomical large scientific facilities has increased dramatically. For example, after multiple astronomical telescopes at home and abroad were put into operation, astronomical data grew rapidly. The data scale of the current mainstream telescopes is in the range of 10TB~150PB, and the data scale of the Large Synoptic Survey Telescope (LSST) has even reached about 150PB. In 20 years, astronomical data has entered the PB era from the TB era, and will soon reach the EB era. The explosive growth of data poses great challenges in many aspects such as data storage, computing, and transmission.

[0003] Faced with large-scale astronomical data, the scientific research efficiency of the traditional scientific model (such as downloading data-analyzing data) is low. For example, due to the limitation of the processing power of traditional software, only a small range of data can be selected for scientific research, and the value of big data cannot be truly realized. Although many large astronomical scientific facilities have established large-scale data storage based on their own astronomical data centers, the download of large-scale astronomical data takes a lot of time due to network limitations. In addition, the storage and calculation of large-scale astronomical data require a lot of resources, and it is difficult for a single astronomer to build a large-scale data processing environment. Especially for the transmission of TB or even PB-level data, the high time transmission cost seriously affects the scientific output of large astronomical scientific facilities.

[0004] In summary, the problems of data calculation and storage faced in the big data era have resulted in the inability to quickly carry out data-based scientific research, thus affecting the output of results from large scientific facilities. Summary of the invention

[0005] In view of this, the present disclosure provides a resource node device and an online scientific research platform for astronomical big data.

[0006] One aspect of the present disclosure provides a resource node device for astronomical big data, characterized in that the resource node device corresponds to a single astronomical data center, the resource node device includes a private resource area and a shared resource area, the shared resource area includes a shared data resource area and a shared computing resource area, the shared computing resource area is used to store mirror data of multiple astronomical applications, and the shared data resource area is used to mount the astronomical data of the astronomical data center in read-only mode; the private resource area is used to store the calculation results of each user calling the astronomical application to calculate the astronomical data of the astronomical data center; the resource node device is configured to: in response to receiving a data calculation request from a management server, determine a target astronomical application matching the data calculation request from the shared computing resource area, determine the target astronomical data to be processed from the shared data resource area; call the target astronomical application to calculate the target astronomical data to obtain a target calculation result; store the target calculation result in a root directory matching the target user who initiated the data request in the private resource area, and return the target calculation result to the management server.

[0007] According to an embodiment of the present disclosure, the resource node device is also configured to: in response to receiving a resource query request from a management server, obtain resource information of the resource node device, wherein the resource information includes at least one of the following: resource node device status, available resources, scheduling type, and available astronomical applications; and return the resource information to the management server.

[0008] According to an embodiment of the present disclosure, the resource node device is also configured to: in response to receiving a calculation result query request from a management server, determine a query user based on the calculation result query request, wherein the calculation result query request is for a target calculation result; and when it is determined that the query user has query authority for the target calculation result, obtain the target calculation result from a private resource area, and return the target calculation result to the management server, so that the management server returns the target calculation result to the query user.

[0009] According to an embodiment of the present disclosure, a resource node device manages a shared resource area and a private resource area of ​​the resource node device through a virtual machine computing cluster, a containerized cluster and / or a job scheduling system.

[0010] Another aspect of the present disclosure provides an online scientific research platform for astronomical big data, including: a unified authentication system, a self-service system, a management server and a plurality of the above-mentioned resource node devices; the unified authentication system is configured to perform identity authentication on users logging into the self-service system; the self-service system is configured to display a front-end page so that users who have passed the identity authentication can submit data requests based on the front-end page, the data requests including data calculation requests, which are used to call astronomical applications to calculate astronomical data; the management server is used to receive the data calculation requests and forward the data calculation requests to the corresponding resource node devices; and forward the target calculation results returned by the resource node devices to the self-service system.

[0011] According to an embodiment of the present disclosure, the management server is also configured to: receive multiple data calculation requests from the self-service system in batches, and forward each of the multiple data calculation requests to a resource node device corresponding to each calculation request, so that multiple resource node devices call their respective astronomical applications in parallel to calculate their respective astronomical data according to their respective data calculation requests.

[0012] According to an embodiment of the present disclosure, a data request includes a data import request, which is used to import new astronomical data; the management server is also configured to: receive a data import request from a self-service system; determine a target resource node device corresponding to the astronomical data from a plurality of resource node devices according to the data import request; and mount the new astronomical data in a shared computing resource area of ​​the target resource node device in read-only mode, so that a user can obtain the new astronomical data online through a data request.

[0013] According to an embodiment of the present disclosure, a data request includes a resource node device creation request, and the resource node device creation request is used to connect a new astronomical data center to an online scientific research platform; the management server is further configured to: receive a resource node device creation request from a self-service system; create a new resource node device corresponding to the new astronomical data center according to the resource node device creation request; mount the astronomical data of the new astronomical data center to a shared data resource area of ​​the new resource node device in read-only mode; and store mirror data of multiple astronomical applications for processing the astronomical data of the new astronomical data center in the shared computing resource area of ​​the new resource node device, so that users can call the astronomical application to calculate the astronomical data of the new astronomical data center through data calculation requests.

[0014] According to an embodiment of the present disclosure, multiple resource node devices correspond to astronomical data centers located in multiple geographical locations respectively; the platform also includes a network system, the network system includes multiple virtual private networks between the management server and the multiple resource node devices, so that the management server interacts with the resource node devices through the virtual private networks between each resource node device; the data request also includes an application access request, and the network system also includes an application gateway, which is used to receive the application access request from the self-service system and access the target astronomical application in the resource node device corresponding to the application access request according to the application access request.

[0015] According to an embodiment of the present disclosure, the management server includes a primary server and a secondary server, and the secondary server is configured to asynchronously back up data from the primary server.

[0016] In the embodiment of the present disclosure, the online scientific platform constructed with the astronomical data stored in the resource node device as the center can realize the integration of computing and astronomical data at the resource node device, avoiding the large-scale transmission of astronomical data. The online scientific platform can integrate the heterogeneous astronomical data centers across regions downward through the resource node device, such as the data resources, storage resources, computing resources, etc. of the astronomical data center; the online scientific platform can also provide users (astronomical scientists) with interfaces for accessing and running scientific applications upward. Therefore, through the online scientific platform, astronomers do not need to spend a lot of time downloading data and perform calculations on a small part of the data locally, nor do they need to build and operate information facilities such as data downloading, computing, and storage locally. In the embodiment of the present disclosure, the corresponding application environment can be built through the scientific platform. By using the online scientific research platform, the storage resources and computing resources of the astronomical data center where the astronomical data is located can be used to perform online astronomical data calculations, which reduces the time for downloading TB / PB / TB-level astronomical data and greatly improves data processing efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The above and other objects, features and advantages of the present disclosure will become more apparent through the following description of the embodiments of the present disclosure with reference to the accompanying drawings, in which:

[0018] Figure 1 The flowchart schematically shows the process of a resource node device executing a computing task according to an embodiment of the present disclosure.

[0019] Figure 2 The system architecture of the resource node device according to the embodiment of the present disclosure is schematically shown.

[0020] Figure 3 The application scenarios of the resource node device and online scientific research platform for astronomical big data disclosed in the present invention are schematically illustrated.

[0021] Figure 4 The system architecture of the online scientific research platform for astronomical big data according to an embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0022] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present disclosure. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of the embodiments of the present disclosure. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of the concepts of the present disclosure.

[0023] The terms used herein are only for describing specific embodiments and are not intended to limit the present disclosure. The terms "comprise", "include", etc. used herein indicate the existence of the features, steps, operations and / or components, but do not exclude the existence or addition of one or more other features, steps, operations or components.

[0024] All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0025] When using expressions such as "at least one of A, B, and C, etc.", they should generally be interpreted according to the meaning of the expression commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0026] In the embodiments of the present disclosure, the collection, updating, analysis, processing, use, transmission, provision, disclosure, storage, etc. of the data involved (for example, including but not limited to user personal information) are in compliance with the provisions of relevant laws and regulations, are used for legitimate purposes, and do not violate public order and good morals. In particular, necessary measures are taken for user personal information to prevent illegal access to user personal information data and maintain the security of user personal information, network security, and national security.

[0027] In the embodiments of the present disclosure, the user's authorization or consent is obtained before obtaining or collecting the user's personal information.

[0028] Faced with a large amount of astronomical data and a scenario where large astronomical scientific facilities for acquiring astronomical data are located in multiple locations, in order to avoid the migration of large-scale astronomical data, the embodiments of the present disclosure provide a resource node device and an online scientific research platform for astronomical big data. The resource node device built with data as the center deploys large-scale, shareable storage and computing resources; the online scientific research platform built based on the resource node device provides astronomers with an online accessible and usable scientific data analysis and processing environment.

[0029] Figure 1 The following is a schematic diagram showing a flow chart of a resource node device executing a computing task according to an embodiment of the present disclosure. Figure 1 As described above, embodiment 100 includes operations S110 to S130.

[0030] In operation S110, in response to receiving a data computing request from a management server, a target astronomical application matching the data computing request is determined from within a shared computing resource region, and target astronomical data to be processed is determined from within the shared data resource region;

[0031] In operation S120, a target astronomical application is called to calculate the target astronomical data to obtain a target calculation result;

[0032] In operation S130, the target calculation result is stored in a root directory in the private resource area that matches the target user who initiated the data request, and the target calculation result is returned to the management server.

[0033] According to an embodiment of the present disclosure, a resource node device corresponds to a single astronomical data center, and the resource node device includes a private resource area and a shared resource area. The shared resource area includes a shared data resource area and a shared computing resource area. The shared computing resource area is used to store mirror data of multiple astronomical applications, and the shared data resource area is used to mount the astronomical data of the astronomical data center in read-only mode; the private resource area is used to store the calculation results of each user calling the astronomical application to calculate the astronomical data of the astronomical data center.

[0034] It can be understood that the resource node device is built with data as the center. Within a single resource node device, task calculations for a certain astronomical data can be implemented without the user having to download astronomical applications or astronomical data on the local computing device, nor is there a need to use local astronomical applications to calculate local astronomical data.

[0035] Specifically, the management server may forward the data computing request initiated by the user to the resource node device, and the resource node device may determine the target astronomical application program to be called and the target astronomical data to be used for this computing task according to the data computing request.

[0036] In the data-centric resource node device, the resource node device can build the operating environment of the target astronomical application through the mirror data of the target astronomical application. In this operating environment, the resource node device can read the mounted target astronomical data and call the target astronomical application to perform various computing tasks on the target astronomical data to obtain the target computing results. The computing tasks can be spectral preprocessing, spectral classification, etc. in the astronomical field.

[0037] It should be noted that, since the target astronomical data is mounted to the resource node device in read-only mode, the user can only read the astronomical data. Therefore, after obtaining the target calculation result, it is supported to store the target calculation result in the resource node device.

[0038] In the embodiments of the present disclosure, a resource node device is constructed around the storage, processing and calculation of data with astronomical data as the center, so that the integration of calculation and astronomical data can be realized inside the resource node device, avoiding the large-scale transmission of astronomical data, reducing the time for downloading TB / PB / TB-level astronomical data, and greatly improving the data processing efficiency. In addition, the resource node device downwardly integrates cross-regional heterogeneous astronomical data centers, such as data resources, storage resources, and computing resources of astronomical data centers; upwardly, it can provide users (astronomical scientists) with interfaces for accessing and running scientific applications through the management server. As a result, through the resource node device, astronomers do not need to spend a lot of time downloading data and calculating a small part of the data locally, nor do they need to build and operate information facilities such as data downloading, calculation, and storage locally.

[0039] According to an embodiment of the present disclosure, the resource node device is also configured to: in response to receiving a resource query request from a management server, obtain resource information of the resource node device, wherein the resource information includes at least one of the following: resource node device status, available resources, scheduling type, and available astronomical applications; and return the resource information to the management server.

[0040] In an embodiment of the present disclosure, a data request may include a resource query request, and the management server may query the resource node device status, available resources, scheduling type, and available astronomical applications of a resource node device according to the resource query request. For example, the resource node device status may include whether it is available, and the available resources include: whether a certain stored astronomical data is available, whether the astronomical data collected by a certain astronomical large scientific device is available. The scheduling type may correspond to the type of server cluster used by the resource node device, such as whether the scheduling is a virtual machine, whether the scheduling is a container cluster, and whether the scheduling is a high-performance computing cluster.

[0041] For example, the management server uses a traditional database system to store real-time resource information of resource node devices.

[0042] According to an embodiment of the present disclosure, the resource node device is also configured to: in response to receiving a calculation result query request from a management server, determine a query user based on the calculation result query request, wherein the calculation result query request is for a target calculation result; and when it is determined that the query user has query authority for the target calculation result, obtain the target calculation result from a private resource area, and return the target calculation result to the management server, so that the management server returns the target calculation result to the query user.

[0043] According to an embodiment of the present disclosure, the resource node device mounts the astronomical data in read-only mode, but the calculation results are stored in the private resource area of ​​the resource node device. In the private resource area, a private root directory can be created for each user so that the user can query the calculated results at any time.

[0044] Because resources in the private resource area are private, only users who have permission to query a certain calculation result can query the calculation result.

[0045] For example, when a query user wants to query a previously stored target calculation result, the query user can initiate a calculation result query request, and the management server forwards the calculation result query request to the resource node device storing the target calculation result. The resource node device determines the query user who needs to query the target calculation result based on the received calculation result query request. If it is determined that the query user has the query permission for the target calculation result, the resource node device obtains the target calculation result from the root directory of the query user in the private resource area, and returns the target calculation result to the management server, so that the management server returns the target calculation result to the query user. If it is determined that the query user does not have the query permission for the target calculation result, a prompt message such as "no permission" can be fed back.

[0046] When a user logs in to an online scientific platform including a resource node device for the first time, if the user has not created a user space in the computing area of ​​a resource node device, the user space will be automatically created; if it has been created, the existing user space will be automatically bound so that the calculation results corresponding to the user can be stored in the private resource area of ​​a resource node device through the root directory.

[0047] According to an embodiment of the present disclosure, a resource node device manages a shared resource area and a private resource area of ​​the resource node device through a virtual machine computing cluster, a containerized cluster and / or a job scheduling system.

[0048] In the embodiments of the present disclosure, the computing resources inside the resource node device may be of multiple different resource types. The present disclosure supports the integration of three different types of computing resources:

[0049] The first is virtualized resources. Physical servers can be virtualized into virtual machines on demand through virtualization technology (Kernel-based Virtual Machine, KVM). Virtual machines are dynamically created from deployed image templates. The corresponding astronomical applications have been installed in the image. All images in the image library support dynamic addition and modification. At the same time, multiple virtual machines can also be deployed into a virtual machine computing cluster. For example, CloudStack is used as a cluster server and virtual machine management system to support the creation of a single virtual machine or batches of virtual machines. The virtual machine computing cluster pre-deploys the image template for virtual machine creation and supports image updates, additions, and customization.

[0050] The second type is a containerized cluster, which can dynamically create and manage containers (Docker) based on the Kubernetes (K8S) architecture. The containerized cluster needs to configure an image library, and the container image can be updated, added, and customized. Similarly, containers can be created individually or in batches.

[0051] The third type is a high-performance computing cluster. The cluster uses Slurm as the job scheduling system, and the required astronomical applications are deployed in advance on the job scheduling system.

[0052] Agent software can be deployed on the resource node device, and will automatically connect to the management server after startup. In the present disclosure, different agent software can be developed for different types of computing clusters that manage shared computing resource areas to implement resource information aggregation and job allocation interfaces. The agent software is deployed on the management node of the cluster.

[0053] In the embodiments of the present disclosure, a resource set of computing, storage and data that can independently accept computing tasks and belongs to the same astronomical data center is called a resource node device. Figure 2 shown.

[0054] Figure 2 The system architecture of the resource node device according to the embodiment of the present disclosure is schematically shown.

[0055] like Figure 2 As shown, from bottom to top, the resource node device includes a private resource area and a shared resource area. The private resource area can store user private resources (User private storage), such as calculation results; the shared resource area can store shared astronomical data and shared astronomical applications.

[0056] In the resource node device, the management of private resource areas and shared resource areas can be achieved through the interaction with the containerized cluster, virtual machine computing cluster, and job scheduling system that manage computing resources in the resource node device through a high-speed network.

[0057] For example, the container cluster can be an open source Container cluster, the virtual machine computing cluster can use the open source Cloudstack private cloud ECS to manage virtual machines, and the job scheduling system can be implemented based on the open source Slurm High Performance Comput (HPC) technology. The job scheduling system can implement job scheduling through virtualization technology or agents.

[0058] The dashed box is a collection of storage resources and computing resources of the resource node device. On top of the storage resources and computing resources, the astronomical application can call the computing resources through the computing server to implement various computing tasks. For example, the astronomical application can be jupyterlab, etc., or it can be based on RDP / SSH, or remote control tools (Virtual Network Console, VNC) and other remote control methods to call the astronomical application in the resource node device.

[0059] In addition, users can submit various data calculation requests through the resource self-services provided by the resource node device, or submit various data requests through the data analysis services to achieve data search, cross-match, visualization requirements, etc.

[0060] The resource dispatch system can allocate various computing resources for resource self-service and data analysis services within the resource node device to realize various computing tasks based on computing resources.

[0061] In addition, the resource node device can manage the user's private resources in the private resource area based on the unified authentication system, and create resource management views (Resource management views) of the resource node device based on the management server and display them to the user.

[0062] Figure 3 The application scenarios of the resource node device and online scientific research platform for astronomical big data disclosed in the present invention are schematically shown. Figure 3As shown, in embodiment 300, the online scientific research platform for astronomical big data includes a self-service system 301, a management server 302, and a plurality of resource node devices, such as resource node device 303-1, resource node device 303-2, etc. In addition, the online scientific research platform also includes a unified authentication system 305, which provides a unified identity authentication service for the computing services in the self-service system 301, the management server 302, and the resource node devices.

[0063] For users who log in to the online scientific research platform, when they log in to the self-service system, the unified authentication system 305 can authenticate the user's identity. If the identity authentication is passed, the user can log in to the self-service system and use the self-service system to query, calculate, and store astronomical data.

[0064] The self-service system 301 is configured to display a front-end page so that the user can submit a data request based on the front-end page. For example, after logging into the online scientific research platform, the user can interact through the front-end page displayed by the self-service system 301, such as submitting a data request through interactive operations.

[0065] According to an embodiment of the present disclosure, the data request includes a data calculation request, and the data calculation request is used to call an astronomical application to calculate astronomical data. For example, an astronomical scientist can, as a user, initiate a data calculation request through an online scientific research platform.

[0066] The management server 302 is connected to the self-service system via a network, and is configured to receive data requests transmitted by the self-service system 301 and forward the data requests to the corresponding resource node devices; and forward resource information returned by the resource node devices to the self-service system, wherein the resource information includes astronomical data and / or calculation results stored in the resource node devices.

[0067] For example, for a data calculation request, the management server 302 can forward the data calculation request to the corresponding resource node device so as to use the astronomical application of the corresponding resource node device to calculate the corresponding astronomical data. In addition, after the corresponding resource node device completes the calculation, the management server 302 can interact with the corresponding resource node device, forward the calculation result returned by the resource node device to the self-service system 301, and display it to the user. Thus, the calculation of astronomical data can be performed online and the calculation result can be obtained without downloading the astronomical data to the local and using the local astronomical application to calculate. Compared with the traditional scientific model, the occupation of local storage and computing resources is reduced, and the use of astronomical data is more convenient.

[0068] Alternatively, for a resource query request, the management server 302 may determine the corresponding resource node device to be queried based on the resource query request, and forward the resource query request to the corresponding resource node device, so as to obtain the astronomical data or other resources of the corresponding resource node device. Afterwards, the management server 302 may forward the astronomical data or other resources obtained from the corresponding resource node device to the self-service system, so as to query the astronomical data online.

[0069] Resource node devices, such as resource node device 303-1, resource node device 303-2, ... correspond to a single astronomical data center. For example, resource node device 303-1 corresponds to the astronomical data center located in city 1 304-1, resource node device 303-2 corresponds to the astronomical data center located in city 2 204-2, ...

[0070] like Figure 3 As shown, for resource node device 303-1, resource node device 303-1 includes private resource area 1, shared computing resource area 1 and shared data resource area 1. For resource node device 303-2, resource node device 303-2 includes private resource area 2, shared computing resource area 2 and shared data resource area 2.

[0071] For example, city 1 is equipped with an astronomical large scientific facility 1, and the astronomical data center located in city 1 can store the astronomical data collected by the astronomical large scientific facility 1. Therefore, the shared data resource area 1 corresponding to city 1 can store the astronomical data. Similarly, the shared data resource area 2 corresponding to city 2 can store the astronomical data collected by the astronomical large scientific facility 2 located in city 2.

[0072] The shared computing resource area is used to store the mirror data of multiple astronomical applications. For example, for the shared computing resource area 1, the mirror data of multiple astronomical applications that can process the astronomical data collected by the astronomical large scientific facility 1 can be stored. Taking the astronomical large scientific facilities 1 and 2 as examples, the astronomical applications stored in the shared computing resource area 1 and the shared computing resource area 2 can be the same or different, so that different astronomical applications can be used to perform different calculations on different astronomical data, or the same astronomical application can be used to perform the same calculations on different astronomical data.

[0073] Similar to the shared computing resource area, the private resource area can be the calculation results stored by multiple users after performing the same / different calculations on the astronomical data stored by the resource node device. Specifically, the private resource area of ​​each resource node device only stores the calculation results after calculating the astronomical data on the resource node device. It can be understood that the private resource areas of different resource node devices can store the calculation results of the same user after calculating different astronomical data.

[0074] In the embodiment of the present disclosure, the online scientific platform constructed with the astronomical data stored in the resource node device as the center can realize the integration of computing and astronomical data at the resource node device, avoiding the large-scale transmission of astronomical data. The online scientific platform can integrate the heterogeneous astronomical data centers across regions downward through the resource node device, such as the data resources, storage resources, computing resources, etc. of the astronomical data center; the online scientific platform can also provide users (astronomical scientists) with interfaces for accessing and running scientific applications upward. Therefore, through the online scientific platform, astronomers do not need to spend a lot of time downloading data and perform calculations on a small part of the data locally, nor do they need to build and operate information facilities such as data downloading, computing, and storage locally. In the embodiment of the present disclosure, the corresponding application environment can be built through the scientific platform. By using the online scientific research platform, the storage resources and computing resources of the astronomical data center where the astronomical data is located can be used to perform online astronomical data calculations, which reduces the time for downloading TB / PB / TB-level astronomical data and greatly improves data processing efficiency.

[0075] According to an embodiment of the present disclosure, the management server is also configured to receive multiple data computing requests from the self-service system in batches, and forward each of the multiple data computing requests to a resource node device corresponding to each data computing request, so that the multiple resource node devices call their respective astronomical applications in parallel to calculate their respective astronomical data according to their respective data computing requests.

[0076] Multiple users can perform online data calculations simultaneously through the self-service system. For example, multiple users in multiple regions can perform multiple calculations on astronomical data of different resource node devices at the same time. In this embodiment, multiple resource node devices can call their own astronomical applications to calculate their own astronomical data in parallel according to their own data calculation requests.

[0077] Alternatively, multiple users in multiple regions can perform multiple calculations on the astronomical data of the same resource node device at the same time. In this embodiment, the resource node device can call the astronomical application corresponding to each data calculation request in parallel to calculate the respective astronomical data.

[0078] Specifically, after the management server receives multiple data calculation requests from the self-service system in batches, it can parse each data calculation request to determine the resource node device where the astronomical data that the data calculation request needs to call online is located, thereby forwarding the data calculation request to the resource node device corresponding to each data calculation request. For the resource node device corresponding to each data calculation request, the resource node device can call the corresponding astronomical application program according to the received data calculation request to calculate the astronomical data indicated by the data calculation request to obtain the calculation result. In addition, the resource node device can also store the calculation result at the address corresponding to the user who initiated the data calculation request in the private resource area.

[0079] In the embodiments of the present disclosure, the online scientific platform can process multiple data computing requests in batches, forward the multiple data computing requests to the corresponding resource node devices to implement parallel and online data computing, and meet the data computing needs of multiple users online at the same time.

[0080] According to an embodiment of the present disclosure, a data request includes a data import request, which is used to import new astronomical data; the management server is also used to: receive a data import request from a self-service system; determine a target resource node device corresponding to the astronomical data from a plurality of resource node devices according to the data import request; and mount the new astronomical data in a shared computing resource area of ​​the target resource node device in read-only mode, so that a user can obtain the new astronomical data online through a data request.

[0081] When faced with newly added astronomical data from a large astronomical scientific facility, or newly added astronomical data from a certain large astronomical scientific facility, users can log in to the online science platform and submit a data import request through the self-service system to import the above new astronomical data into the online science platform.

[0082] Specifically, the management server receives a data import request from the self-service system, and after parsing the data import request, the identification of the target resource node device can be determined, and then the target resource node device corresponding to the astronomical data is determined from multiple resource node devices. The new astronomical data is mounted in a read-only mode to the shared computing resource area of ​​the target resource node device, so that the user can obtain the new astronomical data online through the data request. The data import request may include the storage path of the new astronomical data, and thus, the new astronomical data can be mounted in a read-only mode to the shared computing resource area of ​​the target resource node device according to the storage path.

[0083] For example, for newly added astronomical data of an astronomical large scientific facility, the target resource node device can be the resource node device corresponding to the astronomical data center associated with the astronomical large scientific facility, and the new astronomical data can be mounted in read-only mode to the shared computing resource area of ​​the target resource node device and the address corresponding to the astronomical large scientific facility. Alternatively, for newly added astronomical data of a certain astronomical large scientific facility, the target resource node device can still be the resource node device corresponding to the astronomical data center associated with the astronomical large scientific facility; for a newly added astronomical large scientific facility, a new address can be added, and all astronomical data of the newly added astronomical large scientific facility can be mounted in read-only mode at this address.

[0084] In the embodiments of the present disclosure, when faced with newly added data from a large astronomical scientific facility, or multiple newly added data from a certain astronomical big data, the online scientific platform can quickly store the data without the need to temporarily store the newly added astronomical data through hardware and import it locally, thereby reducing the time cost of storing and using astronomical big data.

[0085] The storage of the newly added astronomical data can rely on the resource node device corresponding to the astronomical data center, and the calculation of the newly added astronomical data can be performed based on the original computing environment of the resource node device (the mirror data of the astronomical application program in the resource node device). For a newly added astronomical data center in a certain region, a resource node device corresponding to the astronomical data center is added to the online scientific platform.

[0086] According to an embodiment of the present disclosure, a data request includes a resource node device creation request, and the resource node device creation request is used to connect a new astronomical data center to an online scientific research platform; the management server is further configured to: receive a resource node device creation request from a self-service system; create a new resource node device corresponding to the new astronomical data center according to the resource node device creation request; mount the astronomical data of the new astronomical data center to a shared data resource area of ​​the new resource node device in read-only mode; and store mirror data of multiple astronomical applications for processing the astronomical data of the new astronomical data center in the shared computing resource area of ​​the new resource node device, so that users can call the astronomical application to calculate the astronomical data of the new astronomical data center through data calculation requests.

[0087] Specifically, a user (such as an administrator) can log in to the online science platform and submit a resource node device creation request through the self-service system. After the management server receives the resource node device creation request from the self-service system, it can create a new resource node device corresponding to the new astronomical data center. For example, the management server creates a new resource node device based on the physical resources that can be called, and associates the new resource node device with the new astronomical data center, and stores the association information between the two. Then, based on the association relationship, the astronomical data of the new astronomical data center can be mounted in read-only mode to the shared data resource area of ​​the new resource node device;

[0088] In an embodiment of the present disclosure, for a newly added astronomical data center in a certain area, a resource node device corresponding to the astronomical data center is added, so that users located in other areas can obtain the astronomical data of the astronomical data center online and use the astronomical data for online calculations, thereby avoiding the low resource utilization efficiency caused by bandwidth limitations in the traditional scientific model.

[0089] According to an embodiment of the present disclosure, multiple resource node devices correspond to astronomical data centers located in multiple geographical locations respectively; the platform also includes a network system, the network system includes multiple virtual private networks between the management server and the multiple resource node devices, so that the management server interacts with the resource node devices through the virtual private networks between each resource node device.

[0090] Although the components of the online science platform are distributed across regions, all resource node devices and the management server are interconnected through the network, thereby achieving information exchange between the management server and each resource node device and improving the stability of data transmission between each resource node device.

[0091] For example, a virtual private network (VPN) such as a VPN link may be deployed between each resource node device and the management server, wherein the VPN on the management server side serves as a primary server and can receive VPN requests from each resource node device. The network system includes a VPN link between each resource node device and the management server.

[0092] According to an embodiment of the present disclosure, the data request also includes an application access request, and the network system also includes an application gateway, which is used to receive the application access request from the self-service system and access the target astronomical application in the resource node device corresponding to the application access request based on the application access request.

[0093] Multiple astronomical applications can be deployed inside each resource node device. In addition to data calculation, users can also directly access each astronomical application deployed on each resource node device, such as jupyterlab, vscode, topcat, ds9 and other application development or data analysis applications.

[0094] Therefore, after deploying the VPN link, the network system is also configured with an application gateway to enable users to directly access the astronomical applications created on each resource node device. The network of each resource node device is planned in the application gateway, and the corresponding network route is set. Therefore, after the user submits an application access request through the self-service system, the application gateway can determine to route the access request to a certain resource node device according to the pre-set network route, so as to access the target astronomical application in the resource node device corresponding to the application access request, thereby realizing the interconnection between each resource node device.

[0095] Figure 4 The system architecture of the online scientific research platform for astronomical big data according to the embodiment of the present disclosure is schematically shown. Figure 4 As shown, users can log in to the online scientific platform through the user terminal and submit various data requests through the self-service system. Each resource node device can send a VPN request to the management server for interaction.

[0096] like Figure 4 As shown in Figure 1, the online science platform supports multiple protocols to access target astronomical applications, such as Secure Shell (SSH), Remote Desktop Protocol (RDP), and HyperText Transfer Protocol (HTTP). After the user initiates an application access request based on SSH, RDP, or HTTP through the self-service system, the application gateway receives and routes the application access request so that the target astronomical application in the resource node device corresponding to the application access request can be accessed.

[0097] The online scientific platform can provide unified identity authentication for each component through an independent unified authentication system.

[0098] like Figure 4 As shown, the LDAP system built based on the Lightweight Directory Access Protocol (LDAP) can provide unified identity authentication services for the management server, resource node device, and computing server in the resource node device. The resource node device automatically connects to the LDAP system after startup.

[0099] In addition to receiving resource information of each resource node device, the management server also includes a main server and a backup server. The backup server is used to asynchronously back up data from the main server to ensure high availability and security of the data.

[0100] For example, the management server can adopt a one-master and one-standby mode. Figure 4 As shown, the two management servers can be set as a primary server and a backup server respectively, and the backup server asynchronously backs up data from the primary server.

[0101] It should be noted that in the embodiments of the present disclosure, certain software, components, models and other existing solutions in the industry may be mentioned, which should be regarded as exemplary. Their purpose is only to illustrate the feasibility of implementing the technical solution of the present disclosure, but it does not mean that the solution has been or will necessarily be used.

[0102] In a specific embodiment, the online science platform can provide computing services based on different components such as the self-service system, management server, network system, and unified authentication system. For example, the basic process for a user to access the online science platform is:

[0103] Step 1: When a new user logs in to the online science platform, he can apply for a user account and set a username and password. After logging in to the online science platform, the unified authentication system can create a user space for the user. For example, the private resource area of ​​each resource node device creates a root directory corresponding to the user. For old users, after logging in to the online science platform, the existing user space can be automatically bound.

[0104] Step 2: Users can submit data requests through the self-service system. For example, if you submit a data query request, you can query the astronomical data of astronomical data centers in different regions online; if you submit a data calculation request, you can select the calculation area (resource node device) where the astronomical data is located, forward the data calculation request to the corresponding calculation area through the management server, and calculate the astronomical data by calling the astronomical application in the calculation area where the astronomical data is located.

[0105] Step 3: Users can also create the required astronomical applications in the specified computing area, such as creating astronomical applications such as jupyterlab, vscode, topcat, ds9, etc. on a resource node device through the VPN link between the application gateway or management server and the resource node device.

[0106] Step 4: The online science platform can provide an online access address for the created astronomical application. Users access the astronomical application at the online access address through the application gateway to perform data calculations.

[0107] Step 5: After completing the calculation using the astronomy application, the user can close the astronomy application and the online science platform will reclaim the computing resources.

[0108] In the embodiments of the present disclosure, for astronomical data at the TB / PB / EB level, the online scientific platform can be used to implement data storage, online data query, online calculation, online calculation result storage, and query operations on previous calculation results. The embodiments of the present disclosure provide a new astronomical data processing mode, which gets rid of the data download and transmission time limitations brought by traditional modes such as downloading data and analyzing data, reduces data usage time, and improves data calculation efficiency.

[0109] According to an embodiment of the present disclosure, the program code for executing the computer program provided by the embodiment of the present disclosure can be written in any combination of one or more programming languages. Specifically, these computing programs can be implemented using high-level process and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, Java, C++, python, "C" language or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on the remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, using an Internet service provider to connect through the Internet).

[0110] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram may represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of the code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box may also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions. It can be understood by those skilled in the art that the features recorded in the various embodiments of the present disclosure can be combined and / or combined in a variety of ways, even if such a combination or combination is not explicitly recorded in the present disclosure. In particular, without departing from the spirit and teaching of the present disclosure, the features described in the various embodiments of the present disclosure may be combined and / or combined in a variety of ways. All of these combinations and / or combinations fall within the scope of the present disclosure.

[0111] The embodiments of the present disclosure are described above. However, these embodiments are only for illustrative purposes and are not intended to limit the scope of the present disclosure. Although the embodiments are described above, this does not mean that the measures in the various embodiments cannot be used in combination to advantage. Without departing from the scope of the present disclosure, those skilled in the art may make a variety of substitutions and modifications, which should all fall within the scope of the present disclosure.

Claims

1. A resource node device for astronomical big data, characterized in that: The resource node device corresponds to a single astronomical data center, and the resource node device includes a private resource area and a shared resource area. The shared resource area includes a shared data resource area and a shared computing resource area. The shared computing resource area is used to store the mirror data of a plurality of the astronomical applications, and the shared data resource area is used to mount the astronomical data of the astronomical data center in a read-only mode. The private resource area is used to store the calculation results after each user calls the astronomical application to calculate the astronomical data of the astronomical data center. The resource node device is configured as follows: In response to receiving a data computing request from a management server, determining a target astronomical application matching the data computing request from the shared computing resource area, and determining target astronomical data to be processed from the shared data resource area; Calling the target astronomical application to calculate the target astronomical data to obtain a target calculation result; The target calculation result is stored in a root directory in the private resource area that matches the target user who initiated the data request, and the target calculation result is returned to the management server.

2. The resource node device according to claim 1, characterized in that: The resource node device is further configured to: In response to receiving a resource query request from the management server, obtaining resource information of the resource node device, wherein the resource information includes at least one of the following: resource node device status, available resources, scheduling type, and available astronomical applications; The resource information is returned to the management server.

3. The resource node device according to claim 1, characterized in that: The resource node device is further configured to: In response to receiving a calculation result query request from the management server, determining a query user according to the calculation result query request, wherein the calculation result query request is for the target calculation result; When it is determined that the querying user has query rights for the target calculation result, the target calculation result is obtained from the private resource area and returned to the management server so that the management server returns the target calculation result to the querying user.

4. The resource node device according to claim 1, characterized in that: The resource node device manages the shared resource area and the private resource area of ​​the resource node device through a virtual machine computing cluster, a containerized cluster and / or a job scheduling system.

5. An online scientific research platform for astronomical big data, characterized in that: The platform comprises: a unified authentication system, a self-service system, a management server and a plurality of resource node devices according to any one of claims 1 to 4; The unified authentication system is configured to authenticate the identity of the user who logs into the self-service system; The self-service system is configured to display a front-end page so that a user who has passed identity authentication can submit a data request based on the front-end page, wherein the data request includes a data calculation request, and the data calculation request is used to call an astronomical application to calculate astronomical data; The management server is configured to receive the data calculation request and forward the data calculation request to the corresponding resource node device; and forward the target calculation result returned by the resource node device to the self-service system.

6. The platform according to claim 1, characterized in that: The management server is also configured to: Batch receive multiple data calculation requests from the self-service system, and forward each of the multiple data calculation requests to the resource node device corresponding to each data calculation request, so that multiple resource node devices call their own astronomical applications in parallel to calculate their own astronomical data according to their own data calculation requests.

7. The platform according to claim 1, characterized in that The data request includes a data import request, and the data import request is used to import new astronomical data; the management server is further configured to: receiving the data import request from the self-service system; determining, according to the data import request, a target resource node device corresponding to the astronomical data from the plurality of resource node devices; as well as The new astronomical data is mounted in a read-only mode to the shared computing resource area of ​​the target resource node device, so that the user can obtain the new astronomical data online through the data request.

8. The platform according to claim 1, characterized in that The data request includes a resource node device creation request, and the resource node device creation request is used to access the new astronomical data center to the online scientific research platform; The management server is also configured to: receiving a request for creating the resource node device from the self-service system; Creating a new resource node device corresponding to the new astronomical data center according to the resource node device creation request; Mounting the astronomical data of the new astronomical data center to the shared data resource area of ​​the new resource node device in a read-only mode; as well as Mirror data of multiple astronomical application programs for processing the astronomical data of the new astronomical data center are stored in the shared computing resource area of ​​the new resource node device, so that the user can call the astronomical application program to calculate the astronomical data of the new astronomical data center through the data calculation request.

9. The platform according to claim 1, characterized in that The plurality of resource node devices respectively correspond to the astronomical data centers located at a plurality of geographical locations; The platform also includes a network system, which includes multiple virtual private networks between the management server and multiple resource node devices, so that the management server interacts with the resource node devices through the virtual private networks between each of the resource node devices; the data request also includes an application access request, and the network system also includes an application gateway, which is configured to receive the application access request from the self-service system and access the astronomical application in the resource node device corresponding to the application access request based on the application access request.

10. The platform according to claim 1, characterized in that The management server includes a primary server and a secondary server, and the secondary server is configured to asynchronously back up data from the primary server.

Images