Space science data cooperative computing method and system based on no-service computing architecture

Abstract

The invention discloses a space science data collaborative computing method and system based on a no-service computing architecture. According to the method, the technical problems that in traditional space science data processing, resource elasticity is insufficient, multidisciplinary cross-domain heterogeneous computing environment collaboration is difficult and data islands are serious are solved by constructing a whole-process and service-free computing framework of data access-function encapsulation-collaborative scheduling-result aggregation. Specifically, tasks such as preprocessing, analysis and modeling of space science data are packaged into a lightweight and non-service calculation function, data calculation collaborative calling in the interdisciplinary field is achieved in combination with a multi-calculation-node permission management mechanism, and calculation resources are distributed according to needs through a dynamic resource scheduling algorithm. The multi-disciplinary cross-domain TB / PB-level spatial data processing method has the advantages of being high in resource utilization rate, low in operation and maintenance cost and high in expansibility, and can be widely applied to space science research scenes such as sun-earth space physics, deep space exploration and high-energy celestial body outbreak.

Description

Technical Field

[0001] This invention belongs to the fields of cloud computing technology and data analysis and mining. More specifically, this invention integrates containerization, user-space virtual file system and edge computing technology to realize a collaborative computing method for multi-source space science data (satellite observation data, detector measured data, ground station received data, etc.), as well as a system for implementing this method. Background Technology

[0002] Space science research relies on massive amounts of multi-source data. For example, the data generated by solar probes, Mars probes, and Earth space probes has reached the TB to PB level. Moreover, the data sources are scattered (across regions and file systems), the computing environments are heterogeneous (multiple algorithm models and programming environments), and the computing demands fluctuate dynamically (long-term time-series data analysis, single image recognition, and joint mining of multi-payload data). Traditional space science data processing solutions have the following core problems.

[0003] Traditional solutions rely on fixed distributed computing clusters (such as Hadoop / Spark clusters). The resource allocation is in a "static pre-configuration" mode. During peak periods, insufficient resources lead to task blocking, while during idle periods, resources are idle (the utilization rate is usually less than 40%), which cannot match the "pulsating" computing needs of spatial data processing.

[0004] Space science research often requires cross-disciplinary collaboration (such as coronal mass ejections, interplanetary shock waves, and geomagnetic storms), but various types of observation data are stored on independent servers, resulting in "data silos." At the same time, the integrated encapsulation, scheduling, and management capabilities of traditional multi-data analysis frameworks cannot achieve cross-disciplinary collaboration of data and computational models, leading to low efficiency in the joint analysis of complex space science events.

[0005] Traditional computing clusters require specialized teams to maintain server hardware, operating systems, and middleware. The operation and maintenance costs account for too high a proportion of the overall R&D costs, and they are difficult to cope with the cluster expansion needs brought about by the large-scale growth of space science data.

[0006] In traditional solutions, tasks such as data preprocessing, feature extraction, and model training are coupled into a single program. If a step needs to be modified, such as the coordinate projection correction of space-based satellite optical observation data, the entire calculation process needs to be reconstructed, resulting in poor flexibility.

[0007] In summary, practical space science data analysis and computation face challenges such as multi-source heterogeneity of data, heterogeneity of analytical methodologies, lack of adaptation to the characteristics of space science data ("high throughput, strong correlation, and multi-entity collaboration"), and unresolved issues such as standardized access to multi-source space data, collaborative scheduling of computational tasks and data locations, and cross-domain data sharing. Solving these problems will significantly improve the integration level of data analysis and knowledge mining in the scientific research field and enhance the user experience. Summary of the Invention

[0008] To address the problems existing in the prior art, the purpose of this invention is to provide a method and system for collaborative computing of space science data based on a serverless computing architecture. To solve the above problems, the following requirements must be met.

[0009] (1) Access to multi-source space science data: Access to multi-source space data sources is achieved through standardized interfaces (such as NASA CDF, ESA FITS, and domestic aerospace data format (such as CDAS) adapter interfaces), including real-time data streams from satellites / probes (such as corona observation data, plasma data, and magnetic field data); historical archived data; and ground station observation data (magnetometers and gasglow imagers). SSL / TLS encrypted transmission is used during the access process to ensure data integrity.

[0010] (2) Data standardization preprocessing: Call the preset serverless computing preprocessing function to process the access data and generate a unified format "space science data asset object". First, remove outliers (such as jump data caused by detector failure) and use Bayesian interpolation to fill in missing values; and establish a user-mode virtual file system file index pointer for data from different sources (CDF, ​​FITS, HDF5).

[0011] (3) Metadata annotation: Add metadata such as "source identifier (e.g., satellite number), timestamp, data type (e.g., plasma bubble event observation data)" to the data unit and store it in the metadata database.

[0012] Data analysis and mining function encapsulation: Establish function categories, including data preprocessing functions such as data filtering (Kalman filtering, moving average) and data dimensionality reduction (PCA, t-SNE); data analysis functions such as solar wind velocity fitting and planetary magnetic field simulation models; and result visualization functions such as spectrum generation and orbital data plotting.

[0013] (4) Functionalization of Multidisciplinary Algorithm Models in Space Science: Using containerization technology, algorithm models for multidisciplinary and cross-domain applications in space science, such as data processing, solar-terrestrial space physics, interplanetary science, high-energy astronomy, and geospace, are encapsulated into serverless computing functions. The input and output of the functions are defined as "space science data unit + parameter configuration" (such as filter window size, fitting algorithm parameters, and space weather event time window), and chained calls between functions are supported (such as "data filtering → simulation of key physical parameters of solar wind → visualization"). The encapsulated serverless computing functions are registered to the function management center, and metadata information such as "computing resource requirements (number of CPU cores, memory), applicable data types, and calling permissions" of the functions are marked to form a searchable and manageable function library.

[0014] (5) Multi-collaborative entity computing task orchestration: Construct an interactive visual interface (Web terminal) to accept user computing orchestration requirements, support data sources, functions (algorithm models) in collaborative entities, and jointly define collaborative computing tasks, including selecting data to be processed from standardized data units (supporting cross-node data combination); selecting the required serverless computing functions from the function library, and defining the function call order (such as "space weather event recognition, feature extraction, model training").

[0015] (6) Task parsing: The user-defined collaborative task is parsed as a “directed acyclic graph (DAG)”, where nodes correspond to a single serviceless computation function or a single data unit; edges correspond to the dependencies between functions and data, and between functions (such as “data unit → space weather event identification → physical parameter extraction function”).

[0016] (7) Dynamic resource scheduling and task execution: The scheduling module decomposes the functions in the DAG graph into parallel tasks and assigns them to different containers for execution. Based on the meta-information of each function, the amount of data to be processed, and historical execution data, resources are scheduled according to the principle of "allocation on demand and priority to the nearest". Resources are allocated only when the function is triggered and released after the function is completed to avoid idleness.

[0017] (8) Calculation result aggregation and feedback: The result processing module collects the execution results of each serviceless calculation function (such as fitting parameters and visualization charts), and aggregates them according to the logic defined in the task (such as merging the fitting results of magnetic field data from multiple satellites into a unified magnetosphere model). It supports online viewing, downloading and secondary analysis of the results. At the same time, the task execution log (resource usage and execution time) is stored in the log database for subsequent optimization of resource scheduling algorithms.

[0018] This invention addresses the technical pain points of traditional space science data processing—namely, insufficient resource elasticity, difficulty in coordinating heterogeneous computing environments across multiple disciplines, and severe data silos—by constructing a full-process, serverless computing framework encompassing "data access - function encapsulation - collaborative scheduling - result aggregation." Specifically, this invention encapsulates computational tasks such as preprocessing, analysis, and modeling of space science data into lightweight, serverless computational functions. Combined with a multi-computing node permission management mechanism, it enables collaborative data computation across disciplines (Solar-Earth space physics, interplanetary science, and high-energy astronomy). Furthermore, a dynamic resource scheduling algorithm allocates computing resources on demand, achieving efficient processing of TB / PB-level space data across multiple disciplines. This system boasts advantages such as high resource utilization, low maintenance costs, and strong scalability, and can be widely applied to space science research scenarios including Solar-Earth space physics, planetary science, deep space exploration, and high-energy celestial bursts.

[0019] This invention comprises the following core modules: a multi-source heterogeneous space science data access module, a space science algorithm model function encapsulation and management module, a task orchestration and management module, a serverless computing scheduling module, a permission management module, a computation result processing module, and a data storage module. First, multi-source heterogeneous space science data is accessed, and a virtual file system index is constructed and metadata is generated. Container technology is used to encapsulate, register, retrieve, and version-manage various space science data analysis and mining algorithm models' serverless computing functions, maintaining a function metadata database. Based on user interaction, target data sources and functions are selected for collaborative computation task orchestration. Tasks are parsed into a DAG graph for scheduling, and the creation, execution, termination, and destruction of tasks are monitored. Task messages and log records are pushed to users in real time. At the end of the computation task, the execution results of each function are aggregated, the computation results are collected, standardized reports are generated, and stored in the metadata database and log database, enabling task retrieval and analysis of computation results.

[0020] The technical solution adopted in this invention is as follows: Figure 1 As shown, it includes the following modules.

[0021] (1) Multi-source heterogeneous space science data access module: This module provides standardized access interfaces (CDF / FITS / CDAS / DAT / SHAPEFILE) for multi-source space science data, enabling SSL / TLS encrypted data transmission, performing data cleaning, format conversion, and metadata annotation, and generating standardized space data units. Specifically, it generates a corresponding space science data unit for each source of space data.

[0022] (2) Functional encapsulation of space science algorithm models into a registration management module: This invention, based on the principle of atomized decomposition of space science computing tasks (preprocessing / analysis / visualization), divides the space science computing tasks to be processed into multiple computing units. Containerization technology is used to encapsulate each computing unit as a serverless computing function. To meet the requirements of serverless computing function reuse and version control, a function registration mechanism is established, providing a registration API for serverless computing function providers to annotate serverless computing function metadata (CPU / memory requirements, input data type, output data type, output data location, and permission scope). Based on the input and output types of the serverless computing function metadata, it can be inferred whether two serverless computing functions have the ability to connect upstream and downstream (e.g., if serverless computing function A outputs a PNG image and serverless computing function B inputs a PNG image, it can be determined that they can build a chain of calls). Based on the requirements of the serverless computing function call chain, it supports the configuration of function chain relationships, forming a searchable and iterable function library, and realizing function version management and canary release.

[0023] (3) Task orchestration and management module: According to the instructions of the collaborating entities, a collaborative computing task is generated by retrieving and selecting space science data units to be processed from the metadata database, selecting the required serverless computing functions from the function library, defining the calling order between the selected serverless computing functions, setting the operation permissions of each collaborating entity, and generating a collaborative computing task. Then, the collaborative computing task is parsed into a directed acyclic graph. In this graph, nodes correspond to space science data units or serverless computing functions, and edges correspond to the dependencies between data and functions, and the dependencies between functions.

[0024] This invention addresses the task definition requirements of collaborative scenarios involving multiple data sources and multiple algorithm models. It provides a visual interface with lists of serverless computation functions and data units to support cross-entity (data provider / computation participant) selection of arbitrary data units and serverless computation functions. Based on task logical relationships, it parses user-defined collaborative processes into directed acyclic graphs (DAGs), clearly defining the "data-function" dependency relationship. Furthermore, based on the requirements for full task lifecycle management, it implements status control for task creation, review, execution, and termination, and stores task configuration information for traceability.

[0025] (4) Serverless computing scheduling module: This module employs a priority queue algorithm, combining the metadata of each serverless computation function in the directed acyclic graph (DAG), the amount of data to be processed, and historical execution data. When a serverless computation function is triggered, resources are allocated to the corresponding function, a serverless computation function instance is created, and execution is performed. Specifically, based on the resource fluctuation characteristics (pulsating demand) of spatial data computation, a priority queue algorithm is used to analyze historical data and function metadata to dynamically allocate the optimal resource allocation scheme (CPU / memory) for each task. Based on the principles of multi-edge node collaboration and proximity computing, interfaces with various cloud services (Kubernetes, AWS, Alibaba Cloud) are connected to achieve on-demand resource allocation (allocation upon triggering, release after execution) and automated deployment of serverless computation functions, deploying and scheduling them in the data storage area based on proximity. Based on the parallelism characteristics of the DAG graph, tasks are decomposed and distributed to serverless computation instances. Data consistency is ensured through object storage differential blocks, and the execution status is monitored and a retry mechanism is triggered.

[0026] (5) Access Control Module: Based on the security requirements of cross-organizational collaboration, this invention defines principal roles (data provider / computation participant / auditor) using the RBAC (role-based access control) mechanism; configures operation permissions such as data viewing, function calling, and result modification for each role according to fine-grained permission control requirements; and records permission change logs for each role according to permission auditing requirements to ensure traceability of the collaboration process and prevent unauthorized access.

[0027] (6) Calculation result processing module: This module aggregates the computation results of various serverless computing instances according to aggregation requirements, collects the outputs of each serverless computing function instance (fitting parameters, physical parameters, simulation values, visualization charts, and report fragments), and merges them into a unified result according to DAG logic; based on the permission configuration of the collaborating entity, it pushes standardized reports (PDF / JSON / PNG) to the corresponding users; and according to secondary analysis needs, it supports online viewing, downloading, and parameter adjustment of results, thereby improving research efficiency.

[0028] (7) Data storage module: Based on the hierarchical storage requirements of spatial data, this module uses MongoDB to store metadata (supporting efficient retrieval of complex data structures), Redis for efficient real-time collection and push of task execution logs in a producer-consumer mode, and Minio cloud-native distributed object storage to store large-scale heterogeneous raw data and result files. According to the data lifecycle management requirements, it enables fast access to hot data (recent observation data) and low-cost storage of cold data (historical archived data), adapting to TB / PB level data scales.

[0029] The collaborative computing method of the present invention is as follows: Figure 2 As shown, specifically: (1) Multi-source space science data access and standardized preprocessing, such as Figure 2 As shown in step (a): Standardized interfaces are used to connect to various space data sources, ensuring data transmission security and format compatibility. This includes supporting access from satellites / probes (such as solar wind plasma data and planetary magnetic field data), employing dedicated interfaces such as NASA CDF, ESA FITS, and domestic aerospace data CDAS. Transmission is encrypted using SSL / TLS to ensure data integrity. Historical data from file systems (such as solar activity observation data from the past year) is also integrated, supporting breakpoint resumption and data verification. Ground station data access is also supported: Data formats from ground-based equipment such as radio telescope arrays and optical observatories are adapted to achieve unified access to ground-based observation data and space-based data. Metadata is added to the accessed data, including "data source, observation timestamp, parameter type, and data accuracy," generating a unified format "space science data unit" which is stored in a metadata database for efficient retrieval.

[0030] (2) Encapsulation and registration of serverless computing functions for space science computing tasks, such as Figure 2As shown in step (b): Based on the business logic of space science data processing, complex computational tasks are broken down into "atomic and reusable" computational units, specifically including data preprocessing and data analysis units. Docker containerization technology is used to encapsulate these computational units into serverless computational functions. Function inputs and outputs are defined as "standardized spatial data units + parameter configurations," supporting chained calls between functions to ensure that modifications to a single function do not affect the overall process. The encapsulated functions are registered in the "Function Management Center," with metadata (CPU core requirements, memory size, applicable data types, and calling permission scope) labeled to form a searchable function library, supporting function version management and canary releases.

[0031] (3) Visual definition of collaborative tasks, as follows Figure 2 As shown in step (c): A web / client-side visual interface is provided, supporting cross-institutional collaborative entities (data providers and computing participants) to jointly define collaborative computing tasks. Specific operations include retrieving and selecting the "space science data unit" to be processed from the metadata database, supporting cross-source data combination (such as satellite data + ground station data); selecting the required serverless computing functions from the function library, and defining the function call order by dragging and dropping (such as "data filtering → feature extraction → space weather event identification"); and setting the operation permissions of each entity based on the RBAC mechanism (such as data providers only being allowed to view data, and computing participants only being allowed to call functions).

[0032] (4) DAG parsing of collaborative tasks as follows Figure 2 As shown in step (d): The task management module parses the user-defined collaborative computing task into a "directed acyclic graph (DAG)", where: nodes correspond to "space science data units" or "serviceless computing functions", and edges correspond to the dependencies of "data → function" and "function → function" (such as "solar wind data unit → filtering function → velocity fitting function"), ensuring the logical correctness of task execution.

[0033] (5) Dynamic resource planning based on priority queues, such as Figure 2As shown in step (e): The scheduling module adopts a priority queue algorithm, combining the meta-information (resource requirements), the amount of data to be processed (such as TB-level Lyman-Alpha Telescope observation data), and historical execution data (resource consumption of functions with similar data) of each function in the DAG graph, and only allocates resources to the corresponding function when the function is triggered. Specifically, firstly, the DAG is split into multiple tasks to generate a task queue. Based on the information of each task in the task queue, the actual storage location of the input data of the corresponding serverless computing function is determined, and the serverless computing function is scheduled to the data storage node where the data it processes is located. Then, the current physical computing resource load and available margin information are obtained. If there is no historical running information for the task, the available margin is used to schedule resources for the corresponding serverless computing function in the task and execute it. If there is historical running information for the task, a greedy algorithm is used to plan the optimal resource allocation scheme and running plan for the serverless computing function corresponding to the task based on the historical usage.

[0034] This invention determines the actual storage location of the input data (i.e., data unit) of the serverless computing function based on the information of each split task, and schedules the serverless computing function to the data storage node. Based on the cloud service (K8s, AWS, Alibaba Cloud) interface, it obtains the load and available balance information of the current physical computing resources (CPU, memory, GPU, disk, network storage volume). If there is no historical information for the current task, it directly attempts to schedule the available balance for the application and schedules resources for the serverless computing function corresponding to the task. If the operation is abnormal, it is re-queued and continues to monitor the available balance of the current physical resources. If the available balance increases, it continues to schedule, with a maximum of five retries. If there is historical running information, it uses a greedy algorithm to plan the optimal resource allocation scheme and running plan for each function based on the historical usage. After the function is completed, it releases the resources immediately to avoid over-allocation or under-allocation of resources.

[0035] Distributed task execution, such as Figure 2 As shown in step (f): The scheduling module decomposes the parallel nodes in the DAG graph into independent tasks and assigns them to different serverless computing instances for execution. It ensures data consistency through erasure coding mechanism and uses message queue to monitor the task execution status in real time (such as running, success, failure). When a serverless computing instance fails, the retry mechanism is automatically triggered to ensure task continuity.

[0036] (6) Aggregation of multi-source results, such as Figure 2 As shown in step (g): The results processing module collects the execution results (such as fitting parameters, visualization charts, and analysis report fragments) of each serverless computation function instance, aggregates them according to the DAG graph logic, and integrates the scattered visualization charts into a complete analysis report.

[0037] (7) Result-based permission feedback, such as Figure 2 As shown in step (h): Based on the permissions configured in step 4, the aggregation results are pushed to the terminals (Web terminals) of each collaborating entity, supporting online viewing, downloading and secondary analysis of the results; at the same time, the task execution logs are stored in the MongoDB log database for subsequent resource scheduling algorithm optimization and problem investigation.

[0038] The advantages of this invention are as follows: (1) Multidisciplinary and cross-domain adaptation in space science: Based on the characteristics of space science research tasks being "multi-type (geophysics, solar physics, planetary science, deep space exploration, high-energy astronomy) and rapidly iterating (frequent algorithm optimization)," this invention breaks down the computational process into atomic, serverless computational functions (such as event recognition, key parameter extraction, and numerical simulation). Modification of a single function (such as changing the plasma bubble recognition algorithm) does not require reconstructing the entire process. At the same time, the system supports interface integration with multiple cloud service providers and can be horizontally expanded to meet PB-level data processing needs, solving the problems of "long expansion cycle and high cost" in traditional clusters.

[0039] (2) Low data processing latency: Based on the characteristics of space science data, which are "massive (TB / PB level) and strongly correlated (requiring multi-source data linkage)," a "priority proximity" scheduling strategy is designed. Based on the description of the function input data type, the serverless computing function is deployed in the region where the data storage node is located to reduce the cross-regional data transmission distance. Combined with the parallel decomposition technology of DAG graph, the synchronous computing of multiple function instances is realized.

[0040] (3) Elastic adaptability of resources for the "pulse-like" characteristics of space science computing: Based on the characteristics of space science data processing, which is characterized by "surge in computing demand during peak periods (such as the release of data products from new scientific missions) and idle resources during off-peak periods", an "on-demand allocation" mode of serverless computing architecture is adopted. Resources (CPU / memory) are dynamically allocated only when a computing task is triggered and released immediately after the task is completed, completely solving the resource waste problem caused by the "static pre-configuration" of traditional fixed clusters. Moreover, it can be expanded in minutes during peak periods to avoid task blocking and perfectly adapt to the fluctuating computing needs of scenarios such as solar exploration and planetary observation. Attached Figure Description

[0041] Figure 1 This is a diagram of the module architecture of the present invention.

[0042] Figure 2 This is a flowchart of the data question-and-answer method in the scientific research field of the present invention. Detailed Implementation

[0043] The present invention will now be described in further detail with reference to the accompanying drawings. The examples given are only for explaining the present invention and are not intended to limit the scope of the present invention.

[0044] This invention is a collaborative computing method and system for space science data based on a serverless architecture, and its implementation includes the following steps: (1) Multi-source data access and standardized preprocessing: ① First, upload 100GB of raw geomagnetic component data and 30GB of raw solar surface observation data through the DAT interface (adapted to the Meridian Engineering Fluxgate Magnetometer) and FITS interface (adapted to SDI / AIA space-based observation data) of the "Data Access Module" of this system. The transmission process uses TLS 1.3 encryption. ② The system automatically calls metadata annotation: Adds source (FGM / SDI / AIA), observation time (2024-01-01 to 2024-01-31), parameter type (geomagnetic component / solar surface pixel), data accuracy (±1NT), physical storage partition (node ​​101 stores the original geomagnetic component data, node 102 stores the original solar surface observation data) metadata information to each data and stores it in the MongoDB metadata database.

[0045] (2) Serviceless computation function encapsulation of space science algorithm models: ① The "Key Parameter Extraction Model for Geomagnetic Wave Events" (based on a multi-layer convolutional neural network model) is encapsulated as a container, registered to the "Function Management Module", and labeled with the function metadata: 2 CPU cores, 4GB memory, 1GB GPU memory, applicable data type is "Meridian Engineering Fluxgate Magnetometer", output data type is CSV, and calling permission is "only for users of the same institution and the reviewer"; ② The "Coronal Mass Ejection Event Key Parameter Extraction Algorithm" (based on the YOLO model) is encapsulated into a container, registered to the "Function Management Module", and labeled with the function meta-information: 1 CPU core, 16GB memory, 4GB GPU memory, applicable data type is "SDI / AIA", output data type is CSV, and calling permission is "only for users of the same institution and the reviewer"; ③ The system has a built-in "Structured Data Merging" function (based on CSV to DataFrame data frame column merging), which is registered in the function library and has the permission "all collaborating entities can call".

[0046] (3) Definition of collaborative task: ① Users can collaboratively define tasks via the web interface, retrieve and select raw geomagnetic component data, drag and drop key parameter extraction models for geomagnetic fluctuation events to set function calls, and specify the output data as A.csv; ② Retrieve and select the raw solar surface observation data, drag and drop the key parameter extraction algorithm for coronal mass ejection events to set the function call, and specify the output data as B.csv; ③ Drag and drop the structured data merge function, specify the input as A.csv and B.csv to merge based on the timestamp column, and specify the output data as C.csv.

[0047] (4) Collaborative Task Analysis: ① The task management module parses the above tasks into a DAG graph and performs topological sorting: "[1.1 Original data of geomagnetic components, 1.2 Original data of solar surface observation] → [2.1 Key parameter extraction model for geomagnetic fluctuation events, 2.2 Key parameter extraction algorithm for coronal mass ejection events] → [3.1A.csv, 3.2B.csv] → [Structured data merging] → [4.1C.csv]", and stores the task configuration in the task management database.

[0048] (5) Dynamic resource scheduling and task execution ① The scheduling module plans resource requirements for the geomagnetic fluctuation event key parameter extraction model (CPU 2 cores, memory 4GB, GPU memory 1GB) and the coronal mass ejection event key parameter extraction algorithm (CPU 1 core, memory 16GB, GPU memory 4GB). The DAG graph is decomposed into two sets of parallel tasks and assigned to three cloud function instances for execution. The geomagnetic fluctuation event key parameter extraction model instance is assigned to node 101 to read geomagnetic data in situ using local I / O. The coronal mass ejection event key parameter extraction algorithm instance is assigned to node 102 to read geomagnetic data and raw solar surface observation data in situ using local I / O. The structured data merging instance is assigned to master node 100. A distributed lock is used to ensure that the same data unit is not processed repeatedly. ② During task execution, the scheduling module monitors the status in real time: it obtains computation log information in real time and pushes it to the message queue, and the task is eventually executed normally.

[0049] (6) Results aggregation and collaborative feedback: ① The results processing module collects intermediate calculation data and execution result data. The geomagnetic wave event key parameter extraction model outputs: extracted H-component geomagnetic wave event data A.csv; the coronal mass ejection event key parameter extraction algorithm outputs: identified and extracted coronal mass ejection event data B.csv; and the structured data merging instance outputs space weather event association data C.csv aggregated based on timestamps. All data is uniformly stored in distributed object storage. ② Add metadata about each piece of output data, including source (function output), generation time, parameter type, data precision, and physical storage partition (100-node object storage identifier), and store it in the MongoDB metadata database; ③ Push information according to permissions: Task record status change prompts, task participants can view intermediate data and aggregated results.

[0050] Although specific embodiments of the invention have been disclosed for illustrative purposes to aid in understanding and implementing the invention, those skilled in the art will understand that various substitutions, variations, and modifications are possible without departing from the spirit and scope of the invention and the appended claims. Therefore, the invention should not be limited to the content disclosed in the preferred embodiments, and the scope of protection claimed by the invention is defined by the claims.

Claims

1. A method for collaborative computing of space science data based on serverless computing architecture, comprising the steps of: 1) A multi-source heterogeneous space science data access module acquires data from selected various types of space data sources and annotates metadata, generates space science data units in a unified format and stores them in a metadata database; wherein a corresponding space science data unit is generated for the data of each space data source; 2) A space science algorithm model function packaging management module divides complex computing tasks into several computing units according to the business logic of space science data processing, encapsulates each computing unit into a serverless computing function using Docker containerization technology; registers the encapsulated serverless computing function to the function management center and annotates the meta information of the serverless computing function, forming a function library; 3) A task scheduling management module retrieves and selects space science data units to be processed from the metadata database, selects the required serverless computing functions from the function library, defines the calling order between the selected serverless computing functions, sets the operation permissions of each collaborative subject, and generates a collaborative computing task; then the collaborative computing task is parsed into a directed acyclic graph; wherein the nodes correspond to space science data units or serverless computing functions, and the edges correspond to the dependency relationship between data and functions, and the dependency relationship between functions; 4) A serverless computing scheduling module uses a priority queue algorithm, combines the meta information of each serverless computing function in the directed acyclic graph, the amount of data to be processed, and historical execution data, and allocates resources to the corresponding serverless computing function when it is triggered, creates a serverless computing function instance and executes it; 5) A computing result processing module aggregates the execution results of each serverless computing function instance according to the directed acyclic graph, and sends the aggregated results to the terminal of the corresponding collaborative subject according to the permissions of the collaborative subject.

2. The method of claim 1, wherein, The serverless computing scheduling module first splits the directed acyclic graph into multiple tasks to generate a task queue, determines the actual storage location of the input data of the corresponding serverless computing function according to the task information in the task queue, and schedules the serverless computing function to the data storage node where the data is processed; then it obtains the current physical computing resource load and available margin information, if there is no historical running information for the task, it uses the available margin to schedule resources for the corresponding serverless computing function in the task and execute it, if there is historical running information for the task, it uses a greedy algorithm to plan the optimal resource allocation scheme and running plan for the corresponding serverless computing function in the task according to the historical usage.

3. The method of claim 2, wherein, When a task runs abnormally, it is re-added to the task queue and continues to monitor the available margin of the current physical resources, if the available margin increases, it continues to schedule.

4. The method according to claim 1 or 2 or 3, characterized in that, The annotated metadata includes data source, observation timestamp, parameter type, and data precision.

5. The method according to claim 1 or 2 or 3, characterized in that, The annotated meta information of the serverless computing function includes CPU core requirement, memory size, applicable data type, and calling permission range.

6. A space science data collaborative computing system based on a service-free computing architecture, characterized in that, The system comprises a permission management module, a multi-source heterogeneous space science data access module, a space science algorithm model function encapsulation management module, a task arrangement management module, a serverless computing scheduling module, a computing result processing module, and a data storage module. The permission management module is configured to set roles of collaborative subjects and configure operation permissions of the roles. The permission management module is further configured to record permission change logs of the roles. The multi-source heterogeneous space science data access module is configured to obtain data from selected space data sources and perform metadata annotation on the data, to generate space science data units in a unified format and store the data units in a metadata database. The space science algorithm model function encapsulation management module is configured to split complex computing tasks into a plurality of computing units according to business logic of space science data processing, to encapsulate each computing unit into a serverless computing function by using a Docker container technology, to register the encapsulated serverless computing functions to a function management center and annotate metadata of the serverless computing functions, and to form a function library. The task arrangement management module is configured to retrieve and select space science data units to be processed from the metadata database, to select required serverless computing functions from the function library, to define calling sequences of the selected serverless computing functions, to set operation permissions of the collaborative subjects, and to generate a collaborative computing task. The serverless computing scheduling module is configured to allocate resources to corresponding serverless computing functions when the serverless computing functions are triggered by using a priority queue algorithm in combination with metadata of the serverless computing functions, to be processed data volume, and historical execution data in the directed acyclic graph. The computing result processing module is configured to aggregate execution results of the serverless computing functions according to the directed acyclic graph, and to send the aggregated results to terminals of corresponding collaborative subjects according to permissions of the collaborative subjects. The data storage module is configured to store the space science data units, the function library, task execution logs, and aggregated results.

7. The system of claim 6, wherein, The data storage module is configured to realize fast access to hot data and low-cost storage of cold data according to data life cycle management requirements.

Images