Star catalog data management methods and star catalog data management system

Abstract

This application relates to a star table data management method and a star table data management system. The star table data management method includes: responding to a file receiving request, receiving a star table file and a column description file, parsing the star table file and column description file to obtain star table metadata, and sending a star table import request; wherein, the star table file stores star table data, the column description file stores information describing each column in the star table file, and the star table metadata includes identification information describing the star table file and structural information describing each column in the star table file; responding to the star table import request, obtaining a data synchronization configuration based on the star table metadata; wherein, the star table import request includes star table metadata; and reading the star table data according to the data structure of the target data table indicated by the data synchronization configuration, and synchronizing the star table data to the target data table in the star table data platform. The star table data management method of this application improves the efficiency of star table data management and flow.

Description

Technical Field

[0001] This application relates to the field of astronomical data management, and more particularly to a method and system for managing star catalog data. Background Technology

[0002] With the development of astronomical observation and big data technologies, star catalogs, as the core carriers of astronomical data, play a crucial role in scientific research, astronomical observation and analysis. Star catalog data is typically characterized by its massive scale and complex feature dimensions. Star catalog metadata, as the "descriptive layer" of star catalog data, plays a key supporting role in the parsing, storage, calculation, and retrieval of star catalogs.

[0003] However, as astronomical research demands star catalog data towards "multi-source integration, real-time response, efficient computation, and collaborative retrieval," related star catalog data management technologies often rely on individual operational systems or observation platforms maintaining the data separately. This data is typically stored in file formats or distributed database tables. The star catalog file structure and column information are generally parsed and managed within each system using manually configured field description documents, parsing scripts, or local configuration tables. When star catalogs are used across systems, data copying or format conversion at the interface level is often used. Each system independently interprets, stores, and retrieves star catalog metadata based on its own configuration, resulting in star catalog data management efficiency that fails to meet the requirements of complex application scenarios. Summary of the Invention

[0004] This application provides a star catalog data management method and a star catalog data management system to address some or all of the shortcomings in related technologies.

[0005] According to a first aspect of the embodiments of this application, a star table data management method is provided, comprising: The system receives a response file receiving request, receives a star table file and a column description file, parses the star table file and the column description file to obtain star table metadata, and sends a star table import request; wherein, the star table file stores star table data, the column description file stores information describing each column in the star table file, and the star table metadata includes identification information describing the star table file and structural information describing each column in the star table file; In response to the star table import request, the data synchronization configuration is obtained based on the star table metadata; wherein, the star table import request includes the star table metadata; According to the data structure of the target data table indicated by the data synchronization configuration, the star table data is read and synchronized to the target data table in the star table data platform.

[0006] Optionally, the structure information includes star table column names and star table column data types, and the identification information includes the star table data entry status; wherein, the star table column names are used to represent the names of each column in the star table file, the star table column data types are used to represent the data types of each column in the star table file, and the number of star table column names and star table column data types are the same and correspond one-to-one; The parsing of the star table file and the column description file to obtain star table metadata includes: Parse the star table file and the column description file to obtain star table metadata other than the star table data entry status; In the case of missing data types in the star table column, the star table column name corresponding to the missing star table column data type is taken as the target star table column name. A similarity search is performed in the historical star table metadata to obtain the star table column name with the highest similarity to the target star table column name as the approximate star table column name. The data type of the star table column corresponding to the approximate star table column name is used as the derivation result of the missing star table column data type. The historical star table metadata is stored in the metadata database. After obtaining the derivation results of all missing star table column data types, the completed star table metadata is used as the star table metadata. After obtaining the star catalog metadata, the following is included: Write the star table metadata into the metadata database.

[0007] Optionally, the step of reading the star table data according to the data structure of the target data table indicated by the data synchronization configuration, and synchronizing the star table data to the target data table in the star table data platform includes: A star table entry workflow is generated based on the data synchronization configuration; wherein, the star table entry workflow defines multiple task nodes and the dependencies between the multiple task nodes, and each task node is used to represent a task; Based on the dependencies, the target task node to be executed is determined, and the task represented by the target task node is executed.

[0008] Optionally, the identification information includes the name of the star table file, the data synchronization configuration includes star table structure information, and obtaining the data synchronization configuration based on the star table metadata includes: determining the star table structure information based on the name of the star table file, the star table column name, and the star table column data type; The task node includes a data table generation node, and the task represented by the data table generation node is to generate the target data table on the star table data platform according to the star table table structure information.

[0009] Optionally, the task node further includes: The synchronous task node is used to represent the task of writing the star table data into the target data table. The updated successful task node represents the task of updating the field of the star table entry status to "success" in the metadata database. Update the failed task node, the task to be represented is: in the meta database, update the field of the star table entry status to failed; The decision task node is used to represent the task of determining whether the update task node was successfully executed or the update task node failed, based on the execution result of the synchronization task node.

[0010] Optionally, the star catalog data platform includes big data computing services and a data warehouse, and the star catalog data management method further includes: Based on the star table metadata, determine the storage location of the star table data that has been successfully written into the first target data table in the big data computing service and the data structure of the first target data table, and in response to the cross-matching service request, trigger a cross-matching service for the star table data in the big data computing service based on the star table data in the big data computing service and the star table metadata. Based on the star table metadata, the storage location of the star table data successfully written into the second target data table in the data warehouse and the data structure of the second target data table are determined. In response to a retrieval service request, a retrieval service for the star table data in the data warehouse is triggered based on the star table data in the data warehouse and the star table metadata.

[0011] Optionally, the star table entry workflow defines task nodes that are executed sequentially, including: a first synchronization task node, a first decision task node, a first update failure task node, a second synchronization task node, a second decision task node, a first update success task node, and a second update failure task node. The task represented by the first synchronization task node is: to write the star table data into the first target data table; The first decision task node is used to represent the following task: determine whether to execute the first update failure task node based on the execution result of the first synchronization task node; when the first synchronization task node fails to execute, the first update failure task node is triggered to be executed. The task represented by the second synchronization task node is: to write the star table data from the first target table to the second target table; The second decision task node is used to represent the following task: based on the execution result of the second synchronization task node, it selects between the first successful update task node and the second failed update task node. When the second synchronization task node is executed successfully, it triggers the execution of the first successful update task node. When the second synchronization task node fails, it triggers the execution of the second failed update task node. The step of synchronizing the satellite data to the target data table in the satellite data platform includes: writing the satellite data into the first target data table and the second target data table in a serial manner.

[0012] Optionally, the star table entry workflow defines task nodes that are executed in parallel order, including: a first synchronous task node, a second synchronous task node, the decision task node, the update failure task node, and the update success task node; The task represented by the first synchronization task node is: to write the star table data into the first target data table; The task represented by the second synchronization task node is: to write the star table data into the second target data table; The decision task node is used to represent the following tasks: branch selection based on the execution results of the first synchronization task node and the second synchronization task node; when the first synchronization task node or the second synchronization task node fails to execute, the update failure task node is triggered; when both the first synchronization task node and the second synchronization task node execute successfully, the update success task node is triggered. The step of synchronizing the satellite data to the target data table in the satellite data platform includes: writing the satellite data into the first target data table and the second target data table in parallel.

[0013] Optionally, if there are failed task nodes, in response to a log viewing request for the failed task node, the failure log result corresponding to the failed task node is returned; and In response to a retry request for the failed task node, the failed task node is re-executed.

[0014] According to a second aspect of the embodiments of this application, a star table data management system is provided, comprising: The star table module is used to respond to file receiving requests, receive star table files and column description files, parse the star table files and column description files to obtain star table metadata, and send star table import requests; wherein, the star table files store star table data, the column description files store information describing each column in the star table files, and the star table metadata includes identification information describing the star table files and structural information describing each column in the star table files; The file storage module is used to receive the star table file and the column description file transmitted by the star table module; The workflow scheduling module is used to respond to the star table entry request transmitted by the star table module and obtain the data synchronization configuration based on the star table metadata; wherein, the star table entry request includes the star table metadata; The data synchronization module is used to read the star table data from the file storage module according to the data structure of the target data table indicated by the data synchronization configuration, and synchronize the star table data to the target data table in the star table data platform.

[0015] Optionally, the star table data management system further includes a metadata database, which stores historical star table metadata. The structure information includes star table column names and star table column data types, and the identification information includes the star table data entry status. The star table column names are used to represent the names of each column in the star table file, and the star table column data types are used to represent the data types of each column in the star table file. The number of star table column names and star table column data types are the same and they correspond one-to-one. The star catalog module is specifically used for: Parse the star table file and the column description file to obtain star table metadata other than the star table data entry status; In the case of missing data types in the star table column, the star table column name corresponding to the missing star table column data type is taken as the target star table column name. A similarity search is performed in the historical star table metadata to obtain the star table column name with the highest similarity to the target star table column name as the approximate star table column name. The star table column data type corresponding to the approximate star table column name is used as the derivation result of the missing star table column data type. After obtaining the derivation results of all missing star table column data types, the completed star table metadata is used as the star table metadata. The star catalog module is also used for: Write the star table metadata into the metadata database.

[0016] Optionally, the workflow scheduling module is further configured to generate a star table entry workflow based on the data synchronization configuration; wherein, the star table entry workflow defines multiple task nodes, the dependencies between the multiple task nodes, and an execution module corresponding to each task node, each task node is used to represent a task, and the execution module is used to execute the task represented by its corresponding task node. The workflow scheduling module is also used to determine the target task node to be executed and the target execution module corresponding to the target task node according to the dependency relationship, and to issue an execution instruction to the target execution module to trigger the target execution module to execute the task represented by the target task node.

[0017] Optionally, the task node includes: The synchronous task node is used to represent the task of writing the star table data into the target data table. The updated successful task node represents the task of updating the field of the star table entry status to "success" in the metadata database. Update the failed task node, the task to be represented is: in the meta database, update the field of the star table entry status to failed; The decision task node is used to represent the task of determining whether the update task node was successfully executed or the update task node failed, based on the execution result of the synchronization task node. The execution module corresponding to the synchronization task node is the data synchronization module, the execution modules corresponding to the successful update task node and the failed update task node are the star table module, and the execution module corresponding to the decision task node is the workflow scheduling module.

[0018] Optionally, the star table data platform includes big data computing services and a data warehouse, and the star table data management system further includes: The cross-matching module is used to determine the storage location of the star table data that has been successfully written into the first target data table in the big data computing service and the data structure of the first target data table based on the star table metadata, and in response to the cross-matching service request, to trigger the provision of cross-matching service for the star table data in the big data computing service based on the star table data and the star table metadata. The retrieval module is used to determine the storage location of the star table data that has been successfully written into the second target data table in the data warehouse and the data structure of the second target data table based on the star table metadata, and in response to a retrieval service request, to trigger a retrieval service for the star table data in the data warehouse based on the star table data and the star table metadata.

[0019] The technical solutions provided by the embodiments of this application may include the following beneficial effects: The star table data management method of this application takes star table metadata and data synchronization configuration as its core, and integrates "understanding the star table structure" and "writing star table data into the target data table" into a unified configuration process: When responding to a file receiving request, the star table file and column description file are received, parsed, and star table metadata including star table identification information and column structure information is generated; when responding to a star table inbound request, the star table metadata is obtained from the star table inbound request, and a data synchronization configuration is generated based on the star table metadata to indicate the structure of the target data table in the star table data platform; then, according to the data structure of the target data table indicated by the data synchronization configuration, the star table data in the star table file is read, and the star table data is synchronously written into the target data table in the star table data platform.

[0020] Compared to other satellite catalog data management technologies, where satellite catalog data is typically maintained independently by each observation platform or usage environment, stored primarily in files or distributed database tables, and where the structure of the catalog file and its column information rely on separate field description documents, parsing scripts, or local configuration tables for parsing and target table mapping, requiring the repeated writing and maintenance of multiple parsing and database configurations for the same catalog at different stages, this application generates unique satellite catalog metadata and data synchronization configurations through a single parsing of the catalog file and column description file. This solidifies the logic of "what this catalog looks like" and "which target table should be written to, and how each column corresponds" into reusable configuration rules. These metadata and synchronization configurations are then used throughout the catalog's database entry and subsequent transfer processes. Under the same data management mechanism based on metadata and synchronization configurations, satellite catalog data management and multi-storage collaboration are simultaneously supported. This reduces the workload of manual configuration and script maintenance, achieving decoupled collaboration between storage, computing, and services based on metadata, thereby improving the overall efficiency of satellite catalog data management and transfer in complex application scenarios.

[0021] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this application. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 This is a flowchart illustrating a star catalog data management method according to an embodiment of this application; Figure 2 yes Figure 1 The flowchart of step S101 in the star table data management method is shown. Figure 3 yes Figure 1 The flowchart of step S103 in the star table data management method is shown. Figure 4 This is a flowchart illustrating a star catalog data management method according to an embodiment of this application; Figure 5 This is a schematic diagram illustrating a star table import workflow according to an embodiment of this application; Figure 6 This is a schematic diagram illustrating another star table entry workflow according to an embodiment of this application; Figure 7 This is a schematic diagram of a star table data management system according to an embodiment of this application. Detailed Implementation

[0024] The technical solutions in the embodiments of this application will be clearly and completely described herein with reference to the accompanying drawings. In the following description, when referring to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements.

[0025] The terms "first" and "second" used in the embodiments of this application are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0026] This application provides a star catalog data management method and a star catalog data management system. The star catalog data management method and system of this application will be described in detail below with reference to the accompanying drawings. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0027] Figure 1 This application illustrates a method for managing star catalog data according to an exemplary embodiment. For example... Figure 1 As shown, the star catalog data management method includes the following steps: S101, Response file receiving request, receive star table file and column description file, parse star table file and column description file to obtain star table metadata, and send star table import request.

[0028] The star catalog file stores star catalog data, where each row corresponds to an independent celestial object, and each column corresponds to a celestial feature of that object. The column description file stores information describing each column in the star catalog file. Each row in the column description file describes one column, and each column in the column description file represents the corresponding star catalog column metadata. This metadata includes at least the column name and data type, indicating the name of the celestial feature and its data type within the star catalog data. The star catalog metadata includes identification information describing the star catalog file and structural information describing each column. The identification information may include the name of the star catalog file. After receiving the star catalog file and column description file, they can be stored in the file storage module.

[0029] It should be noted that, in the various embodiments of this specification, the specific file formats of the star table file and column description file are not limited, and can be any file format suitable for star table data storage and description, such as CSV (Comma-Separated Values).

[0030] In one optional embodiment, the structure information includes the star table column names and star table column data types, and the identification information includes the star table data entry status. The star table column names represent the names of each column in the star table file, and the star table column data types represent the data types of each column in the star table file. The number of star table column names and star table column data types are the same and they correspond one-to-one. For example... Figure 2 As shown, step S101 includes the following steps: S1011. Parse the star table file and column description file to obtain star table metadata other than the star table data entry status.

[0031] S1012. When a data type in the star list column is missing, the star list column name corresponding to the missing data type is taken as the target star list column name. A similarity search is performed in the historical star list metadata to obtain the star list column name with the highest similarity to the target star list column name as the approximate star list column name. The data type of the star list column corresponding to the approximate star list column name is used as the derivation result of the missing star list column data type. The historical star list metadata is stored in the metadata database.

[0032] S1013. After obtaining the derivation results of all missing star table column data types, the completed star table metadata is used as the star table metadata.

[0033] The metadata of star table columns is the foundation for star table data parsing, synchronization, and subsequent use. However, in practical applications, user-uploaded column description files often contain missing metadata such as star table column data types. This embodiment uses a derivation mechanism based on historical star table metadata to automatically complete missing star table column data types. On the one hand, this derivation mechanism avoids the inefficiency and error-proneness of relying on manual completion, significantly improving the processing efficiency of star table data entry. On the other hand, reusing professionally constructed public star table metadata resources as the basis for derivation helps ensure the rationality and consistency of the completion results, providing reliable structured metadata support for the entire chain of subsequent star table data synchronization, cross-matching services, and retrieval services, thereby further optimizing the overall operational efficiency and stability of star table data management.

[0034] Step S101 is followed by the step of writing the star table metadata into the metadata database.

[0035] In specific implementations, the star table metadata can be stored in the metadata database in two separate table structures: a star table metadata table and a star table column metadata table. The star table metadata table stores identification information describing the star table files. Besides fields recording the data entry status, it includes a star table number to uniquely identify the file and a star table name. The star table column metadata table stores structural information describing each column in the star table file. Besides the column name and data type, it includes a column number and a star table number field. The column number field indicates the star table to which the column belongs and corresponds to the star table number in the star table metadata table. Specifically, the similarity search in step S1012 is performed in the public history of the star table column metadata tables already stored in the metadata database.

[0036] It should be noted that, in the various embodiments of this specification, the star table number can be used as an identifier field in the star table metadata table to uniquely identify the star table, and the star table name can also be set as a unique identifier within a preset management scope. The star table column number is used to uniquely identify a star table column metadata record in the star table column metadata table. In an optional embodiment, a user ID field can also be added to the star table metadata table to indicate the user to whom the star table belongs, so as to support star table management in a multi-user environment.

[0037] S102. Respond to the Star Table entry request and obtain the data synchronization configuration based on the Star Table metadata.

[0038] The star table import request includes star table metadata, and the data synchronization configuration may include star table creation structure information. In an optional embodiment, step S102 may include the step of determining the star table creation structure information based on the star table file name, star table column name, and star table column data type.

[0039] S103. Based on the data structure of the target data table indicated by the data synchronization configuration, read the star table data and synchronize the star table data to the target data table in the star table data platform.

[0040] The star table data management method of this application takes star table metadata and data synchronization configuration as its core, and integrates "understanding the star table structure" and "writing star table data into the target data table" into a unified configuration process: When responding to a file receiving request, the star table file and column description file are received, parsed, and star table metadata including star table identification information and column structure information is generated; when responding to a star table inbound request, the star table metadata is obtained from the star table inbound request, and a data synchronization configuration is generated based on the star table metadata to indicate the structure of the target data table in the star table data platform; then, according to the data structure of the target data table indicated by the data synchronization configuration, the star table data in the star table file is read, and the star table data is synchronously written into the target data table in the star table data platform.

[0041] Compared to other satellite catalog data management technologies, where satellite catalog data is typically maintained independently by each observation platform or usage environment, stored primarily in files or distributed database tables, and where the structure of the catalog file and its column information rely on separate field description documents, parsing scripts, or local configuration tables for parsing and target table mapping, requiring the repeated writing and maintenance of multiple parsing and database configurations for the same catalog at different stages, this application generates unique satellite catalog metadata and data synchronization configurations through a single parsing of the catalog file and column description file. This solidifies the logic of "what this catalog looks like" and "which target table should be written to, and how each column corresponds" into reusable configuration rules. These metadata and synchronization configurations are then used throughout the catalog's database entry and subsequent transfer processes. Under the same data management mechanism based on metadata and synchronization configurations, satellite catalog data management and multi-storage collaboration are simultaneously supported. This reduces the workload of manual configuration and script maintenance, achieving decoupled collaboration between storage, computing, and services based on metadata, thereby improving the overall efficiency of satellite catalog data management and transfer in complex application scenarios.

[0042] In an alternative embodiment, such as Figure 3 As shown, step S103 includes the following steps: S1031. Generate a star table entry workflow based on data synchronization configuration.

[0043] The star table import workflow defines multiple task nodes and the dependencies between them, with each task node representing a task.

[0044] S1032. Based on the dependencies, determine the target task node to be executed and execute the task represented by the target task node.

[0045] Specifically, if a task node fails to execute, subsequent task nodes that depend on the successful execution of that task node will no longer be scheduled for execution in the current star table entry workflow.

[0046] Specifically, the star catalog import workflow can adopt a workflow orchestration method based on a Directed Acyclic Graph (DAG). Each task node in the star catalog import workflow corresponds to a node in the DAG, and the dependencies between task nodes are represented by directed edges in the DAG. Based on this DAG workflow scheduling mechanism, the process of synchronizing star catalog data from parsing to one or more target data tables in the star catalog data platform can be modeled. Each processing step in the star catalog import workflow is flexibly defined in the form of task nodes and the dependencies between them, and it supports scheduling and executing task nodes serially or in parallel. In practical implementation, the execution order and concurrency of task nodes can be dynamically adjusted based on operating parameters such as the amount of star catalog data and task priority to adapt to the processing needs of star catalog data of different scales.

[0047] By adopting a task orchestration method based on directed acyclic graphs, circular dependencies between tasks can be avoided, ensuring that the execution of tasks corresponding to all preceding task nodes is triggered only after all preceding task nodes have been completed. This improves the scheduling efficiency and execution stability of the star table entry workflow while ensuring task dependency constraints.

[0048] During the synchronization of star table data from file storage to one or more target data tables in the star table data platform, data synchronization-related task nodes can be executed in a distributed manner within the star table inbound workflow. Specifically, the distributed approach can be achieved by breaking down the overall synchronization process into several synchronization sub-steps undertaken by multiple task nodes, and by utilizing the parallel scheduling and execution of task nodes that are independent of each other in a directed acyclic graph within the same star table inbound workflow. By simultaneously advancing synchronization operations for different data ranges or target data tables on multiple parallel task nodes, the processing bottleneck of sequential execution of a single task node can be overcome, significantly improving the overall synchronization efficiency of massive star table data and shortening the time from the completion of uploading star table files and column description files to the star table data being available for subsequent processing in the star table data platform. Simultaneously, by monitoring the execution results of task nodes and establishing a mechanism to prevent the scheduling or retry of failed task nodes, when a data synchronization-related task node fails, subsequent task nodes in its dependency chain can be blocked or corresponding retry processing can be triggered, preventing synchronization process interruption or further error propagation and ensuring the reliability of the star table data synchronization process. Furthermore, by splitting the overall synchronization process into multiple independently schedulable task nodes and adjusting the number of parallel task nodes in the same star table ingestion workflow based on the amount of star table data and task priority, the parallelism can be flexibly increased as the star table data scale grows to adapt to the synchronization needs of star table data of different scales, providing a timely and complete star table data foundation for subsequent cross-matching and retrieval services based on the star table data platform.

[0049] In an optional embodiment, the task node may include a data table generation node, which represents the task of generating a target data table on the star table data platform based on the star table building structure information.

[0050] Task nodes may further include: synchronization task nodes, successful update task nodes, failed update task nodes, and decision task nodes. The synchronization task node represents the task of writing the star table data into the target data table. The successful update task node represents the task of updating the field of the star table's entry status to "successful" in the metadata database. The failed update task node represents the task of updating the field of the star table's entry status to "failed" in the metadata database. The decision task node represents the task of determining whether to execute the successful update task node or the failed update task node based on the execution result of the synchronization task node.

[0051] In the event of a failure to insert data into the StarTable, the corresponding data entry failure log information can be recorded in the StarTable metadata table, and the StarTable data entry status can be updated to identify the cause and related status of the data entry anomaly. This helps users locate and analyze the data entry anomaly based on the StarTable metadata table and take corresponding corrective measures accordingly.

[0052] In one alternative embodiment, the star table data platform includes big data computing services and a data warehouse, such as Figure 4 As shown, the star catalog data management method also includes the following steps: S104. Based on the star table metadata, determine the storage location of the star table data that has been successfully written into the first target data table in the big data computing service and the data structure of the first target data table, and in response to the cross-matching service request, trigger the cross-matching service for the star table data in the big data computing service based on the star table data and star table metadata in the big data computing service.

[0053] S105. Based on the star table metadata, determine the storage location of the star table data that has been successfully written into the second target data table in the data warehouse and the data structure of the second target data table, and in response to the retrieval service request, trigger a retrieval service for the star table data in the data warehouse based on the star table data and star table metadata in the data warehouse.

[0054] Specifically, the star tables corresponding to the star table data that have been successfully written into the big data computing service and data warehouse can be divided into two categories: one is the star table uploaded and entered into the database through steps S101 to S103, and the other is a large number of public star tables that are pre-imported during the initialization or operation of the star table data platform. Their corresponding star table metadata is also stored in the metadata database and participates in subsequent cross-matching and retrieval processing.

[0055] According to the star catalog data management method provided in this embodiment, the same star catalog data can be distributed and synchronized as needed among file storage, big data computing services, and data warehouses.

[0056] Specifically, after parsing the star table file and column description file, on the one hand, the star table file and column description file are stored centrally in the form of raw files by the file storage, serving as a unified data source for subsequent re-entry into the database or troubleshooting; on the other hand, the star table data, based on the star table creation structure information recorded in the star table metadata, is written into the first target data table in the big data computing service and the second target data table in the data warehouse through the star table entry workflow.

[0057] The big data computing service fully leverages the advantages of distributed computing of massive data. Based on the star table data stored in the big data computing service and combined with the star table metadata stored in the metadata database, it can build distributed computing capabilities for batch cross-matching processing between different star tables, providing upper-layer business with computing and data processing capabilities to perform cross-matching operations on massive star table data.

[0058] On the data warehouse side, the structured storage and query optimization capabilities of the data warehouse can be utilized to build multi-dimensional structured query and indexing capabilities for star table data based on the star table data stored in the data warehouse and combined with the star table metadata stored in the metadata database. This provides multi-dimensional fast filtering and low-latency response of star table data for various business retrieval requests, and supports filtering star table data according to various conditions such as star table name, column fields, and observation time interval.

[0059] By centralizing the computing power supporting cross-matching processing on the big data computing service side and the query power supporting retrieval processing on the data warehouse side, the "computing-storage" functions of computationally intensive processing and query-intensive processing are decoupled and complemented. This avoids resource contention and performance conflicts caused by a single storage engine simultaneously bearing computationally intensive and query-intensive tasks such as cross-matching processing and retrieval processing, thereby significantly improving the overall processing efficiency and operational stability of the star table cross-matching service and star table retrieval service provided for all successfully entered star tables.

[0060] In an optional embodiment, synchronizing the satellite catalog data to the target data table in the satellite catalog data platform in step S103 includes: writing the satellite catalog data into the first target data table and the second target data table in a serial manner.

[0061] like Figure 5 As shown, the Star Table import workflow defines task nodes that are executed sequentially, including: first synchronization task node, first decision task node, first update failure task node, second synchronization task node, second decision task node, first update success task node, and second update failure task node. Figure 5 Each box represents a task node, and the arrows between the boxes indicate the dependencies between the task nodes.

[0062] The first synchronization task node represents the task of writing star table data into the first target data table. The first decision task node represents the task of determining whether to execute the first update failure task node based on the execution result of the first synchronization task node; if the first synchronization task node fails, it triggers the execution of the first update failure task node. The second synchronization task node represents the task of writing star table data from the first target table into the second target table. The second decision task node represents the task of selecting between the first update success task node and the second update failure task node based on the execution result of the second synchronization task node; if the second synchronization task node succeeds, it triggers the execution of the first update success task node; if the second synchronization task node fails, it triggers the execution of the second update failure task node.

[0063] In an optional embodiment, synchronizing the satellite catalog data to the target data table in the satellite catalog data platform in step S103 includes writing the satellite catalog data to the first target data table and the second target data table in parallel.

[0064] like Figure 6 As shown, the star table import workflow defines task nodes that are executed in parallel order, including: the first synchronization task node, the second synchronization task node, the decision task node, the update failure task node, and the update success task node. Figure 6 Each box represents a task node, and the arrows between the boxes indicate the dependencies between the task nodes.

[0065] The first synchronous task node represents the task of writing star table data into the first target data table. The second synchronous task node represents the task of writing star table data into the second target data table. The decision task node represents the task of selecting a branch based on the execution results of the first and second synchronous task nodes. When either the first or second synchronous task node fails, the update failure task node is triggered; when both the first and second synchronous task nodes succeed, the update success task node is triggered.

[0066] According to the star table data management method provided in this embodiment, the parsing of star table metadata and data synchronization configuration are used as the core control information, and the parsing of star table files and column description files, the generation and storage of star table metadata, the writing of star table data into the star table data platform, and the subsequent cross-matching and retrieval services are linked together into one.

[0067] After parsing the star table file and column description file, the star table file and column description file are stored as raw files in the file storage. The star table metadata is written to the metadata database, and a data synchronization configuration is generated based on the star table metadata. The star table data is then synchronized to the first target data table on the big data computing service side and the second target data table on the data warehouse side of the star table data platform, respectively, according to the data structure of the target data table indicated by the data synchronization configuration, through the star table inbound workflow.

[0068] The big data computing service side leverages its distributed computing capabilities and combines them with the star table metadata in the metadata database to build batch computing capabilities for star table data cross-matching services; the data warehouse side utilizes structured storage and query optimization capabilities, combined with the star table metadata in the metadata database, to build multi-dimensional structured query and indexing capabilities for star table data, providing star table retrieval services with multi-dimensional fast filtering and low-latency response of star table data.

[0069] Thus, under the constraints of unified star table metadata and data synchronization configuration, unified management of star table data from reception, parsing, and storage to collaborative flow between big data computing services and data warehouses is achieved. This also enables the functional division of cross-matching and retrieval services on different storage sides. It supports unified storage of multi-source star tables and efficient flow between multiple storage systems, while avoiding resource contention and performance conflicts caused by a single storage engine simultaneously handling computationally intensive and query-intensive tasks. This improves the automation and processing efficiency of star table data management, enhances the flexibility of star table data usage in different business scenarios, and improves the overall reliability of star table data management and services.

[0070] In an alternative embodiment, such as Figure 4 As shown, the star catalog data management method also includes the following steps: S106. In the case of a failed task node, in response to a log viewing request for the failed task node, return the failure log result corresponding to the failed task node.

[0071] S107. In response to a retry request for a failed task node, re-execute the failed task node.

[0072] By providing failure logs and retry mechanisms for failed task nodes, the reliability and operational efficiency of the star table data management process in this application can be significantly improved. The failure logs record key information related to the failed task node, such as execution parameters, error type, and timestamps, facilitating quick identification of anomalies during star table import or synchronization, such as abnormal data format or insufficient resources, avoiding blind troubleshooting of the star table import workflow. Retry requests for failed task nodes allow for re-execution without restarting the entire star table import workflow, and can further trigger the re-execution of subsequent dependent task nodes when necessary. This saves computational resources and processing time while ensuring the continuity of key stages such as star table import and synchronization, effectively reducing the risk of process interruption due to a single task failure. Through the combination of failure logs and retry mechanisms, the star table data management method in this application possesses stronger fault tolerance and problem localization and recovery capabilities in complex scenarios involving massive amounts of star table data, which is beneficial for improving service stability and ensuring the integrity of star table data processing results.

[0073] Based on the same inventive concept as the star catalog data management method, embodiments of this application also provide a star catalog data management system, such as... Figure 7 As shown, the star table data management system 1 includes: star table module 2, file storage module 3, workflow scheduling module 4, data synchronization module 5, and star table data platform 6.

[0074] The Star Table module 2 is used to respond to file reception requests, receive the Star Table file and column description file, parse the Star Table file and column description file to obtain Star Table metadata, and send Star Table import requests.

[0075] The star table file stores star table data, the column description file stores information describing each column in the star table file, and the star table metadata includes identification information describing the star table file and structural information describing each column in the star table file.

[0076] File storage module 3 is used to receive the star table file and column description file transmitted by star table module 2.

[0077] Workflow scheduling module 4 responds to the star table import request transmitted by the star table module and obtains the data synchronization configuration based on the star table metadata. The star table import request includes the star table metadata. The data synchronization module 5 is used to read the star table data from the file storage module 3 according to the data structure of the target data table indicated by the data synchronization configuration, and synchronize the star table data to the target data table in the star table data platform 6.

[0078] In one alternative embodiment, such as Figure 7 As shown, the star table data management system also includes a metadata database 7. The metadata database 7 stores historical star table metadata, with structural information including star table column names and star table column data types, and identification information including the star table data entry status. Specifically, the star table column names represent the names of each column in the star table file, and the star table column data types represent the data types of each column in the star table file. The number of star table column names and star table column data types are the same and they correspond one-to-one.

[0079] The StarTable module 2 is specifically used for: parsing the StarTable file and column description file to obtain StarTable metadata other than the StarTable data entry status; when there are missing StarTable column data types, using the StarTable column name corresponding to the missing StarTable column data type as the target StarTable column name, performing a similarity search in the historical StarTable metadata to obtain the StarTable column name with the highest similarity to the target StarTable column name as the approximate StarTable column name, and using the StarTable column data type corresponding to the approximate StarTable column name as the derivation result of the missing StarTable column data type; after obtaining the derivation results of all missing StarTable column data types, using the completed StarTable metadata as the StarTable metadata. After obtaining the complete StarTable metadata, the StarTable module 2 is also used to write the StarTable metadata into the metadata database 7.

[0080] In an optional embodiment, the workflow scheduling module 4 is further configured to generate a star table entry workflow based on data synchronization configuration. The star table entry workflow defines multiple task nodes, dependencies between these task nodes, and an execution module corresponding to each task node. Each task node represents a task, and the execution module executes the task represented by its corresponding task node. The workflow scheduling module 4 is also configured to determine the target task node to be executed and its corresponding target execution module based on the dependencies, and issue an execution instruction to the target execution module to trigger it to execute the task represented by the target task node.

[0081] In one alternative embodiment, the task nodes include: a synchronization task node, a successful update task node, a failed update task node, and a decision task node.

[0082] The synchronization task node represents the task of writing the star table data into the target data table. The successful update task node represents the task of updating the field indicating successful entry into the star table in metadata database 7. The failed update task node represents the task of failing to update the field indicating failed entry into the star table in metadata database 7. The decision task node represents the task of determining whether to execute the successful update task node or the failed update task node based on the execution result of the synchronization task node.

[0083] Among them, the execution module corresponding to the synchronization task node is the data synchronization module 5, the execution module corresponding to the successful update task node and the failed update task node is the star table module 2, and the execution module corresponding to the decision task node is the workflow scheduling module 4.

[0084] In one alternative embodiment, such as Figure 7 As shown, the Star Table data platform 6 includes a big data computing service 61 and a data warehouse 62. The Star Table data management system also includes a cross-matching module 8 and a retrieval module 9.

[0085] The cross-matching module 8 is used to determine the storage location and data structure of the first target data table that has been successfully written into the big data computing service 61 based on the star table metadata, and in response to the cross-matching service request, to trigger the cross-matching service for the star table data in the big data computing service 61 based on the star table metadata.

[0086] The retrieval module 9 is used to determine the storage location and data structure of the second target data table that has been successfully written into the data warehouse 62 based on the star table metadata, and in response to the retrieval service request, to trigger a retrieval service for the star table data in the data warehouse 62 based on the star table data and star table metadata.

[0087] When there is a failed task node in the star table inbound workflow, the workflow scheduling module 4 can return the failure log result corresponding to the failed task node when responding to the log viewing request for the failed task node; when responding to the retry request for the failed task node, it can re-execute the failed task node.

[0088] like Figure 7 As shown, to facilitate understanding of the overall architecture of the Star Table Data Management System 1 proposed in this application, the Star Table Data Management System 1 can be understood from the perspective of logical function division as including three parts: business layer 101, data bus layer 102, and storage layer 103.

[0089] The business layer 101 includes the star list module 2, the retrieval module 9, and the cross-matching module 8.

[0090] The star table module 2 responds to the file receiving request, receives the star table file and column description file, transmits the star table file and column description file to the file storage module 3 for storage, parses and derives the star table file and column description file to obtain star table metadata, writes the star table metadata into the metadata database 7, and sends it to the workflow scheduling module 4 along with the star table entry request to trigger star table entry and star table metadata update.

[0091] The retrieval module 9 provides efficient structured query services for successfully imported star tables based on the star table data in the data warehouse 62 and combined with the star table metadata stored in the metadata database 7.

[0092] The cross-matching module 8 relies on the star table data in the big data computing service 61 and combines the star table metadata stored in the metadata database 7 to carry out cross-matching and other related services for the star tables that have been successfully entered into the database, which are oriented towards massive star table data.

[0093] The data bus layer 102 includes a workflow scheduling module 4 and a data synchronization module 5.

[0094] Workflow scheduling module 4 responds to the Star Table entry request, improves the data synchronization configuration based on the Star Table metadata, defines the Star Table entry workflow, sets up multiple task nodes corresponding to various task types in the Star Table entry workflow, and establishes dependencies between multiple task nodes. The Star Table entry workflow can be organized in the form of DAG and is scheduled and executed by workflow scheduling module 4. It also supports viewing the logs of failed tasks and re-executing them when task execution fails.

[0095] The storage layer 103 includes a file storage module 3, a big data computing service 61, a metadata database 7, and a data warehouse 62.

[0096] File storage module 3 is used to store the uploaded star table files and column description files, serving as a unified original data source for star table data.

[0097] Big Data Computing Service 61 provides distributed computing and storage capabilities for cross-matching module 8 for massive star catalog data.

[0098] Metadatabase 7 stores the metadata of the star table, including the star table metadata table and the star table column metadata table, providing unified metadata support for star table entry, retrieval and cross-matching.

[0099] Data warehouse 62 stores star table data for retrieval services, providing efficient structured storage and query support for star table data for retrieval module 9.

[0100] At the system architecture level, the data bus, consisting of the workflow scheduling module 4 and the data synchronization module 5, can combine with the star table inbound workflow defined in the form of DAG to uniformly orchestrate and schedule the processes of receiving, parsing, and inbounding star table data, as well as the subsequent synchronization and service processes in the big data computing service 61 and the data warehouse 62. Through a layered and decoupled architecture, the business layer and the storage layer 103 are structurally decoupled, so that the business layer and the storage layer 103 no longer directly call each other, but instead connect to the data bus through star table metadata and data synchronization configuration respectively. Under the premise that the star table metadata and data synchronization configuration mechanism remain unchanged, the specific modules in each layer can be independently extended and iterated.

[0101] Meanwhile, the data bus acts as a "glue" between business needs and underlying storage and computing resources. On the one hand, it leverages the Star Table entry workflow to flexibly orchestrate complex entry and synchronization processes and automatically link Star Table metadata and Star Table data management. On the other hand, it promotes the mutual complementarity between the cross-matching service capabilities of the Big Data Computing Service 61 side and the retrieval service capabilities of the Data Warehouse 62 side, and provides a monitorable and retryable process control mechanism, thereby providing core support for the efficient management, flexible flow and reliable service of Star Table data.

[0102] The above are merely embodiments of this application, used to help understand the system architecture and working mechanism of this application, and are not intended to limit this application. For those skilled in the art, all modifications, equivalent substitutions, and improvements made without departing from the spirit and substance of this application should fall within the scope of protection of the claims of this application.

[0103] Each module of the aforementioned star table data management system corresponds to the steps of the aforementioned star table data management method. For details on the implementation process of the functions and roles of each module in the aforementioned star table data management system, please refer to the implementation process of the corresponding steps in the aforementioned star table data management method. The same technical effect can be achieved, and will not be repeated here.

[0104] An electronic device provided in this application includes one or more processors for implementing the above-described star table data management method.

[0105] In some embodiments, the electronic device may include a storage medium. For example, a computer-readable storage medium may store a program that can be invoked by a processor, and may include a non-volatile storage medium. In some embodiments, the electronic device may include memory and an interface. In some embodiments, the electronic device may also include other hardware depending on the specific application.

[0106] The computer-readable storage medium of this application embodiment stores a program thereon, which, when executed by a processor, is used to implement the star catalog data management method described above.

[0107] This application may take the form of a computer program product implemented on one or more computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing program code. Computer-readable storage media include permanent and non-permanent, removable and non-removable media, and information storage can be implemented using any method or technology. Information may be computer-readable instructions, data structures, program modules, or other data. Examples of computer-readable storage media include, but are not limited to: phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by a computing device.

[0108] The aforementioned electronic device can execute the star catalog data management method provided in the embodiments herein. The aforementioned electronic device may include the aforementioned star catalog data management system, such as one or more of a processor, controller, and PC (Personal Computer) terminal device. The server terminal device and the PC terminal device may include, but are not limited to, a server, desktop computer, tablet computer, or laptop computer.

[0109] It should be noted that the technical solutions or features described in the above embodiments can be combined or supplemented with each other without conflict. The scope of protection of this application is not limited to the precise structures described in the above embodiments and shown in the accompanying drawings; all modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A method for managing star catalog data, characterized in that, include: The system receives a response file receiving request, receives a star table file and a column description file, parses the star table file and the column description file to obtain star table metadata, and sends a star table import request; wherein, the star table file stores star table data, the column description file stores information describing each column in the star table file, and the star table metadata includes identification information describing the star table file and structural information describing each column in the star table file; In response to the star table import request, the data synchronization configuration is obtained based on the star table metadata; wherein, the star table import request includes the star table metadata; According to the data structure of the target data table indicated by the data synchronization configuration, read the star table data and synchronize the star table data to the target data table in the star table data platform; The structure information includes star table column names and star table column data types, and the identification information includes the star table data entry status. The star table column names are used to represent the names of each column in the star table file, and the star table column data types are used to represent the data types of each column in the star table file. The number of star table column names and star table column data types are the same and they correspond one-to-one. The parsing of the star table file and the column description file to obtain star table metadata includes: Parse the star table file and the column description file to obtain star table metadata other than the star table data entry status; In the case of missing data types in the star table column, the star table column name corresponding to the missing star table column data type is taken as the target star table column name. A similarity search is performed in the historical star table metadata to obtain the star table column name with the highest similarity to the target star table column name as the approximate star table column name. The data type of the star table column corresponding to the approximate star table column name is used as the derivation result of the missing star table column data type. The historical star table metadata is stored in the metadata database. After obtaining the derivation results of all missing star table column data types, the completed star table metadata is used as the star table metadata.

2. The star catalog data management method as described in claim 1, characterized in that, After obtaining the star catalog metadata, the following is included: Write the star table metadata into the metadata database.

3. The star catalog data management method as described in claim 2, characterized in that, The step of reading the star table data according to the data structure of the target data table indicated by the data synchronization configuration, and synchronizing the star table data to the target data table in the star table data platform includes: A star table entry workflow is generated based on the data synchronization configuration; wherein, the star table entry workflow defines multiple task nodes and the dependencies between the multiple task nodes, and each task node is used to represent a task; Based on the dependencies, the target task node to be executed is determined, and the task represented by the target task node is executed.

4. The star catalog data management method as described in claim 3, characterized in that, The identification information includes the name of the star table file, the data synchronization configuration includes the star table creation structure information, and obtaining the data synchronization configuration based on the star table metadata includes: determining the star table creation structure information based on the name of the star table file, the star table column name, and the star table column data type; The task node includes a data table generation node, and the task represented by the data table generation node is to generate the target data table on the star table data platform according to the star table table structure information.

5. The star catalog data management method as described in claim 4, characterized in that, The task node also includes: The synchronous task node is used to represent the task of writing the star table data into the target data table. The updated successful task node represents the task of updating the field of the star table entry status to "success" in the metadata database. Update the failed task node, the task to be represented is: in the meta database, update the field of the star table entry status to failed; The decision task node is used to represent the task of determining whether the update task node was successfully executed or the update task node failed, based on the execution result of the synchronization task node.

6. The star catalog data management method as described in claim 5, characterized in that, The star catalog data platform includes big data computing services and a data warehouse, and the star catalog data management method further includes: Based on the star table metadata, determine the storage location of the star table data that has been successfully written into the first target data table in the big data computing service and the data structure of the first target data table, and in response to the cross-matching service request, trigger a cross-matching service for the star table data in the big data computing service based on the star table data in the big data computing service and the star table metadata. Based on the star table metadata, the storage location of the star table data successfully written into the second target data table in the data warehouse and the data structure of the second target data table are determined. In response to a retrieval service request, a retrieval service for the star table data in the data warehouse is triggered based on the star table data in the data warehouse and the star table metadata.

7. The star catalog data management method as described in claim 6, characterized in that, The star table entry workflow defines task nodes that are executed sequentially, including: a first synchronization task node, a first decision task node, a first update failure task node, a second synchronization task node, a second decision task node, a first update success task node, and a second update failure task node. The task represented by the first synchronization task node is: to write the star table data into the first target data table; The first decision task node is used to represent the following task: determine whether to execute the first update failure task node based on the execution result of the first synchronization task node; when the first synchronization task node fails to execute, the first update failure task node is triggered to be executed. The task represented by the second synchronization task node is: to write the star table data from the first target table to the second target table; The second decision task node is used to represent the following task: based on the execution result of the second synchronization task node, it selects between the first successful update task node and the second failed update task node. When the second synchronization task node is executed successfully, it triggers the execution of the first successful update task node. When the second synchronization task node fails, it triggers the execution of the second failed update task node. The step of synchronizing the satellite data to the target data table in the satellite data platform includes: writing the satellite data into the first target data table and the second target data table in a serial manner.

8. The star catalog data management method as described in claim 6, characterized in that, The star table entry workflow defines task nodes that are executed in parallel order, including: a first synchronous task node, a second synchronous task node, the decision task node, the update failure task node, and the update success task node. The task represented by the first synchronization task node is: to write the star table data into the first target data table; The task represented by the second synchronization task node is: to write the star table data into the second target data table; The decision task node is used to represent the following tasks: branch selection based on the execution results of the first synchronization task node and the second synchronization task node; when the first synchronization task node or the second synchronization task node fails to execute, the update failure task node is triggered; when both the first synchronization task node and the second synchronization task node execute successfully, the update success task node is triggered. The step of synchronizing the satellite data to the target data table in the satellite data platform includes: writing the satellite data into the first target data table and the second target data table in parallel.

9. The star catalog data management method as described in claim 5, characterized in that, Also includes: In the event that a task node fails to execute, in response to a log viewing request for the failed task node, the failure log result corresponding to the failed task node is returned. as well as In response to a retry request for the failed task node, the failed task node is re-executed.

10. A star catalog data management system, characterized in that, include: The star table module is used to respond to file reception requests, receive star table files and column description files, parse the star table files and column description files to obtain star table metadata, and send star table import requests. The star table files store star table data, and the column description files store information describing each column in the star table files. The star table metadata includes identification information describing the star table files and structural information describing each column in the star table files. The star table data management system also includes a metadata database, which stores historical star table metadata. The structural information includes star table column names and star table column data types, and the identification information includes the star table data import status. The star table column names are used to represent the names of each column in the star table files, and the star table column data types are used to represent... The data types of each column in the star table file are identical in number and correspond one-to-one with the star table column names and data types. The star table module is specifically used for: parsing the star table file and the column description file to obtain star table metadata other than the star table data entry status; in the case of missing star table column data types, using the star table column name corresponding to the missing star table column data type as the target star table column name, performing a similarity search in the historical star table metadata to obtain the star table column name with the highest similarity to the target star table column name as the approximate star table column name, and using the star table column data type corresponding to the approximate star table column name as the derivation result of the missing star table column data type; after obtaining the derivation results of all missing star table column data types, using the completed star table metadata as the star table metadata. The file storage module is used to receive the star table file and the column description file transmitted by the star table module; The workflow scheduling module is used to respond to the star table entry request transmitted by the star table module and obtain the data synchronization configuration based on the star table metadata; wherein, the star table entry request includes the star table metadata; The data synchronization module is used to read the star table data from the file storage module according to the data structure of the target data table indicated by the data synchronization configuration, and synchronize the star table data to the target data table in the star table data platform.

11. The star catalog data management system as described in claim 10, characterized in that, The star catalog module is also used for: Write the star table metadata into the metadata database.

12. The star catalog data management system as described in claim 11, characterized in that, The workflow scheduling module is also used to generate a star table entry workflow based on the data synchronization configuration; wherein, the star table entry workflow defines multiple task nodes, the dependencies between the multiple task nodes, and the execution module corresponding to each task node, each task node is used to represent a task, and the execution module is used to execute the task represented by its corresponding task node. The workflow scheduling module is also used to determine the target task node to be executed and the target execution module corresponding to the target task node according to the dependency relationship, and to issue an execution instruction to the target execution module to trigger the target execution module to execute the task represented by the target task node.

13. The star catalog data management system as described in claim 12, characterized in that, The task nodes include: The synchronous task node is used to represent the task of writing the star table data into the target data table. The updated successful task node represents the task of updating the field of the star table entry status to "success" in the metadata database. Update the failed task node, the task to be represented is: in the meta database, update the field of the star table entry status to failed; The decision task node is used to represent the task of determining whether the update task node was successfully executed or the update task node failed, based on the execution result of the synchronization task node. The execution module corresponding to the synchronization task node is the data synchronization module, the execution modules corresponding to the successful update task node and the failed update task node are the star table module, and the execution module corresponding to the decision task node is the workflow scheduling module.

14. The star catalog data management system as described in claim 13, characterized in that, The star table data platform includes big data computing services and a data warehouse, and the star table data management system also includes: The cross-matching module is used to determine the storage location of the star table data that has been successfully written into the first target data table in the big data computing service and the data structure of the first target data table based on the star table metadata, and in response to the cross-matching service request, to trigger the provision of cross-matching service for the star table data in the big data computing service based on the star table data and the star table metadata. The retrieval module is used to determine the storage location of the star table data that has been successfully written into the second target data table in the data warehouse and the data structure of the second target data table based on the star table metadata, and in response to a retrieval service request, to trigger a retrieval service for the star table data in the data warehouse based on the star table data and the star table metadata.

Images