Astronomical star catalog data archiving cluster, method and system

Abstract

The present invention discloses an astronomical star catalog data archiving cluster, method and system. It comprises: a plan generation node, which is used to obtain the incremental star catalog data newly added in the astronomical star catalog data and the index auxiliary information corresponding to the incremental star catalog data, and send the incremental star catalog data and the index auxiliary information to a metadata server node, wherein the index auxiliary information includes the index identifier of each data in the incremental star catalog data; the metadata server node is used to partition each data in the incremental star catalog data according to the index auxiliary information and the preset partition mapping relationship, obtain the data storage node corresponding to each data, and send the indication information of the data storage node corresponding to each data to the plan generation node; the plan generation node is also used to archive the incremental star catalog data according to the indication information of the data storage node corresponding to each data. The present invention improves the real-time performance of data archiving.

Description

Technical Field

[0001] The embodiments of the present invention relate to the technical field of astronomical data processing, and in particular to an astronomical star catalog data archiving cluster, method and system. Background Art

[0002] Astronomical catalog data records information such as the position and parameters of celestial bodies. The astronomical community has developed a special storage format standard for astronomical data, the Flexible Image Transport System (FITS). FITS files consist of a file header and a data body. The file header stores the descriptive metadata information of the file in the form of key / value pairs, and the data body stores the complete data of the catalog. Astronomical catalog data usually has a large amount of data, and users have high requirements for range queries. Reasonable archiving methods can improve the real-time performance of data queries. Therefore, how to archive astronomical catalog data in real time is a problem that needs to be solved.

[0003] In terms of astronomical data archiving, some researchers have proposed a method based on a streaming data processing architecture, which uses a data source to first extract metadata, and then transmits the metadata to the archiving platform through the network for data processing. This archiving method realizes the real-time archiving of metadata of the original observation data set, and can provide rapid data positioning services to the outside world. However, with this method, users can only obtain which file the star catalog data is stored in, and cannot find the data itself. This method cannot meet scenarios with higher requirements, such as range retrieval of astronomical star catalog data, and querying information such as celestial body positions and parameters. Another existing archiving method is to store star catalog data in the database and provide data query services to the outside world only after all star catalog data have been collected, but this method cannot guarantee the real-time nature of data archiving. Summary of the invention

[0004] The present invention provides an astronomical star catalog data archiving cluster, method and system to solve the problem of insufficient real-time performance of astronomical star catalog data archiving in the prior art.

[0005] According to one aspect of the present invention, there is provided an astronomical star catalog data archiving cluster, comprising: a metadata server node, a plan generation node and at least one data storage node, wherein the plan generation node is connected to the metadata server node and the at least one data storage node respectively;

[0006] The plan generation node is used to obtain the incremental star catalog data newly added in the astronomical star catalog data and the index auxiliary information corresponding to the incremental star catalog data, and send the incremental star catalog data and the index auxiliary information to the metadata server node, wherein the index auxiliary information includes the index identifier of each data in the incremental star catalog data;

[0007] The metadata server node is used to partition each piece of data in the incremental star catalog data according to the index auxiliary information and the preset partition mapping relationship, obtain the data storage node corresponding to each piece of data, and send the indication information of the data storage node corresponding to each piece of data to the plan generation node;

[0008] The plan generation node is also used to archive the incremental star catalog data according to the indication information of the data storage node corresponding to each piece of data.

[0009] According to another aspect of the present invention, a method for archiving astronomical star catalog data is provided, which is applied to the astronomical star catalog data archiving cluster according to any embodiment of the present invention, and the method comprises:

[0010] Acquire the newly added incremental star catalog data in the astronomical star catalog data and the index auxiliary information corresponding to the incremental star catalog data through the plan generation node; the index auxiliary information includes the index identifier of each data in the incremental star catalog data;

[0011] Partition each piece of data in the incremental star catalog data according to the index auxiliary information and the preset partition mapping relationship through the metadata server node to obtain the data storage node corresponding to each piece of data, and send the indication information of the data storage node corresponding to each piece of data to the plan generation node;

[0012] The incremental star catalog data is archived by a plan generation node according to the indication information of the data storage node corresponding to each piece of data.

[0013] According to another aspect of the present invention, there is provided an astronomical star catalog data archiving system, the system comprising the astronomical star catalog data archiving cluster according to any embodiment of the present invention, and a data acquisition module, the data acquisition module being connected to the astronomical star catalog data archiving cluster;

[0014] The data acquisition module is used to take the newly added astronomical star catalog data in the astronomical star catalog data as incremental star catalog data, generate corresponding index auxiliary information for the incremental star catalog data, and send the incremental star catalog data and the index auxiliary information to the astronomical star catalog data archiving cluster; the index auxiliary information includes an index identifier of each data in the incremental star catalog data;

[0015] The astronomical star catalog data archiving cluster is used to archive the incremental star catalog data according to the index auxiliary information and the preset partition mapping relationship.

[0016] An astronomical star catalog data archiving cluster, method and system according to an embodiment of the present invention comprises: a metadata server node, a plan generation node and at least one data storage node, wherein the plan generation node is connected to the metadata server node and the at least one data storage node respectively; the plan generation node is used to obtain the newly added incremental star catalog data in the astronomical star catalog data and the index auxiliary information corresponding to the incremental star catalog data, and send the incremental star catalog data and the index auxiliary information to the metadata server node, wherein the index auxiliary information includes the index identifier of each data in the incremental star catalog data; the metadata server node is used to partition each data in the incremental star catalog data according to the index auxiliary information and the preset partition mapping relationship, obtain the data storage node corresponding to each data, and send the indication information of the data storage node corresponding to each data to the plan generation node; the plan generation node is also used to archive the incremental star catalog data according to the indication information of the data storage node corresponding to each data. The embodiment of the present invention can improve the real-time performance of data archiving by partitioning and archiving the newly added incremental star catalog data according to the corresponding index auxiliary information and the preset partition mapping relationship, thereby solving the problem of insufficient real-time performance of astronomical star catalog data archiving in the prior art.

[0017] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0019] Figure 1 A schematic diagram of the structure of an astronomical star catalog data archiving cluster provided in the first embodiment of the present invention;

[0020] Figure 2 A schematic diagram of a flow chart of an astronomical star catalog data archiving method provided in Embodiment 2 of the present invention;

[0021] Figure 3 A schematic diagram of the structure of an astronomical star catalog data archiving system provided in Embodiment 3 of the present invention;

[0022] Figure 4 A schematic diagram of the structure of an astronomical star catalog data archiving system provided by an embodiment of the present invention;

[0023] Figure 5 A schematic diagram of a process for archiving astronomical star catalog data provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0024] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiment of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiment of the present invention. Obviously, the described embodiment is only an embodiment of a part of the present invention, rather than all the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work should belong to the scope of protection of the present invention. It should be understood that the various steps recorded in the method implementation of the present invention can be performed in different orders and / or in parallel. In addition, the method implementation may include additional steps and / or omit the steps shown. The scope of the present invention is not limited in this respect.

[0025] The term "including" and its variations used herein are open inclusions, i.e., "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". The relevant definitions of other terms will be given in the following description.

[0026] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, any variations of the terms "including" and "having" are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0027] It should be noted that the modifications of "one" and "plurality" mentioned in the present invention are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0028] The names of the messages or information exchanged between multiple devices in the embodiments of the present invention are only used for illustrative purposes, and are not used to limit the scope of these messages or information.

[0029] The archiving of astronomical catalog data, from raw data generation to providing retrieval services, requires the following steps:

[0030] (1) Detect the original data at the data source.

[0031] (2) Store the original data files.

[0032] (3) Offline cleaning of data files to generate star catalog data.

[0033] (4) Generate auxiliary information according to the sky area division protocol.

[0034] (5) Store star catalog data and refresh sky area index.

[0035] (6) The star catalog data retrieval service can query incremental data.

[0036] In the above archiving method, after the astronomical star catalog data is generated, it is first stored as a file in FITS format, and then the data file is loaded for offline cleaning to generate incremental star catalog data, and then the pseudo-spherical index auxiliary information is generated using an offline tool, and finally the star catalog data is stored in the warehouse and the sky area index is refreshed. However, the above method cannot guarantee the real-time archiving of astronomical star catalog data. Therefore, an embodiment of the present invention provides a method that uses a distributed data processing architecture to process the archiving and storage of astronomical star catalog data to solve the above problem.

[0037] Embodiment 1

[0038] Figure 1 A structural schematic diagram of an astronomical star catalog data archiving cluster provided in Example 1 of the present invention. The astronomical star catalog data archiving cluster can be applicable to the situation of archiving astronomical star catalog data. The astronomical star catalog data archiving cluster can execute an astronomical star catalog data archiving method. In this embodiment, the astronomical star catalog data archiving cluster includes but is not limited to: a distributed storage cluster, etc.

[0039] like Figure 1 As shown, an astronomical star catalog data archiving cluster provided by the first embodiment of the present invention includes: a metadata server node 10, a plan generation node 20 and at least one data storage node 30, wherein the plan generation node 20 is connected to the metadata server node 10 and the at least one data storage node 30 respectively;

[0040] The plan generation node 20 is used to obtain the incremental star catalog data newly added in the astronomical star catalog data and the index auxiliary information corresponding to the incremental star catalog data, and send the incremental star catalog data and the index auxiliary information to the metadata server node 10, wherein the index auxiliary information includes the index identifier of each data in the incremental star catalog data;

[0041] The metadata server node 10 is used to partition each piece of data in the incremental star catalog data according to the index auxiliary information and the preset partition mapping relationship, obtain the data storage node corresponding to each piece of data, and send the indication information of the data storage node 30 corresponding to each piece of data to the plan generation node 20;

[0042] The plan generation node 20 is further used to archive the incremental star catalog data according to the indication information of the data storage node 30 corresponding to each piece of data.

[0043] Among them, the plan generation node 20 can be a node for receiving external request information and executing corresponding tasks. The astronomical star catalog data can be the quantitative processing result of the corresponding celestial body observation data. A piece of astronomical star catalog data can include multiple data, each of which can be understood as the data of a certain point on the celestial body. The multiple data included in a piece of astronomical star catalog data are data of the same celestial body. The incremental star catalog data can be the original data file of the astronomical star catalog data newly added to the astronomical star catalog data. The index auxiliary information can include the index identifier of each data in the incremental star catalog data, and the index identifier can be the unique identifier of each data.

[0044] In this embodiment, the plan generation node 20 can be connected to the metadata server node 10, and the acquired incremental star catalog data and the corresponding index auxiliary information are sent to the metadata server node 10. The acquisition method of the incremental star catalog data and the corresponding index auxiliary information in this embodiment is not limited in this embodiment. For example, the plan generation node 20 can directly receive the incremental star catalog data collected by the data collection end and the index auxiliary information obtained after processing.

[0045] The metadata server node 10 may be a server that stores and manages a large amount of metadata. Metadata is descriptive information about data, including the attributes, structure, format, location, etc. of the data. The preset partition mapping relationship may include a mapping relationship between at least one data storage node and an index identifier range corresponding to the stored data. The preset partition mapping relationship may be used to indicate which index identifiers of astronomical catalog data may be stored in each data storage node 30. The indication information may be information for indicating the data storage node where each piece of data should be stored. The data storage node 30 may be a node for storing astronomical catalog data. The data storage node 30 may be one or more. The data stored in the same data storage node 30 is data of the same celestial body.

[0046] In this embodiment, the metadata server node 10 can determine which data storage node 30 each data in the incremental star catalog data should be stored in based on the index auxiliary information and the preset partition mapping relationship, obtain the data storage node 30 corresponding to each data, and send the corresponding indication information to the plan generation node 20.

[0047] In this embodiment, the plan generation node 20 can be connected to at least one data storage node 30. After receiving the indication information of the data storage node corresponding to each piece of data, the plan generation node 20 can store each piece of data in the incremental star catalog data to the corresponding data storage node 30 according to the indication information to complete the archiving of the incremental star catalog data.

[0048] An astronomical star catalog data archiving cluster provided in the first embodiment of the present invention comprises: a metadata server node, a plan generation node and at least one data storage node, wherein the plan generation node is connected to the metadata server node and the at least one data storage node respectively; the plan generation node is used to obtain the newly added incremental star catalog data in the astronomical star catalog data and the index auxiliary information corresponding to the incremental star catalog data, and send the incremental star catalog data and the index auxiliary information to the metadata server node, wherein the index auxiliary information includes the index identifier of each data in the incremental star catalog data; the metadata server node is used to partition each data in the incremental star catalog data according to the index auxiliary information and the preset partition mapping relationship, obtain the data storage node corresponding to each data, and send the indication information of the data storage node corresponding to each data to the plan generation node; the plan generation node is also used to archive the incremental star catalog data according to the indication information of the data storage node corresponding to each data. The embodiment of the present invention can improve the real-time performance of data archiving by partitioning and archiving the newly added incremental star catalog data according to the corresponding index auxiliary information and the preset partition mapping relationship, thereby solving the problem of insufficient real-time performance of astronomical star catalog data archiving in the prior art.

[0049] Based on the above embodiment, a variant embodiment of the above embodiment is proposed. It should be noted that in order to make the description concise, only the differences from the above embodiment are described in the variant embodiment.

[0050] In one embodiment, the metadata server node 10 is specifically used to: determine the index identifier of each data in the incremental star catalog data through the index auxiliary information; for each data, use the data storage node 30 corresponding to the index identifier range including the index identifier of the data as the data storage node 30 corresponding to the data.

[0051] In this embodiment, after receiving the incremental star catalog data and the corresponding index auxiliary information, the metadata server node 10 can determine the index identifier of each data in the incremental star catalog data based on the index auxiliary information, and then determine the index identifier range of the index identifier corresponding to each data based on the relationship between the data storage node 30 included in the preset partition mapping relationship and the index identifier range corresponding to the stored data, and use the data storage node 30 corresponding to the index identifier range as the data storage node 30 corresponding to the data.

[0052] In one embodiment, the metadata server node 10 is also used to: generate metadata information according to the data storage node 30 corresponding to each piece of data; update the locally stored metadata information according to the metadata information, and the metadata information is used to indicate the data information of the data stored by the data storage node.

[0053] The metadata information may be data information for indicating the data stored in the data storage node. The metadata information may be used to determine which astronomical star catalog data is stored in the data storage node 30 .

[0054] In this embodiment, after partitioning each piece of data in the incremental star catalog data, the metadata server node 10 can generate metadata information according to the data storage node 30 corresponding to each piece of data, and update the locally stored metadata information according to the generated metadata information, thereby ensuring that the metadata information stored in the metadata server node 10 can promptly reflect the data storage status of each data storage node 30.

[0055] It is understandable that in this embodiment, the metadata server node 10 may also update the metadata information after receiving the information that the plan generation node 20 successfully archives the incremental star catalog data, and this embodiment does not limit this.

[0056] In one embodiment, the plan generation node 20 is also used to: receive retrieval request information, and send the retrieval request information to the metadata server node 10; the metadata server node 10 is also used to send the metadata information of each corresponding data storage node 30 to the plan generation node 20 according to the retrieval request information; the plan generation node 20 is also used to query the corresponding retrieval result from the original star catalog according to the metadata information and the retrieval request information; wherein the original star catalog includes astronomical star catalog data of the same celestial body.

[0057] The search request information may be information requesting the search of astronomical star catalog data. The original star catalog may be a table storing astronomical star catalog data corresponding to a celestial body, and an original star catalog includes astronomical star catalog data of the same celestial body.

[0058] In this embodiment, the plan generation node 20 can receive the user's retrieval request information and send the retrieval request information to the metadata server node 10. The metadata server node 10 can analyze the retrieval request information and send the metadata information of each data storage node 30 queried corresponding to the retrieval request information to the plan generation node 20. The plan generation node 20 can perform a query process according to the metadata information and the retrieval request information, and query the corresponding retrieval result from the original star catalog.

[0059] In one embodiment, the retrieval request information includes a sky area division protocol identifier and a retrieval range; accordingly, the plan generation node 20 is specifically used to: generate a query auxiliary table based on the sky area division protocol identifier and the retrieval range in combination with a pseudo-spherical index algorithm; generate a query plan based on the query auxiliary table and the metadata information; and query the original star catalog through the query plan to obtain a retrieval result.

[0060] Among them, the sky area division protocol identifier can be an identifier of the sky area division protocol, and the sky area division protocol can be a space division technology used for sky area division and analysis. The retrieval range can be used to retrieve relevant parameters of celestial bodies. The pseudo-spherical indexing algorithm is a method for indexing high-dimensional data, especially when processing point clouds, images or other space-related data in multi-dimensional space, for efficient storage and retrieval of data. The pseudo-spherical indexing algorithm generally involves mapping the data onto a pseudo-sphere, and then indexing and querying in a hierarchical manner. The query auxiliary table can be a table constructed based on the sky area division protocol identifier and the retrieval range. The query plan can be a set of steps executed by a database management system to complete a query.

[0061] In this embodiment, the astronomical star catalog data of different celestial bodies can be stored according to the same or different sky area division protocols, which is not limited in this embodiment. For example, the sky area division protocol can be Hierarchical Triangular Mesh (HTM), Hierarchical Equal Area isoLatitude Pixelization (HEALPix) or other protocols.

[0062] In this embodiment, since different astronomical star catalog data may use different sky area division protocols, the search request information may also include a sky area division protocol identifier. The plan generation node 20 may generate a query auxiliary table based on the sky area division protocol identifier and the search range in combination with the pseudo-spherical index algorithm. Since the query auxiliary table contains relevant information that the user needs to retrieve, a query plan may be generated based on the query auxiliary table and metadata information, thereby querying the original star catalog through the query plan to obtain a search result. The specific generation method of the query plan is not limited in this embodiment.

[0063] In one embodiment, the search range includes the searched celestial body and the right ascension, declination and radius information corresponding to the searched celestial body; accordingly, the plan generation node 20 is specifically used to: process the right ascension, declination and radius information based on a pseudo-spherical index algorithm to obtain an index identifier of the searched celestial body; and use the index identifier of the searched celestial body and the sky area division protocol identifier as table columns respectively to generate a query auxiliary table.

[0064] Among them, Right Ascension (RA) can be an angular coordinate in the celestial coordinate system, used to measure the angle of a celestial body relative to the vernal equinox. Declination (Dec) can be another angular coordinate in the celestial coordinate system, used to measure the angle of a celestial body relative to the celestial equator. The radius information can refer to the distance of the celestial body from the observer. The retrieved celestial body can be the celestial body that the user needs to retrieve.

[0065] In this embodiment, the plan generation node 20 can process the right ascension, declination and radius information of the retrieved celestial body based on the pseudo-spherical index algorithm to obtain the index identifier of the retrieved celestial body, and then the index identifier of the retrieved celestial body and the sky area division protocol identifier can be used as table columns respectively to generate a query auxiliary table.

[0066] Embodiment 2

[0067] Figure 2 This is a flow chart of an astronomical star catalog data archiving method provided in the second embodiment of the present invention. The second embodiment is optimized on the basis of the above embodiments. For details not yet provided in this embodiment, please refer to the first embodiment.

[0068] like Figure 2 As shown, a method for archiving astronomical star catalog data provided in the second embodiment of the present invention is applied to the astronomical star catalog data archiving cluster described in any embodiment of the present invention, and includes the following steps:

[0069] S210, acquiring incremental star catalog data newly added in the astronomical star catalog data and index auxiliary information corresponding to the incremental star catalog data through a planned generation node; the index auxiliary information includes an index identifier of each data in the incremental star catalog data.

[0070] Among them, the plan generation node can be a node for receiving external request information and executing corresponding tasks. The astronomical star catalog data can be the quantitative processing result of the corresponding celestial body observation data. An astronomical star catalog data can include multiple data, each of which can be understood as the data of a certain point on a celestial body. The multiple data included in an astronomical star catalog data are data of the same celestial body. The incremental star catalog data can be the original data file of the astronomical star catalog data newly added to the astronomical star catalog data. The index auxiliary information can include an index identifier for each data in the incremental star catalog data, and the index identifier can be a unique identifier for each data.

[0071] In this embodiment, the incremental star catalog data newly added to the astronomical star catalog data can be acquired through the plan generation node, and the corresponding index auxiliary information can be acquired.

[0072] S220. Partition each piece of data in the incremental catalog data according to the index auxiliary information and the preset partition mapping relationship through the metadata server node to obtain the data storage node corresponding to each piece of data, and send the indication information of the data storage node corresponding to each piece of data to the plan generation node.

[0073] The metadata server node may be a server that stores and manages a large amount of metadata. The preset partition mapping relationship may include a mapping relationship between at least one data storage node and an index identifier range corresponding to the stored data, and the preset partition mapping relationship may be used to indicate which astronomical star catalog data of which index identifiers can be stored in each data storage node. The indication information may be information for indicating the data storage node where each piece of data should be stored. The data storage node may be a node for storing astronomical star catalog data.

[0074] In this embodiment, the metadata server node can determine which data storage node each piece of data in the incremental star catalog data should be stored in according to the index auxiliary information and the preset partition mapping relationship, and obtain the data storage node corresponding to each piece of data.

[0075] S230, archiving the incremental star catalog data according to the indication information of the data storage node corresponding to each piece of data through the plan generation node.

[0076] In this embodiment, after obtaining the indication information of the data storage node corresponding to each data, each data in the incremental star catalog data can be stored in the corresponding data storage node according to the indication information through the plan generation node to complete the archiving of the incremental star catalog data.

[0077] A method for archiving astronomical star catalog data provided in a second embodiment of the present invention includes: obtaining newly added incremental star catalog data in astronomical star catalog data and index auxiliary information corresponding to the incremental star catalog data through a plan generation node; the index auxiliary information includes an index identifier for each piece of data in the incremental star catalog data; partitioning each piece of data in the incremental star catalog data according to the index auxiliary information and a preset partition mapping relationship through a metadata server node to obtain a data storage node corresponding to each piece of data, and sending indication information of the data storage node corresponding to each piece of data to the plan generation node; archiving the incremental star catalog data through the plan generation node according to the indication information of the data storage node corresponding to each piece of data. By partitioning and archiving the newly added incremental star catalog data according to the corresponding index auxiliary information and the preset partition mapping relationship, the real-time performance of data archiving can be improved, solving the problem of insufficient real-time performance of astronomical star catalog data archiving in the prior art.

[0078] In one embodiment, the metadata server node partitions each data in the incremental star catalog data according to the index auxiliary information and a preset partition mapping relationship to obtain a data storage node corresponding to each data, including: determining the index identifier of each data in the incremental star catalog data according to the index auxiliary information through the metadata server node; for each data, using the data storage node corresponding to the index identifier range including the index identifier of the data as the data storage node corresponding to the data.

[0079] In this embodiment, the metadata server node can determine the index identifier of each data in the incremental star catalog data based on the index auxiliary information, and then determine the index identifier range of the index identifier corresponding to each data based on the relationship between the data storage nodes included in the preset partition mapping relationship and the index identifier range corresponding to the stored data, and use the data storage node corresponding to the index identifier range as the data storage node corresponding to the data.

[0080] In one embodiment, the method further includes: generating metadata information through a metadata server node according to the data storage node corresponding to each piece of data, and updating the locally stored metadata information according to the metadata information, wherein the metadata information is used to indicate data information of the data stored in the data storage node.

[0081] The metadata information may be data information for indicating the data stored in the data storage node. The metadata information may be used to determine which astronomical star catalog data is stored in the data storage node 30 .

[0082] In this embodiment, the metadata server node can generate metadata information according to the data storage node corresponding to each piece of data, and update the locally stored metadata information according to the generated metadata information, thereby ensuring that the metadata information stored in the metadata server node can promptly reflect the data storage status of each data storage node.

[0083] Embodiment 3

[0084] Figure 3 This is a schematic diagram of the structure of an astronomical star catalog data archiving system provided in the third embodiment of the present invention, which can execute the astronomical star catalog data archiving method described in any embodiment of the present invention. Figure 3 As shown, the astronomical star catalog data archiving system includes: an astronomical star catalog data archiving cluster 100, and a data acquisition module 200, the data acquisition module 200 is connected to the astronomical star catalog data archiving cluster 100;

[0085] The data acquisition module 200 is used to take the newly added astronomical star catalog data in the astronomical star catalog data as incremental star catalog data, generate corresponding index auxiliary information for the incremental star catalog data, and send the incremental star catalog data and the index auxiliary information to the astronomical star catalog data archiving cluster 100; the index auxiliary information includes an index identifier of each data in the incremental star catalog data;

[0086] The astronomical star catalog data archiving cluster 100 is used to archive the incremental star catalog data according to the index auxiliary information and the preset partition mapping relationship.

[0087] Among them, the data acquisition module 200 may be a module for collecting astronomical star catalog data. The astronomical star catalog data may be the quantitative processing result of the corresponding celestial observation data. The incremental star catalog data may be the original data file of the astronomical star catalog data newly added in the astronomical star catalog data. The index auxiliary information may include an index identifier for each data in the incremental star catalog data, and the index identifier may be a unique identifier for each data. The astronomical star catalog data archiving cluster 100 may be a cluster of modules for processing astronomical star catalog data. The preset partition mapping relationship may include a mapping relationship between at least one data storage node and an index identifier range corresponding to the stored data.

[0088] In this embodiment, the data acquisition module 200 may acquire newly added astronomical star catalog data in the astronomical star catalog data to obtain incremental star catalog data, generate corresponding index auxiliary information for the incremental star catalog data, and send the incremental star catalog data and the index auxiliary information to the astronomical star catalog data archiving cluster 100. The astronomical star catalog data archiving cluster 100 may archive the incremental star catalog data according to the index auxiliary information and the preset partition mapping relationship.

[0089] The present embodiment provides an astronomical star catalog data archiving system, including an astronomical star catalog data archiving cluster and a data acquisition module, wherein the data acquisition module is connected to the astronomical star catalog data archiving cluster; the data acquisition module is used to use the newly added astronomical star catalog data in the astronomical star catalog data as incremental star catalog data, generate corresponding index auxiliary information for the incremental star catalog data, and send the incremental star catalog data and the index auxiliary information to the astronomical star catalog data archiving cluster; the index auxiliary information includes an index identifier for each data in the incremental star catalog data; the astronomical star catalog data archiving cluster is used to archive the incremental star catalog data according to the index auxiliary information and a preset partition mapping relationship. By partitioning and archiving the newly added incremental star catalog data according to the corresponding index auxiliary information and the preset partition mapping relationship, the real-time performance of data archiving can be improved, and the problem of insufficient real-time performance of astronomical star catalog data archiving in the prior art is solved.

[0090] The embodiments of the present invention provide several specific implementation methods based on the technical solutions of the above embodiments.

[0091] As a specific implementation of this embodiment, Figure 4 A schematic diagram of the structure of an astronomical star catalog data archiving system provided by an embodiment of the present invention is shown in FIG. Figure 4 As shown, the astronomical star catalog data archiving cluster may be a distributed storage cluster in a database module. The astronomical star catalog data archiving system may further include a user terminal connected to the distributed storage cluster. Figure 5 A schematic diagram of a process for archiving astronomical star catalog data provided by an embodiment of the present invention is as follows: Figure 5 As shown, the astronomical star catalog data archiving system can archive the astronomical star catalog data through the following steps:

[0092] 1. Get the sky region index information from the database

[0093] The data acquisition module is deployed in the astronomical star catalog observation data source. The specific functions of the data acquisition module can be implemented based on the Agent. The Agent will try to establish a connection with the database module. After the connection is established, the Agent will obtain the sky area index information corresponding to the current data source from the database module. The sky area index information can be a specific sky area division protocol, such as HTM, HEALPix, etc.

[0094] 2. Detect incremental raw data

[0095] The Agent will poll and scan the monitoring data source to capture data change information and scan the data source file directory at a fixed time period. When a new data file is written to the data source file directory, the Message Digest Algorithm 5 (MD5) code of the data source file directory will change. The Agent will parse the data source dynamic information based on the MD5 code, locate the new FITS format file, and obtain the incremental original data file.

[0096] 3. Complete data cleaning and generate index auxiliary information in memory

[0097] The Agent reads the incremental raw data file into memory and cleans the star catalog observation data in the incremental raw data file. Based on the sky area division protocol obtained from the database module, the right ascension and declination in the astronomical star catalog are converted into one-dimensional pixel IDs and index auxiliary information is generated. Finally, the collected data information and index auxiliary information are sent to the distributed storage cluster of the database module.

[0098] 4. Insert incremental star catalog and related index data in real time

[0099] In response to the data insertion operation, the distributed storage cluster determines the information triggered by the data insertion operation as the insertion request information, and transmits the insertion request information to the plan generation node, so that the plan generation node obtains the insertion request information. The plan generation node divides the incremental star table into multiple partitions according to the corresponding mapping relationship by inserting the index identifier corresponding to each data in the incremental star table, and automatically allocates these partitions to different data storage nodes in the distributed storage cluster. The metadata server node can update the metadata information corresponding to each data storage node, and determine which data storage node to store the partitioned inserted data through the mapping relationship between each data storage node and the index identifier.

[0100] 5. Star catalog data retrieval service can query incremental data

[0101] The database module obtains the retrieval request information, which is determined by the user client in response to the information input operation, and the retrieval request information includes the retrieval required sky area division protocol identifier and the retrieval range; the metadata information corresponding to each data storage node is obtained from the metadata server node; a pseudo-spherical index is created based on the sky area division protocol identifier corresponding to the protocol and the retrieval range to obtain a query auxiliary table; a query plan is generated based on the query auxiliary table and the metadata information, and the original star catalog is queried based on the query plan, and the retrieval results are obtained from the corresponding target data storage nodes, and the star catalog data corresponding to the retrieval results are summarized and fed back to the user client.

[0102] In order to solve the problem of insufficient support for real-time query of astronomical star catalog data in existing archiving systems, this embodiment first obtains sky area index information from the database, obtains incremental original data through detection, completes data cleaning and generates index auxiliary information in the memory, and then inserts the incremental star catalog and related index data in real time. Finally, the star catalog data retrieval service can query the incremental data, thereby ensuring the real-time query of astronomical data and avoiding the problem of the traditional archiving method that requires repeated loading of star catalog observation data files in the later stage.

[0103] This embodiment is based on a distributed data processing architecture and uses real-time archiving to replace the traditional offline archiving mode. It is suitable for star catalog data archiving scenarios in the astronomical field that have high real-time requirements. By innovatively integrating star catalog data archiving and star catalog data index creation, the system's real-time performance and star catalog data availability are improved. The archiving platform uses a distributed data processing architecture to process star catalog data in real time, reducing physical cache consumption and avoiding the problem of requiring a large amount of physical cache; while star catalog data is being stored, a pseudo-spherical index is created according to the sky area division protocol. The archived data can be used directly by users, avoiding repeated processing of star catalog data and improving convenience.

[0104] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0105] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. An astronomical star catalog data archiving cluster, characterized in that: include: A metadata server node, a plan generation node and at least one data storage node, wherein the plan generation node is connected to the metadata server node and the at least one data storage node respectively; The plan generation node is used to obtain the incremental star catalog data newly added in the astronomical star catalog data and the index auxiliary information corresponding to the incremental star catalog data, and send the incremental star catalog data and the index auxiliary information to the metadata server node, wherein the index auxiliary information includes the index identifier of each data in the incremental star catalog data; The metadata server node is used to partition each piece of data in the incremental star catalog data according to the index auxiliary information and the preset partition mapping relationship, obtain the data storage node corresponding to each piece of data, and send the indication information of the data storage node corresponding to each piece of data to the plan generation node; The plan generation node is also used to archive the incremental star catalog data according to the indication information of the data storage node corresponding to each piece of data.

2. The astronomical star catalog data archiving cluster according to claim 1, characterized in that: The preset partition mapping relationship includes a mapping relationship between at least one data storage node and an index identification range corresponding to the stored data. Accordingly, the metadata server node is specifically used to: Determine the index identifier of each piece of data in the incremental star catalog data by using the index auxiliary information; For each piece of data, a data storage node corresponding to an index identifier range including the index identifier of the data is used as the data storage node corresponding to the data.

3. The astronomical star catalog data archiving cluster according to claim 1, characterized in that: The metadata server node is also used for: Generate metadata information according to the data storage node corresponding to each piece of data; The locally stored metadata information is updated according to the metadata information, where the metadata information is used to indicate data information of the data stored in the data storage node.

4. The astronomical star catalog data archiving cluster according to claim 1, characterized in that: The plan generation node is also used to: receive retrieval request information, and send the retrieval request information to the metadata server node; The metadata server node is further used to send metadata information of each corresponding data storage node to the plan generation node according to the retrieval request information; The plan generation node is further used to query the corresponding search result from the original star catalog according to the metadata information and the search request information; The original star catalog includes astronomical star catalog data of the same celestial body.

5. The astronomical star catalog data archiving cluster according to claim 4, characterized in that: The search request information includes a sky area division protocol identifier and a search scope; accordingly, the plan generation node is specifically used to: Based on the sky area division protocol identifier and the search range, a query auxiliary table is generated in combination with a pseudo-spherical index algorithm; Generate a query plan based on the query auxiliary table and the metadata information; The original star catalog is queried through the query plan to obtain the retrieval result.

6. The astronomical star catalog data archiving cluster according to claim 5, characterized in that: The search scope includes the searched celestial body and the right ascension, declination and radius information corresponding to the searched celestial body; accordingly, the plan generation node is specifically used for: Processing the right ascension, the declination and the radius information based on a pseudo-spherical index algorithm to obtain an index identifier of the retrieved celestial body; The index identifier of the retrieved celestial body and the sky area division protocol identifier are respectively used as table columns to generate a query auxiliary table.

7. A method for archiving astronomical star catalog data, characterized in that: Applied to the astronomical star catalog data archiving cluster as described in claims 1-6, the method comprises: Acquire the newly added incremental star catalog data in the astronomical star catalog data and the index auxiliary information corresponding to the incremental star catalog data through the plan generation node; the index auxiliary information includes the index identifier of each data in the incremental star catalog data; Partition each piece of data in the incremental star catalog data according to the index auxiliary information and the preset partition mapping relationship through the metadata server node to obtain the data storage node corresponding to each piece of data, and send the indication information of the data storage node corresponding to each piece of data to the plan generation node; The incremental star catalog data is archived by a plan generation node according to the indication information of the data storage node corresponding to each piece of data.

8. The method for archiving astronomical star catalog data according to claim 7, characterized in that: The preset partition mapping relationship includes a mapping relationship between at least one data storage node and an index identification range corresponding to the stored data. Accordingly, partitioning each piece of data in the incremental star catalog data according to the index auxiliary information and the preset partition mapping relationship by the metadata server node to obtain a data storage node corresponding to each piece of data includes: The metadata server node determines the index identifier of each piece of data in the incremental star catalog data according to the index auxiliary information; for each piece of data, the data storage node corresponding to the index identifier range including the index identifier of the data is used as the data storage node corresponding to the data.

9. The method for archiving astronomical star catalog data according to claim 7, characterized in that: The method further comprises: The metadata server node generates metadata information according to the data storage node corresponding to each piece of data, and updates the locally stored metadata information according to the metadata information, wherein the metadata information is used to indicate data information of the data stored in the data storage node.

10. An astronomical star catalog data archiving system, characterized in that: The system comprises the astronomical star catalog data archiving cluster as described in claims 1-6, and a data acquisition module, wherein the data acquisition module is connected to the astronomical star catalog data archiving cluster; The data acquisition module is used to take the newly added astronomical star catalog data in the astronomical star catalog data as incremental star catalog data, generate corresponding index auxiliary information for the incremental star catalog data, and send the incremental star catalog data and the index auxiliary information to the astronomical star catalog data archiving cluster; the index auxiliary information includes an index identifier of each data in the incremental star catalog data; The astronomical star catalog data archiving cluster is used to archive the incremental star catalog data according to the index auxiliary information and the preset partition mapping relationship.

Images