Massive star catalogue distributed cross identification method and device

Abstract

The invention relates to the technical field of star catalogue cross identification, and provides a distributed cross identification method and device for massive star catalogues. The method comprises the following steps: dividing a celestial sphere surface into a plurality of blocks, and dividing all celestial bodies in a star catalogue into corresponding blocks according to mapping positions of the celestial bodies on the celestial sphere surface to form data partitions; performing cross identification on a first data partition in the first star catalogue, wherein the cross identification comprises the following steps: searching a second data partition matched with the first data partition in a second star catalogue; performing cross identification on each celestial body in the first data partition and each celestial body in the second data partition; the second data partition comprises a data partition located in the first block and a data partition located in a peripheral block of the first block in the second star catalogue; the first block is a block where the first data partition is located. Under the application scene of massive star catalogues, the data access times required by cross identification can be reduced, the cross identification efficiency can be improved, and the resource occupation can be reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of star catalog cross-certification, and in particular to a method and device for distributed cross-certification of massive star catalogs. Background Art

[0002] With the development of observation equipment and technology, astronomy has entered the full-band era. Different astronomical observation centers or astronomical equipment collect data for different bands of celestial bodies. Fusion of star catalogs of different bands to obtain multi-band or full-band data containing more information is the basis of astronomical research, especially for multi-band data mining and statistical analysis. With the growth of observation data, star catalog data has reached TB and PB levels. Traditional relational database storage and cross-verification technology based on relational databases cannot meet the technical requirements of massive star catalog data.

[0003] In view of this, overcoming the defects of the prior art is an urgent problem to be solved in the field of this technology. Summary of the invention

[0004] The technical problem to be solved by the present invention is that the existing technology is difficult to meet the needs of cross-certification of massive star catalogs.

[0005] The present invention adopts the following technical solution:

[0006] In a first aspect, the present invention provides a distributed cross-verification method for a massive star catalog, which divides the celestial sphere into a plurality of blocks, and divides each celestial body in the star catalog into a corresponding block according to the mapping position of the celestial body on the celestial sphere to form a corresponding data partition; cross-verification is sequentially performed on each data partition in the first star catalog; wherein the cross-verification of the first data partition in the first star catalog comprises:

[0007] Searching for a second data partition in a second star catalog that matches the first data partition;

[0008] Cross-verifying each celestial body in the first data partition with each celestial body in the second data partition;

[0009] The second data partition includes a data partition located in the first block and a data partition located in a block surrounding the first block in the second star catalog; the first block is the block where the first data partition is located.

[0010] Preferably, the cross-verification of each celestial body in the first data partition and each celestial body in the second data partition specifically includes:

[0011] sorting the celestial bodies in the second data partition according to a preset order;

[0012] According to a preset order, the celestial bodies in the first data partition are matched as the celestial bodies to be matched in turn; wherein the i-th celestial body is matched as the celestial body to be matched, specifically including:

[0013] Starting from the i-th search position in the second data partition, search for the matching celestial body that is closest to the celestial body to be matched in turn. When a matching celestial body is found, use the next celestial body position of the matching celestial body as the i+1-th search position to match the i+1-th celestial body; wherein the preset order is the size of the right ascension value or the size of the declination value.

[0014] Preferably, searching the second data partition in the second star catalog that matches the first data partition specifically includes:

[0015] Determine a unique index value for each block, and assign the index value of the block to each data partition in the corresponding star catalog;

[0016] Taking the data partition in the second star catalog having the same index value as the first data partition as the central data partition;

[0017] Taking the center point of the central data partition as the center and the shape of the block where the central data partition is located as the reference shape, the reference shape is expanded by a preset multiple to generate a coverage area;

[0018] Each data partition of the second star catalog within the coverage area is used as the second data partition.

[0019] Preferably, dividing the celestial sphere into a plurality of blocks specifically includes:

[0020] Perform multiple divisions, and in each division, divide each i-th level block into N i+1-th level blocks of the same area, until all M-th level blocks are obtained, and use the blocks of each level as each block;

[0021] In the first division, the celestial sphere is divided into K first-level blocks of equal area.

[0022] Preferably, determining a unique index value for each block specifically includes:

[0023] Each time an i+1-th level block is obtained by division, the i+1-th level blocks belonging to the same i-th level block are sorted according to the size of the right ascension value or the size of the declination value to obtain the sequence number of each i+1-th level block;

[0024] Using the hierarchical bit sequence of the i-th level block as the initial bit sequence of the i+1-th level block, adding the serial number of the i+1-th level block to the i+1-th level position of the initial bit sequence of the i+1-th level block, to obtain the hierarchical bit sequence of the i+1-th level block;

[0025] The level bit sequence and level number of the i+1th level block are used together to form the index value of the i+1th level block; wherein the level bit sequence includes M level positions, and each level position is located at a different bit position.

[0026] Preferably, each star catalog is pre-stored in a distributed database, specifically including:

[0027] Each data partition of the star catalog is divided into corresponding data nodes according to blocks; wherein, multiple blocks divided into one data node are continuous, and one star catalog is evenly divided into N data nodes; N is the number of data nodes in the distributed database.

[0028] Preferably, the step of sequentially cross-verifying each data partition in the first star catalog specifically includes:

[0029] N threads are used to respectively read the data of the first star catalog in the N data nodes, so that the N threads perform the cross-certification operation of the first star catalog in parallel.

[0030] Preferably, the method further comprises:

[0031] The ID number of the first celestial body, the ID number of the second celestial body, and the distance between the first celestial body and the second celestial body are written into the cross-verification result table; wherein the first celestial body and the second celestial body are celestial bodies that are matched by cross-verification in the first star catalog and the second star catalog.

[0032] In a second aspect, the present invention further provides a massive star catalog distributed cross-certification device, which is used to implement the massive star catalog distributed cross-certification method described in the first aspect, and the device includes:

[0033] At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the processor to execute the massive star catalog distributed cross-certification method described in the first aspect.

[0034] In a third aspect, the present invention further provides a non-volatile computer storage medium, wherein the computer storage medium stores computer executable instructions, and the computer executable instructions are executed by one or more processors to complete the method described in the first aspect.

[0035] In a fourth aspect, a chip is provided, comprising: a processor and an interface, for calling and running a computer program stored in the memory from a memory, and executing the method of the first aspect.

[0036] According to a fifth aspect, a computer program product comprising instructions is provided. When the instructions are executed on a computer or a processor, the computer or the processor executes the method according to the first aspect.

[0037] The present invention divides the celestial sphere into blocks, so that the data in the blocks can be used as units for cross-verification. When cross-verification is performed, only the data in the corresponding blocks (i.e., the second data partition) need to be traversed without considering the data outside the blocks (the second data partition). Therefore, in the application scenario of massive star catalogs, the number of data accesses required for cross-verification can be reduced, the efficiency of cross-verification can be improved, and resource occupation can be reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments of the present invention. Obviously, the drawings described below are only some embodiments of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0039] Figure 1 It is a flow chart of a first method for distributed cross-certification of a massive star catalog provided by an embodiment of the present invention;

[0040] Figure 2 Schematic diagram of a first massive star catalog distributed cross-certification method provided by an embodiment of the present invention;

[0041] Figure 3 It is a flow chart of a second method for distributed cross-certification of a massive star catalog provided by an embodiment of the present invention;

[0042] Figure 4 It is a flowchart of a third method for distributed cross-certification of a massive star catalog provided by an embodiment of the present invention;

[0043] Figure 5 It is a flowchart of a fourth method for distributed cross-certification of a massive star catalog provided by an embodiment of the present invention;

[0044] Figure 6 It is a flowchart of a fifth method for distributed cross-certification of a massive star catalog provided by an embodiment of the present invention;

[0045] Figure 7 is a schematic diagram of a second massive star catalog distributed cross-certification method provided by an embodiment of the present invention;

[0046] Figure 8 It is a flowchart of a sixth method for distributed cross-certification of a massive star catalog provided by an embodiment of the present invention;

[0047] Fig. 9 It is a flowchart of a seventh method for distributed cross-certification of a massive star catalog provided by an embodiment of the present invention;

[0048] Fig.10 It is a flowchart of an eighth method for distributed cross-certification of a massive star catalog provided by an embodiment of the present invention;

[0049] Fig.11 is a schematic diagram of a third massive star catalog distributed cross-certification method provided by an embodiment of the present invention;

[0050] Fig.12 It is a schematic diagram of the architecture of a massive star catalog distributed cross-certification device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0052] Unless the context requires otherwise, throughout the specification and claims, the term "including" is to be interpreted as open inclusion, that is, "including, but not limited to". In the description of the specification, the terms "one embodiment", "some embodiments", "exemplary embodiments", "examples", "specific examples" or "some examples" and the like are intended to indicate that specific features, structures, materials or characteristics associated with the embodiment or example are included in at least one embodiment or example of the present disclosure. The schematic representation of the above terms does not necessarily refer to the same embodiment or example. In addition, the specific features, structures, materials or characteristics may be included in any one or more embodiments or examples in any appropriate manner, that is, although they may be carried in the embodiments or examples of the above terms due to reasons such as the order and position of appearance, it is not limited to that they can be carried in combination by one embodiment or example.

[0053] In the description of the present invention, the terms "first" and "second" are used only for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of the present disclosure, unless otherwise specified, the meaning of "multiple" is two or more. In addition, for example, the same type of nouns may be described as two independent individuals by adding "A" and "B" at the end. In this case, the corresponding features defined as "A" and "B" are only used to distinguish the same type of individuals for description purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features.

[0054] In the description of the present invention, the expression "A and / or B" (where A and B are used to formally represent specific characteristic contents) will be involved, and the corresponding expressions include the following three combinations: only A, only B, and a combination of A and B.

[0055] As used herein, "about," "substantially," or "approximately" includes the stated value and an average value that is within an acceptable range of deviation from the particular value as determined by one of ordinary skill in the art taking into account the measurements in question and the errors associated with the measurement of the particular quantity (i.e., the limitations of the measurement system).

[0056] In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0057] Embodiment 1:

[0058] In the prior art, star catalog authentication is usually performed on the entire star catalog. For example, the means used in the prior art to cross-authenticate star catalog A and star catalog B is: for celestial body a in star catalog A, it is necessary to compare the position of celestial body a with each celestial body in star catalog B until the celestial body in star catalog B that is closest to celestial body a is obtained. This method results in that for each cross-authentication of a celestial body in star catalog A, it is necessary to traverse star catalog B once, resulting in a large amount of time and resources being consumed when cross-authenticating massive star catalogs with large observation data. In order to solve this problem, embodiment 1 of the present invention provides a distributed cross-authentication method for massive star catalogs, which divides the celestial sphere into multiple blocks, and divides each celestial body in the star catalog into corresponding blocks according to the mapping position of the celestial body on the celestial sphere to form corresponding data partitions; cross-authenticates each data partition in the first star catalog in turn; wherein, cross-authenticates the first data partition in the first star catalog, such as Figure 1 As shown, including:

[0059] In step 201, a second data partition matching the first data partition is searched in the second star catalog; the division method used to divide the celestial sphere into multiple blocks is obtained by those skilled in the art based on empirical analysis. In actual use, any method that can divide the celestial sphere into multiple blocks can be applied to this embodiment. It should be noted that the celestial bodies in the star catalog are actually celestial body data. For the sake of simplicity, this embodiment simply refers to celestial body data as celestial body.

[0060] In step 202, each celestial body in the first data partition and each celestial body in the second data partition are cross-verified; the cross-verification is specifically to verify whether two celestial bodies in different data tables are the same celestial body. The data tables used for celestial body verification are usually celestial body data observed by different devices at the same time or at a similar time. Due to the difference in observation equipment, the observed celestial bodies may be offset in position, making it difficult for the celestial bodies to completely overlap, so cross-verification is required.

[0061] The second data partition includes a data partition located in the first block and a data partition located in a block surrounding the first block in the second star catalog; the first block is the block where the first data partition is located.

[0062] The data partitions of the blocks surrounding the first block may be data blocks that share the same edge or point with the first block, or may be all blocks within a preset number of blocks from the first block. The specific selection method of the surrounding blocks is determined by those skilled in the art according to demand analysis. The purpose of including the data partitions of the blocks surrounding the first block in the second data partition is to include the surrounding blocks in the scope of cross-certification, thereby compensating for the possible position offset of celestial bodies between the two star catalogs.

[0063] This embodiment divides the celestial sphere into blocks, so that cross-verification can be performed using the data in the blocks as units. When cross-verification is performed, only the data in the corresponding blocks (i.e., the second data partition) need to be traversed without considering the data outside the blocks (the second data partition). This can reduce the number of data accesses required for cross-verification in the application scenario of massive star catalogs, improve the efficiency of cross-verification, and reduce resource usage.

[0064] In an optional implementation, the sphere may be divided into a plurality of diamond blocks, each of which is a block. In this case, the second data partition corresponding to the corresponding first data partition may be expressed as follows: Figure 2 shown.

[0065] In order to further improve the efficiency of cross-verification, this embodiment also provides a preferred implementation method, that is, the celestial bodies in the first data partition are cross-verified with the celestial bodies in the second data partition, such as Figure 3As shown, specifically including:

[0066] In step 301, the celestial bodies in the second data partition are sorted according to a preset order.

[0067] In step 302, the celestial bodies in the first data partition are matched as celestial bodies to be matched in turn according to a preset order; wherein, matching the i-th celestial body as the celestial body to be matched specifically includes: starting from the i-th search position in the second data partition, searching for the matching celestial body closest to the celestial body to be matched in turn, and when the matching celestial body is found, matching the i+1-th celestial body with the next celestial body position of the matching celestial body as the i+1-th search position; wherein, the preset order is the size of the right ascension value or the size of the declination value.

[0068] For example, if the first data partition is arranged in a preset order as celestial body a, celestial body b and celestial body c; the second data partition is arranged in a preset order as celestial body d, celestial body e and celestial body f; if in a round of matching, celestial body a is used as the celestial body to be matched, and celestial body d, celestial body e and celestial body f are searched to obtain that celestial body a matches celestial body e, that is, when celestial body a is used as the celestial body to be matched, the matched celestial body obtained is celestial body e, then in the next round of matching, the search is performed from celestial body f, and celestial body d and celestial body e located before celestial body f are no longer included in the subsequent search range. The celestial body to be matched and the corresponding matching celestial body can be regarded as data obtained by observing the same celestial body.

[0069] This embodiment sorts the celestial bodies in the second data partition according to the size of the right ascension value or the size of the declination value, and sequentially traverses and accesses the first data partition according to the same rule, so that the subsequent search range can be gradually narrowed by finding the matching celestial bodies, the number of data accesses can be further reduced, and the cross-verification efficiency can be improved.

[0070] In an actual application scenario, the second data partition matching the first data partition in the second star catalog is searched, such as Figure 4 As shown, specifically including:

[0071] In step 401, a unique index value is determined for each block, and the index value of the block is assigned to each data partition in the corresponding star catalog.

[0072] In step 402, a data partition in the second star catalog having the same index value as the first data partition is used as a central data partition.

[0073] In step 403, taking the center point of the central data partition as the center and the shape of the block where the central data partition is located as the reference shape, the reference shape is expanded by a preset multiple to generate a coverage area.

[0074] In step 404, each data partition of the second star catalog within the coverage area is used as a second data partition, wherein the second data partition includes the central data partition.

[0075] In order to further improve the efficiency of massive star catalogs, this embodiment also uses a distributed database for data storage, that is, each star catalog is pre-stored in the distributed database, specifically including:

[0076] The data partitions of the star table are divided into corresponding data nodes according to blocks; wherein, the multiple blocks divided into one data node are continuous, and one star table is evenly divided into N data nodes; N is the number of data nodes in the distributed database. The continuity between the multiple blocks divided into one data node means that the multiple blocks can be connected to form a complete coverage area, which is manifested in that at least two of the multiple blocks share the same edge or point in actual use.

[0077] Correspondingly, the cross-verification of each data partition in the first star catalog in sequence specifically includes: using N threads to read the data of the first star catalog in N data nodes respectively, so that the N threads perform the cross-verification operation of the first star catalog in parallel. The data in the star catalog is a celestial body, and in specific use, includes celestial body information such as the declination value and the right ascension value of the celestial body.

[0078] In actual use, the method also includes: writing the ID number of the first celestial body, the ID number of the second celestial body, and the distance between the first celestial body and the second celestial body into a cross-verification result table; wherein the first celestial body and the second celestial body are celestial bodies that are matched by cross-verification in the first star catalog and the second star catalog.

[0079] In a preferred embodiment, the celestial sphere is divided into a plurality of blocks, such as Figure 5 As shown, specifically including:

[0080] In step 501 , multiple divisions are performed, and in each division, each i-th level block is divided into N i+1-th level blocks of the same area.

[0081] In step 502, until all the M-th level blocks are obtained, each level block is regarded as each block; the blocks of each level are all the level blocks obtained by the division. In the first division, the celestial sphere is divided into K first-level blocks of the same area.

[0082] In actual use, the values ​​of M, N and K are obtained by technicians in this field according to the needs. In a specific application scenario, K = 12, N = 4, then for the celestial sphere, the number of L-level blocks N = 3×4 L .

[0083] The unique index value is determined for each block, such as Figure 6 As shown, specifically including:

[0084] In step 601, each i+1th level block is divided and sorted according to the size of the right ascension value or the size of the declination value to obtain the serial number of each i+1th level block.

[0085] In step 602, the hierarchical bit sequence of the i-th level block is used as the initial bit sequence of the i+1-th level block, and the serial number of the i+1-th level block is added to the i+1-th level position of the initial bit sequence of the i+1-th level block to obtain the hierarchical bit sequence of the i+1-th level block; wherein, if the i+1-th level block is located in the corresponding i-th level block, then the i-th level block is the i-th level block to which the i+1-th level block belongs, that is, the above-mentioned i-th level block.

[0086] In step 603, the hierarchical bit sequence and the level number of the i+1th level block are used together to form the index value of the i+1th level block; wherein the hierarchical bit sequence includes M level positions, and each level position is located at a different bit position. The number of bits of the hierarchical bit sequence is obtained by a technician in this field according to the demand analysis, and the value of each bit in the initial bit sequence of each level 1 block is a preset value (such as: 0). It should be noted that the bit sequence in this embodiment is a bit value sequence, that is, a sequence composed of multiple bits. The level number refers to the level block, such as the level number of the i-th level block is i, and so on, which will not be elaborated here.

[0087] For example, if M=15, Figure 7 As shown, Figure 7 The three division processes are shown, and a 32-bit bit sequence is used as the hierarchical bit sequence, in the order of high to low bits (in Figure 7 The highest 4-bit sequence is assigned to the first level, and the subsequent 2-bit sequences are assigned to the second, third, ..., and fifteenth level. Assuming that the first level block obtained in the first division is numbered 3, Figure 7In the first-level block [3], the highest 4 bits of the 32-bit bit sequence of all 0s are filled with 3 (binary 0011), and the hierarchical bit sequence of the first-level block is 001100000000000000000000000000000. In the second division, the first-level block [3] of the first division is further divided to obtain 4 second-level blocks, namely [3, 0], [3, 1], [3, 2] and [3, 3], the generated hierarchical bit sequences are 0011000000000000000000000000000000 (the hierarchical bit sequence of [3, 0]), 00110100000000000000000000000000000 (the hierarchical bit sequence of [3, 1]), 00111000000000000000000000000 000 (the hierarchical bit sequence of [3, 2]) and 00111100000000000000000000000000000 (the hierarchical bit sequence of [3, 3]), and so on. For the first-level block with sequence number 2 in the first division, the hierarchical bit sequences of the four second-level blocks obtained in the second division are 0010000000000000000000000000000000 (the hierarchical bit sequence of [3, 3]), respectively. (the level bit sequence of [2, 0]), 001001000000000000000000000000000 (the level bit sequence of [2, 1]), 001010000000000000000000000000000 (the level bit sequence of [2, 2]), and 0010110000000000000000000000000000 (the level bit sequence of [2, 3]).

[0088] The level bit sequence of the third-level block [3, 0, 1] obtained in the third division is 0011000100000000000000000000000000, and the level bit sequence of the third-level block [3, 3, 3] is 0011111100000000000000000000000000.

[0089] It should be noted that for the sake of simplicity, this embodiment uses [n1, n2, ..., n j ] to represent each level of blocks, where n j Represents the serial number of the j-th level block, and n1, n2, etc. represent the serial numbers of the parent blocks.

[0090] This embodiment iteratively divides the celestial sphere surface and encodes the index value according to the method described in steps 601 to 603, so that the position of the level block on the celestial sphere surface can be determined according to the index value of the corresponding level block, and the surrounding blocks of the corresponding block can be quickly determined.

[0091] In an optional implementation, dividing each celestial body in the star catalog into corresponding blocks specifically includes: dividing each celestial body in the star catalog into M-th level blocks. Each data partition of the second star catalog within the coverage area is obtained by iterative query, specifically including:

[0092] From each i+1-th level block located in the i-level target block, the i+1-level target block that has a spatial intersection with the coverage area is obtained by screening, until all M-level target blocks are screened, that is, each block within the coverage area, and the data partition corresponding to the second star catalog in each M-level target block is the data partition of the second star catalog within the coverage area; wherein, the level 1 target block that has a spatial intersection with the coverage area is obtained by screening from the 1-level blocks.

[0093] In actual use, bit sequences are also used to represent level numbers. For example, a 64-bit bit sequence is used as an index value, where the upper 32 bits represent the level of the block, i.e., the level number sequence, and the lower 32 bits are the level bit sequence. In actual use, the level number sequence of the block is shifted left by 32 bits, and then the level bit sequence is supplemented in the lower 32 bits to obtain a new value, which is the unique index value. Due to the limitation of the number of encoding bits, the unique condition of the level bit sequence represented by the 64-bit bit sequence is that it needs to meet 3×4 L <2 32 , L is the number of the highest level, that is, the M mentioned above. By calculating the maximum integer value of L, it is 15, which means that it can only express 15 levels at most.

[0094] Embodiment 2:

[0095] The present invention is based on the method described in Example 1, combined with specific application scenarios, and uses technical descriptions in related scenarios to illustrate the implementation process of the present invention in characteristic scenarios.

[0096] This embodiment uses the DAMO distributed database as an example, in which the DAMO distributed database provides stateless computing nodes (abbreviated as: SP), distributed data storage units (abbreviated as: BP, also known as: data nodes), and provides distributed data storage.

[0097] Taking M=13 as an example, the present embodiment provides a massive star catalog distributed cross-verification method, such as Figure 8 As shown, it includes the following steps:

[0098] In step 701, star catalog A (i.e., the first star catalog in Example 1) and star catalog B (i.e., the second star catalog in Example 1) are respectively created in the DAMO distributed database. In addition to the basic information of the star catalog, an additional index value field is added to represent the data partition where the celestial body is located.

[0099] In step 702, the celestial sphere is partitioned using the preset partitioning rules for star catalog A and star catalog B to obtain blocks. The data in star catalog A and star catalog B are divided according to the 13th level blocks to obtain data partitions, and the index value field is used to represent each data partition, that is, the index value of each data partition (3×4 13 The index value of each data partition at the 13th layer (i.e., the index value of each 13th level block) is calculated, and is divided into N intervals in sequence (the N value is equal to the number of BP nodes). The preset partitioning rule has been described in detail in Example 1, that is, the celestial sphere is iteratively divided until all 13th level blocks are obtained.

[0100] In step 703, the star catalog data is entered into star catalog A and star catalog B respectively. During the entry process, the index value in layer 13 is calculated according to the right ascension value and declination value of the star catalog, so that the data will fall evenly on different BP nodes according to our partitioning rules.

[0101] In step 704, a star catalog cross-verification result table is created to record the association ID of star catalog A, the association ID of star catalog B, and the spherical distance between the two star catalog data (ie, celestial bodies).

[0102] In step 705, an SQL query is executed in the star table A. After obtaining the number of index values ​​of each data interval in the table A, all index values ​​are obtained in a paging manner and sorted and output to a text file.

[0103] In step 706, the text file generated in step 705 is divided into N parts according to the preset partitioning rules, so that each part corresponds to a BP node, and corresponding N files are generated; empty files are eliminated; the preset partitioning rules here refer to dividing the index values ​​corresponding to the blocks with adjacent positions on the celestial sphere into one text file, which is based on the same concept as the above-mentioned block division and will not be repeated here.

[0104] In step 707, N parallel threads are constructed to read the first line of the N files respectively, that is, the index value currently to be calculated.

[0105] In step 708, each thread performs SQL query in star table A and star table B according to the index value currently required for calculation, and obtains the celestial body data in the corresponding data partition of star table A and star table B for cross-verification, such as Fig. 9As shown, the specific steps are as follows:

[0106] In step 801, the star catalog data set ListA in the index partition (ie, the data partition represented by the index value currently required to be calculated) in table A is read.

[0107] In step 802, the center point of the index partition and the longest semi-axis of the corresponding rhombus are calculated.

[0108] In step 803, the index set covering all partitions of level 13 is calculated through the center point and the longest semi-axis (spherical distance) of the index partition (expanded by 1.2 times, covering a maximum of 9 surfaces, and twice the length, generally speaking, it can be within 2 times). That is, the central data partition in Example 1 is expanded to generate a coverage area, and the index values ​​corresponding to all data partitions (i.e., the second data partition) in the coverage area form the index set.

[0109] In step 804, the data set ListB in the corresponding partition of table B is queried through the calculated index set; that is, the celestial bodies in table B located in the second data partition are obtained.

[0110] In step 805, the data in ListA and ListB are sorted based on the declination values ​​in the data.

[0111] In step 806, the distance (spherical distance) of the point (i.e., celestial body) in ListA to the point (i.e., celestial body) in ListB that is closest to the point (i.e., celestial body) is searched in order, that is, the celestial body in ListA is used as the celestial body to be matched, and the corresponding matching celestial body in ListB is found. After finding the celestial body, the corresponding index Index in ListB (i.e., the position of the matching celestial body) is recorded, and the subsequent points in ListA are searched starting from the Index.

[0112] In step 807, the search is continued in sequence until the traversal of ListA is completed or the Index of ListB reaches the end of the collection.

[0113] In step 808, the corresponding ID values ​​and spherical distances in the matched ListA and ListB are written into the cross-authentication result table.

[0114] Since the on-site reading rules and data partitioning rules are consistent, the DAMO distributed database will assign queries for star catalog A and star catalog B to the corresponding BP nodes for execution. That is, the query tasks of the N parallel threads constructed in step 707 are queried in parallel on different BP nodes, thereby improving data query efficiency.

[0115] After each thread executes step 708, it reads the next line of the file as the next index value to be calculated, and repeats the calculation process of step 708 until all files are read. In this way, the distributed cross-verification process of the entire star catalog is completed, and the results of the cross-verification are saved in the cross-verification result table.

[0116] This embodiment divides the celestial sphere into blocks, so that the data in the blocks can be used as units for cross-verification, so that when cross-verification is performed, only the data in the corresponding blocks (i.e., the second data partition) need to be traversed, without considering the data outside the blocks (the second data partition), so that in the application scenario of massive star catalogs, the number of data accesses required for cross-verification can be reduced, the efficiency of cross-verification can be improved, and resource usage can be reduced. In addition, by adopting the DAMO distributed database for data storage and using multi-threaded parallel computing, the ability of multi-machine parallel collaborative computing is utilized to solve the storage and cross-verification problems of massive star catalog data.

[0117] Embodiment 3:

[0118] The present invention also provides a preferred implementation method based on the methods described in Example 1 and Example 2 for specific application scenarios, such as Fig.10 As shown, the celestial sphere is divided into multiple blocks, specifically including:

[0119] In step 901, after K i-level blocks are obtained, the number of celestial bodies in the first star catalog that fall into each i-level block is calculated. If the number of celestial bodies in the first star catalog in the corresponding i-level block is less than or equal to a second preset number, the i-level block is used as the minimum divided block, and the division of the minimum divided block is stopped; the second preset number is obtained by a technician in this field based on demand analysis. In an optional embodiment, the second preset number may be 1.

[0120] In step 902, if the number of celestial bodies in the corresponding i-level block of the first star catalog is greater than the second preset number, the i-level block is used as a divisible block, and the divisible block is evenly divided into K i+1-level blocks until all the lowest-level blocks are divided into minimum divided blocks.

[0121] The lowest level block refers to the block that has not been divided. When the first division is performed, the celestial sphere is evenly divided into K×M level 1 blocks, and for multiple divisible blocks of the same level, the divisible blocks containing the least number of celestial bodies are divided first. K and M are obtained by those skilled in the art based on empirical analysis. For example, if the second preset number is 10, then Fig.11 As shown in the figure, when the number of celestial bodies in the corresponding block is greater than 10, it is further divided into lower layers, and some of the blocks finally obtained can be as follows Fig.11 As shown, the block size does not necessarily remain the same, but the number of celestial bodies in each lowest-level block is less than or equal to 10. Fig.11 Take the block division in a level 1 block as an example.

[0122] In actual use, after the i-level blocks are divided, each i-level block is judged, specifically including: using two bits as a block identifier, if the number of celestial bodies in the first star catalog in the corresponding i-level block is less than or equal to a second preset number, then the block is marked with a block identifier of 11, indicating that the block is the minimum divided block; if the number of celestial bodies in the first star catalog in the corresponding i-level block is greater than the second preset number, then the block is marked with a block identifier of 00, indicating that the block is a divisible block and the division has not been completed. After each division to obtain the smallest divided block, the block identifiers of each parent block of the smallest divided block are updated. Specifically, when the block identifiers of the direct subordinate blocks of the corresponding parent block are all 11 or 10, the block identifier of the parent block is changed to 10, indicating that the block is a divisible block, but the division of the block has been completed (that is, all its subordinate blocks have been divided into the smallest divided block), until the block identifiers of all level 1 blocks are 11 or 10, the division is stopped.

[0123] This embodiment performs iterative block division and stops when the number of celestial bodies in the divided blocks is less than or equal to a second preset number, thereby controlling the division progress, reducing the number of divisions for areas with fewer celestial bodies, and improving the efficiency of cross-certification.

[0124] The cross-verification of each data partition in the first star catalog in sequence specifically includes:

[0125] Each time a minimum partition block is obtained by division, the first object in the first star catalog located in the minimum partition block is cross-verified with the second object in the second star catalog, and the second object that is cross-verified and matches the first object is deleted from the second star catalog. When the number of second objects in the second star catalog is zero, the block division of the celestial sphere is stopped. For multiple minimum partition blocks of the same level, the minimum partition block containing the least number of celestial bodies is preferentially cross-verified. Since in steps 901-902, for multiple divisible blocks of the same level, the divisible block containing the least number of celestial bodies is divided preferentially, a method of cross-verifying the minimum partition blocks in the order of the number of celestial bodies contained from small to large is formed as a whole, and the matched celestial bodies are deleted from the star catalog, thereby reducing the number of celestial bodies that are cross-verified with the minimum partition blocks when the number of celestial bodies is large in the later stage, thereby improving the efficiency of cross-verification.

[0126] It should be noted that the first celestial body in this embodiment is a celestial body in the first star catalog, and the second celestial body is a celestial body in the second star catalog, which is different from the meaning in Example 1. For a j-level block, its direct superior block is the j-1-level block containing the j-level block, and its superior block is all blocks containing the j-level block, such as the j-2-level block containing the above j-1-level block; its blocks of the same level are other j-level blocks located in the j-1-level block; its subordinate block is the j+1-level block contained in the j-level block.

[0127] The method of cross-verifying the first object in the first star catalog located in the smallest partition block with the second object in the second star catalog is implemented based on the same concept as that in Example 1, namely:

[0128] The first object in the first star catalog located in the smallest partition block is divided into the first data partition, and the second data partition matching the first data partition in the second star catalog is searched; each celestial body in the first data partition is cross-certified with each celestial body in the second data partition; wherein the second data partition includes the data partition in the second star catalog located in the smallest partition block and the data partitions located in the blocks surrounding the smallest partition block.

[0129] Among them, the surrounding blocks of the minimum dividing block are specifically: taking the center point of the minimum dividing block as the center, taking the shape of the block where the minimum dividing block is located as the reference shape, and expanding the reference shape by a preset multiple to generate a coverage area; taking all the minimum dividing blocks that intersect with the coverage area as the surrounding blocks of the minimum dividing block.

[0130] Since the number of celestial bodies distributed at each position of the first star catalog may not be uniform, there are some cases where the minimum partition blocks are large and some are small. At this time, the second data partitions found may also have a large difference in data volume. In order to improve the efficiency of cross-verification, this embodiment also provides a preferred implementation method, that is, the cross-verification of each celestial body in the first data partition and each celestial body in the second data partition specifically includes:

[0131] When the number of celestial bodies in the second data partition is less than a third preset number, each celestial body in the first data partition is directly cross-certified with each celestial body in the second data partition. The third preset number is obtained by those skilled in the art based on empirical analysis.

[0132] When the number of celestial bodies in the second data partition is less than the third preset number, the celestial bodies in the second data partition are screened based on the right ascension value and declination value of the coverage area generated by the smallest divided block, and the celestial bodies in the second star catalog located in the coverage area are obtained, and the celestial bodies in the first data partition are cross-certified with the celestial bodies in the second star catalog located in the coverage area.

[0133] This embodiment also takes into account that in actual use, multiple star catalogs may be cross-certified. Here, this embodiment also provides the following preferred implementation methods for various cross-certification scenarios, specifically including:

[0134] When it is necessary to cross-check one star catalog with one or more other star catalogs, such as determining whether there is a celestial body in the three star catalogs A, B, and C that is the same as the celestial body in star catalog D (i.e., it matches), it is necessary to cross-check star catalog A with star catalog D, cross-check star catalog B with star catalog D, and cross-check star catalog C with star catalog D. The star catalog (such as the above-mentioned star catalog D) is taken as the first star catalog, and is divided into blocks in the manner described in steps 901-902. The other star catalogs (such as the above-mentioned three star catalogs A, B, and C) are sequentially taken as the second star catalogs for cross-checking.

[0135] When multiple star catalogs need to be cross-verified, such as determining whether there are celestial bodies that match each other in the three star catalogs A, B, and C, it is necessary to cross-verify star catalog A with star catalog B, cross-verify star catalog A with star catalog C, and cross-verify star catalog B with star catalog C. The star catalog containing the least number of celestial bodies is used as the first star catalog, and the other star catalogs are used as the second star catalogs. Block division is performed according to the method described in this embodiment, and each second star catalog is cross-verified with the first star catalog, and the celestial bodies that are matched by cross-verification are deleted from the first star catalog and the second star catalog.

[0136] After the cross-verification of the first star catalog with the other star catalogs is completed, a star catalog containing the least number of celestial bodies is reselected as the first star catalog, and the other star catalogs are used as the second star catalog. On the basis of retaining the old block division, the smallest divided block is found for cross-verification with the second star catalog, and the block division and cross-verification are continued according to the method described in step 901-step 902, and so on, until all star catalogs are used as the first star catalog and cross-verified with the other star catalogs.

[0137] Embodiment 4:

[0138] like Fig.12 , is a schematic diagram of the architecture of a massive star catalogue distributed cross-certification device according to an embodiment of the present invention. The massive star catalogue distributed cross-certification device according to this embodiment includes one or more processors 21 and a memory 22. Fig.12 A processor 21 is taken as an example.

[0139] The processor 21 and the memory 22 may be connected via a bus or other means. Fig.12 The example of connecting through bus is taken in the following.

[0140] The memory 22 is a non-volatile computer-readable storage medium that can be used to store non-volatile software programs and non-volatile computer executable programs, such as the massive star catalog distributed cross-authentication method in Example 1, Example 2, or Example 3. The processor 21 executes the massive star catalog distributed cross-authentication method by running the non-volatile software program and instructions stored in the memory 22.

[0141] The memory 22 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage devices. In some embodiments, the memory 22 may optionally include a memory remotely arranged relative to the processor 21, and these remote memories may be connected to the processor 21 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0142] The program instructions / modules are stored in the memory 22, and when executed by the one or more processors 21, the massive star catalog distributed cross-certification method in the above-mentioned embodiment 1, embodiment 2 or embodiment 3 is executed.

[0143] It is worth noting that the information interaction, execution process, etc. between the modules and units within the above-mentioned devices and systems are based on the same concept as the processing method embodiment of the present invention. The specific contents can be found in the description of the method embodiment of the present invention and will not be repeated here.

[0144] A person skilled in the art may understand that all or part of the steps in the various methods of the embodiments may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the storage medium may include: a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, etc.

[0145] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A distributed cross-verification method for massive star catalogs, characterized in that: The celestial sphere is divided into a plurality of blocks, and each celestial body in the star catalog is divided into corresponding blocks according to the mapping position of the celestial body on the celestial sphere to form corresponding data partitions; each data partition in the first star catalog is cross-verified in turn; wherein the cross-verification of the first data partition in the first star catalog includes: Searching for a second data partition in a second star catalog that matches the first data partition; Cross-verifying each celestial body in the first data partition with each celestial body in the second data partition; The second data partition includes a data partition located in the first block and a data partition located in a block surrounding the first block in the second star catalog; the first block is the block where the first data partition is located.

2. The method for distributed cross-certification of massive star catalogs according to claim 1, characterized in that: The cross-verification of each celestial body in the first data partition and each celestial body in the second data partition specifically includes: sorting the celestial bodies in the second data partition according to a preset order; According to a preset order, the celestial bodies in the first data partition are matched as the celestial bodies to be matched in turn; wherein the i-th celestial body is matched as the celestial body to be matched, specifically including: Starting from the i-th search position in the second data partition, search for the matching celestial body that is closest to the celestial body to be matched in turn. When a matching celestial body is found, use the next celestial body position of the matching celestial body as the i+1-th search position to match the i+1-th celestial body; wherein the preset order is the size of the right ascension value or the size of the declination value.

3. The method for distributed cross-certification of massive star catalogs according to claim 1, characterized in that: The step of searching for a second data partition in the second star catalog that matches the first data partition specifically includes: Determine a unique index value for each block, and assign the index value of the block to each data partition in the corresponding star catalog; Taking the data partition in the second star catalog having the same index value as the first data partition as the central data partition; Taking the center point of the central data partition as the center and the shape of the block where the central data partition is located as the reference shape, the reference shape is expanded by a preset multiple to generate a coverage area; Each data partition of the second star catalog within the coverage area is used as the second data partition.

4. The method for distributed cross-certification of massive star catalogs according to claim 3, characterized in that: The dividing of the celestial sphere into a plurality of blocks specifically includes: Perform multiple divisions, and in each division, divide each i-th level block into N i+1-th level blocks of the same area, until all M-th level blocks are obtained, and use the blocks of each level as each block; In the first division, the celestial sphere is divided into K first-level blocks of equal area.

5. The method for distributed cross-verification of massive star catalogs according to claim 4, characterized in that: Determining a unique index value for each block specifically includes: Each time an i+1-th level block is obtained by division, the i+1-th level blocks belonging to the same i-th level block are sorted according to the size of the right ascension value or the size of the declination value to obtain the sequence number of each i+1-th level block; Using the hierarchical bit sequence of the i-th level block as the initial bit sequence of the i+1-th level block, adding the serial number of the i+1-th level block to the i+1-th level position of the initial bit sequence of the i+1-th level block, to obtain the hierarchical bit sequence of the i+1-th level block; The level bit sequence and level number of the i+1th level block are used together to form the index value of the i+1th level block; wherein the level bit sequence includes M level positions, and each level position is located at a different bit position.

6. The method for distributed cross-certification of massive star catalogs according to claim 4, characterized in that: Each star catalog is pre-stored in a distributed database, including: Each data partition of the star catalog is divided into corresponding data nodes according to blocks; wherein, multiple blocks divided into one data node are continuous, and one star catalog is evenly divided into N data nodes; N is the number of data nodes in the distributed database.

7. The method for distributed cross-verification of massive star catalogs according to claim 6, characterized in that: The cross-verification of each data partition in the first star catalog in sequence specifically includes: N threads are used to respectively read the data of the first star catalog in the N data nodes, so that the N threads perform the cross-certification operation of the first star catalog in parallel.

8. The method for distributed cross-verification of massive star catalogs according to any one of claims 1 to 7, characterized in that: The method also includes: The ID number of the first celestial body, the ID number of the second celestial body, and the distance between the first celestial body and the second celestial body are written into the cross-verification result table; wherein the first celestial body and the second celestial body are celestial bodies that are matched by cross-verification in the first star catalog and the second star catalog.

9. A non-volatile computer storage medium, characterized in that: The computer storage medium stores computer executable instructions, which are executed by one or more processors to complete the massive star catalog distributed cross-certification method described in any one of claims 1-8.

10. A massive star catalogue distributed cross-certification device, characterized in that: The device comprises: At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the processor to execute the massive star catalog distributed cross-certification method described in any one of claims 1-8.

Images