An astronomical cross-verification confidence calculation method for memory access optimization
Patent Information
- Application Number
- CN202610564227.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-04-27
- Publication Date
- 2026-07-24
Smart Images
Figure CN122450860A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of astronomical big data processing, high-performance computing, and statistical probability calculation technology, and in particular to a memory-optimized method for calculating the confidence level of astronomical cross-validation. Background Technology
[0002] In the field of astronomical observation, with the upgrading of observational equipment (such as EP and eROSITA), observations of the same celestial region often generate multiple star catalogs with different bands and different messengers. Astronomical cross-verification refers to determining which records correspond to the same physically real celestial object by comparing the positions, errors, fluxes, and other physical characteristics of celestial objects in different catalogs. To measure the reliability of this matching relationship, Bayesian confidence level calculation has become a core component of current quantitative analysis.
[0003] At the computational implementation level, the data layout determines how celestial properties are organized in physical memory, directly affecting computational performance. However, existing astronomical identification tools (such as NWAY) are mainly based on object-oriented design patterns, using an array of structures (AoS) data layout to encapsulate each main source and its corresponding set of candidate bodies into nested object structures.
[0004] Based on this, the above model suffers from the following technical drawbacks when processing massive datasets on the scale of hundreds of millions: 1) Memory wall and memory access performance bottleneck: Since objects and their nested candidate sets are usually discretely distributed in physical memory, the computing engine needs to frequently cross discontinuous memory addresses to perform "pointer chasing" when evaluating confidence. This causes the hardware prefetching mechanism of modern processors to fail, resulting in extremely high cache miss rates and ineffective utilization of memory bandwidth, severely limiting data throughput; 2) Large memory redundancy overhead: Traditional AoS layouts often carry a large amount of non-computation-related metadata, pointers, and alignment padding fields when storing each celestial object. When processing candidate pairs on the scale of hundreds of millions, this redundancy overhead increases exponentially, often leading to single-machine memory overflow, making it difficult for ordinary workstations to handle real-time or near-real-time processing of large-scale astronomical data; 3) Computational efficiency is hindered and SIMD is limited: The nested logical structure makes the data layout and computational logic deeply coupled. Because data is not contiguous in memory, processors struggle to use vectorized instructions such as Single Instruction Multiple Data (SIMD) to accelerate multiple candidate pairs in parallel, resulting in huge computational overhead during the computationally intensive likelihood evaluation and probability reduction stages, which prevents modern hardware from fully realizing its performance potential.
[0005] Therefore, it is necessary to propose an astronomical cross-validation confidence calculation method to address the technical problems caused by traditional nested data objects in existing large-scale datasets, such as irregular memory access patterns, memory wall bottlenecks, and limited computational efficiency. Summary of the Invention
[0006] The purpose of this invention is to provide a memory-optimized astronomical cross-validation confidence calculation method that solves the aforementioned technical problems, based on existing Bayesian statistical confidence calculation methods.
[0007] Therefore, the technical solution of the present invention is as follows:
[0008] A memory-optimized method for calculating the confidence level of astronomical cross-validation includes the following steps:
[0009] S1. Obtain the astronomical cross-verification results of the main star catalog and the candidate star catalog, and flatten them into: a one-dimensional main source index array that sequentially records the position index of the main star catalog objects in the original star catalog, and a one-dimensional candidate source index array that synchronously and sequentially records the position index of the candidate objects in the candidate star catalog that form candidate pairs with each main star catalog object in the original star catalog.
[0010] S2. Construct an offset array to record the candidate interval of each primary catalog object in the one-dimensional primary source index array corresponding to multiple candidate catalog objects in the one-dimensional candidate source index array.
[0011] S3. Construct multiple columnar one-dimensional continuous celestial body attribute arrays for the main star catalog objects, which record the positional and physical characteristics of the main star catalog objects according to the recording order of the main star catalog objects in the one-dimensional main source index array; construct multiple columnar one-dimensional continuous celestial body attribute arrays for the candidate star catalog objects, which record the positional and physical characteristics of the candidate star catalog objects according to the recording order of the candidate star catalog objects in the one-dimensional candidate source index array.
[0012] S4. Based on the one-dimensional primary source index array and the one-dimensional candidate source index array, and using the offset array, perform a linear scan on all candidate pairs in sequence to prefetch the celestial properties used to calculate the logarithmic position correlation likelihood value and physical feature consistency score of each candidate pair, and generate an unnormalized logarithmic score array after accumulation.
[0013] S5. Treat all candidate catalog objects corresponding to each main catalog object as a candidate group, use the offset array to pre-fetch multiple unnormalized logarithm scores corresponding to the candidate group in batches, and perform log-sum-exp reduction operation within the group to calculate the normalized probability of each candidate pair in the candidate group and the total confidence of the candidate group.
[0014] S6. Judge the result of step S5 according to the set threshold and output the candidate pairs with high credibility.
[0015] Furthermore, the specific operation of step S2 is as follows:
[0016] S201. Based on the total number of celestial bodies in the main star catalog. The construction length is offset array and set the offset array sequentially. index number 0~ ;
[0017] S202, in the offset array The internal sequence records the candidate intervals of each primary catalog object in the one-dimensional primary source index array and multiple candidate catalog objects in the one-dimensional candidate source index array. array elements The corresponding storage in the one-dimensional primary source index array is the first A primary star catalog object in a one-dimensional candidate source index array The first candidate star index astronomical number, array element Corresponding storage one-dimensional primary source index array Middle A primary star catalog object in a one-dimensional candidate source index array The index number of the first candidate star catalog object in the catalog;
[0018] S203. Set the sentinel value, which is the total number of all candidate pairs in the astronomical cross-verification results of the main star catalog and the candidate star catalog, and store it in the offset array. The last array middle.
[0019] Furthermore, in step S3, the positional features include right ascension, declination, and observation error radius, and the physical features are selected from at least one of magnitude, flow rate, and color.
[0020] Furthermore, the specific operation steps of step S4 are as follows:
[0021] S401. Using the offset array, the right ascension, declination, and observation error radius of the primary and candidate objects in each candidate pair are pre-selected sequentially. The spherical angular distance and joint position error of each candidate pair are calculated to obtain the logarithmic position correlation likelihood value.
[0022] S402. Using the offset array, pre-extract the physical characteristics of the main catalog objects and candidate catalog objects in each candidate pair in turn to calculate the physical characteristic consistency score of each candidate pair.
[0023] S403. Sequentially read the logarithmic positional relevance likelihood value and physical feature consistency score of each candidate pair in step S401, and perform weighted summation to generate an unnormalized logarithmic score array that is the same length as the one-dimensional candidate source index array and is contiguous in memory.
[0024] Further, in step S401, the right ascension, declination, and observation error radius of the primary and candidate catalog objects in each candidate pair are pre-selected sequentially. The method for calculating the logarithmic position correlation likelihood value of each candidate pair is as follows:
[0025] 1) Sequentially read the index numbers of the main star catalog objects in the one-dimensional main source index array, and read the candidate intervals corresponding to the index numbers in the offset array according to the index numbers of the main star catalog objects, so as to determine the first and last index numbers of the corresponding multiple candidate star catalog objects in the one-dimensional candidate source index array;
[0026] 2) Based on the index number of the main star catalog objects, pre-select the data corresponding to the same index number from the one-dimensional continuous right ascension array, one-dimensional continuous declination array, and one-dimensional continuous observation error radius array of the main star catalog objects respectively;
[0027] 3) Based on the first and last index numbers of the corresponding candidate star catalog objects, pre-retrieve the data corresponding to the same first and last index numbers in batches from the one-dimensional continuous right ascension array, one-dimensional continuous declination array, and one-dimensional continuous observation error radius array of the candidate star catalog objects;
[0028] 4) Following the prefetching order of steps 2) and 3), calculate the spherical angular distance and joint position error between the main catalog object and its corresponding candidate catalog objects in turn, so as to obtain the logarithmic position correlation likelihood value of each candidate pair.
[0029] Further, in step S402, the method for pre-taking various physical characteristics of the primary and candidate celestial bodies in each candidate pair and calculating the physical characteristic consistency score of each candidate pair is as follows:
[0030] 1) Based on the index number of the main star catalog celestial body determined in step S401, pre-retrieve the data corresponding to the same index number from the continuous array of one-dimensional physical characteristics of the main star catalog celestial bodies;
[0031] 2) Based on the first and last index numbers of the corresponding candidate star catalog objects determined in step S401, pre-retrieve data corresponding to the same first and last index numbers in batches from the continuous array of one-dimensional physical characteristics of the candidate star catalog objects;
[0032] 3) Calculate the physical feature consistency score for each candidate pair in the order of prefetching in steps 1) and 2).
[0033] Furthermore, for any primary star catalog object, the steps for calculating the normalized probability of each candidate pair within its corresponding candidate group and the total confidence score of the candidate group are as follows:
[0034] S501, using offset array Unnormalized log score of batch prefetched candidate groups
[0035] S502, Use the maximum log score within the candidate group as the offset;
[0036] S503, Calculate the aggregate likelihood logarithm : , The score of the kth unnormalized logarithm within the candidate group;
[0037] S504. Obtain the source density of the corresponding covered sky area for the main star catalog and candidate star catalog. and The background source score is obtained by logarithmic addition of the two. : ;
[0038] S505, Calculate the total logarithmic partition function of candidate groups. : ;
[0039] S506. Calculate the normalized probability of each candidate pair within the candidate group. : and the total confidence level of the candidate group to which it belongs. : .
[0040] Furthermore, the specific steps of step S501 are as follows: based on the celestial body index number of the main star catalog... Read the candidate interval corresponding to the index number within the offset array, and determine the first and last index numbers of the consecutively stored segments of the corresponding candidate catalog objects in the one-dimensional candidate source index array; based on the first and last index numbers of the candidate catalog objects, pre-fetch the unnormalized logarithmic scores corresponding to the same first and last index numbers in batches from the unnormalized logarithmic score array.
[0041] Compared with existing technologies, this memory-optimized astronomical cross-identification confidence calculation method, in the evaluation and calculation of cross-identification confidence of large-scale astronomical catalogs, achieves continuous structured linear data storage by constructing a one-dimensional primary source index array, a one-dimensional candidate source index array, and a columnar one-dimensional continuous celestial object attribute array recording primary catalog objects and candidate catalog objects using the same recording order. Simultaneously, it constructs an offset array that can be used as an address guide to achieve batch prefetching of data. Furthermore, it reorganizes the discrete traversal process based on candidate groups in traditional methods into a rule execution mode combining candidate pair-level linear scanning and candidate group-level segmented reduction. Compared with implementations that maintain candidate relationships using object pointers, DataFrames, or nested containers, this method enables the CPU to access memory in a streaming manner and more fully utilizes hardware prefetching and multi-level caching, significantly improving memory access locality and execution efficiency, significantly increasing execution efficiency, reducing cache miss rate, and simultaneously reducing data structure redundancy and peak memory overhead, thus reducing peak runtime memory usage. Attached Figure Description
[0042] Figure 1 A flowchart of the memory access optimization astronomical cross-validation confidence calculation method of the present invention;
[0043] Figure 2 This is a schematic diagram of the data layout for the memory-optimized astronomical cross-validation confidence calculation method in an embodiment of the present invention. Detailed Implementation
[0044] The present invention will be further described below with reference to the accompanying drawings and specific embodiments, but the following embodiments are by no means intended to limit the present invention.
[0045] See Figure 1 The specific implementation steps of the memory-optimized astronomical cross-validation confidence calculation method are described below.
[0046] S1. Obtain the astronomical cross-verification results of the main star catalog and the candidate star catalogs, and flatten them into a one-dimensional main source index array that sequentially records the position indices of objects in the main star catalog within the original star catalog by traversing all candidate pairs. And a one-dimensional candidate source index array that synchronously records the position indices of all candidate objects in the original star catalog corresponding to the objects in the main star catalog. .
[0047] In step S1, the astronomical cross-verification results of the main star catalog and the candidate star catalog are candidate results with a nested grouping structure. Directly verifying the confidence of these candidate results is easily affected by processing defects such as irregular memory access, candidate group boundary dependency, and object construction, thereby limiting cache locality, vectorization capability, and overall execution efficiency.
[0048] Based on this, this step involves constructing a one-dimensional master source index array. and one-dimensional candidate source index array The flattened data from the astronomical cross-verification results of the main star catalog and candidate star catalogs is stored. The specific operations of this step are: constructing one-dimensional main source index arrays respectively. and one-dimensional candidate source index array Then, iterate through all candidate pairs in the astronomical cross-validation results, and use a one-dimensional primary source index array. The sequence records the position indices of objects in the main star catalog within the original star catalog; a one-dimensional candidate source index array. The position index of candidate catalog objects that form candidate pairs with objects in each main catalog is recorded in the original catalog in a synchronous sequence.
[0049] Therefore, step S1 eliminates pointer dependencies in traditional nested objects, simplifies complex topological relationships into linear correspondences, and transforms the originally discrete grouping logic into two contiguous one-dimensional integer arrays of different lengths; among them, the one-dimensional main source index array... Each primary star in the index has a unique index number, and a one-dimensional candidate source index array. Although there are duplicates, each candidate catalog object also has a unique index number; and for the one-dimensional primary source index array For any primary star in the catalog, its one-dimensional candidate source index array The corresponding candidate star catalog objects in the data are stored as a continuous segment.
[0050] S2. Construct an offset array to record the one-dimensional master source index array. Each primary star in the catalog corresponds to a one-dimensional candidate source index array. The candidate intervals of multiple candidate star catalogs are used to explicitly express the grouping boundaries of candidate groups formed by multiple candidate stars without relying on nested objects.
[0051] Specifically, the implementation steps of step S2 are described below.
[0052] S201 is based on the total number of celestial bodies in the main star catalog. The construction length is offset array and set the offset array sequentially. index number 0~ .
[0053] S202, in the offset array Internal sequential record one-dimensional primary source index array Each primary star catalog object should be placed in a one-dimensional candidate source index array. Candidate intervals for multiple candidate star catalogs , where array elements Corresponding storage one-dimensional primary source index array Middle A primary star catalog object in a one-dimensional candidate source index array The index number of the first candidate star in the catalog, array element Corresponding storage one-dimensional primary source index array Middle A primary star catalog object in a one-dimensional candidate source index array The index number of the first candidate star catalog object in the catalog; This serves as the sequence number of the celestial bodies in the main star catalog, and also as a one-dimensional main source index array. Index number of celestial objects in the Middle Main Stars Index.
[0054] S203. Set the sentinel value, which is the total number of candidate pairs in the astronomical cross-validation results, and store it in the offset array. The last array In order to ensure the one-dimensional primary source index array The boundaries of all candidate star catalog objects corresponding to the last primary star catalog object can be correctly accessed.
[0055] In step S2, the offset array It can be used as an "address guide" in subsequent operations; specifically, based on the serial numbers of the celestial bodies in the main star catalog. It can realize a one-dimensional master source index array The prefetch index number is The data; simultaneously, based on the one-dimensional primary source index array index number It is possible to use the offset array Reading adjacent elements and The index number of the candidate star catalog object is stored internally to determine the sequence number of the main star catalog object. The first and last index numbers of the corresponding candidate star catalog objects can then be used to realize a one-dimensional candidate source index array. The data segment between the first and last index numbers is prefetched. The offset array... The setting of the sentinel value is used to allow the computation kernel to locate any set of computation ranges with very low overhead (constant time complexity).
[0056] S3. Construct multiple columnar one-dimensional continuous celestial body attribute arrays for the main star catalog objects, which are respectively arranged according to the one-dimensional main source index array. The recording order of objects in the inner main catalog records the positional and physical characteristics of the objects in the main catalog; multiple columnar one-dimensional continuous object attribute arrays of candidate catalog objects are constructed, which are respectively arranged according to the one-dimensional candidate source index array. The recording order of celestial objects in the inner candidate catalog records the positional and physical characteristics of the celestial objects in the candidate catalog; among them, the positional characteristics include equatorial longitude, equatorial latitude and observation error radius, and the physical characteristics are selected from at least one of magnitude, flux and color.
[0057] Specifically, the implementation steps of step S3 are described as follows.
[0058] S301. Construct multiple columnar one-dimensional continuous attribute arrays for the main star catalog objects, specifically a one-dimensional continuous right ascension array, a one-dimensional continuous declination array, a one-dimensional continuous observation error radius array, and at least one-dimensional continuous physical characteristic array; each group is arranged according to the one-dimensional main source index array. The order in which the celestial bodies in the inner primary star catalog are recorded records the different celestial attributes of each primary star catalog body.
[0059] S302. Construct multiple columnar one-dimensional continuous attribute arrays for the candidate star catalog, specifically a one-dimensional continuous right ascension array, a one-dimensional continuous declination array, a one-dimensional continuous observation error radius array, and at least one-dimensional continuous physical characteristic array; similarly, each array is arranged according to a one-dimensional candidate source index array. The order in which the celestial bodies in the inner candidate star catalog are recorded, and the different celestial properties of each candidate star catalog are recorded respectively.
[0060] In steps S301 and S302 above, the one-dimensional continuous right ascension array sequentially records the equatorial longitude RA of the main celestial bodies or candidate celestial bodies, the one-dimensional continuous declination array sequentially records the equatorial latitude Dec of the main celestial bodies or candidate celestial bodies, and the one-dimensional continuous observation error radius array sequentially records the observation error radius of the main celestial bodies or candidate celestial bodies. The physical features of objects in the main catalog or candidate catalog are recorded sequentially in a one-dimensional continuous physical feature array. A specific physical feature (magnitude, flux, or color) of an object in the main catalog or candidate catalog is recorded sequentially in a one-dimensional continuous physical feature array.
[0061] Based on the above construction method, the one-dimensional continuous celestial body attribute array of the main star catalog objects has the same properties as the one-dimensional main source index array. With the same length and consistent correspondence, the one-dimensional continuous object attribute array of candidate star catalog objects has the same properties as the one-dimensional candidate source index array. The same length and consistent correspondence; therefore, in subsequent calculations, when prefetching various celestial properties of objects in the main catalog and candidate catalogs, the offset array can also be used. Implemented as an "address guide".
[0062] In summary, the advantage of constructing a one-dimensional continuous celestial attribute array for each celestial body attribute in step S3 is that, in the actual calculation process, only the specific data involved in the calculation needs to be obtained according to the index number provided by the bias array, avoiding the loading of irrelevant metadata, and significantly improving the cache hit rate and memory bandwidth utilization of modern processors; at the same time, the separate construction of the one-dimensional continuous celestial attribute array and the index array in step S1 can also avoid introducing irrelevant redundant fields in the numerical calculation process, thereby improving the effective data density.
[0063] S4. Based on the one-dimensional primary source index array and one-dimensional candidate source index array The algorithm utilizes an offset array to perform a linear scan on all candidate pairs sequentially. This prefetches celestial properties used to calculate the logarithmic position-related likelihood value and physical characteristic consistency score for each candidate pair. These properties are then accumulated to generate a one-dimensional candidate source index array. Unnormalized logarithmic score arrays of equal length and contiguous memory .
[0064] Specifically, the implementation steps of step S4 are described as follows.
[0065] S401, according to the one-dimensional primary source index array and one-dimensional candidate source index array The order is determined, and the right ascension, declination, and observation error radius of the primary and candidate objects in each candidate pair are pre-selected using the offset array. The spherical angular distance and joint position error of each candidate pair are calculated to obtain the logarithmic position correlation likelihood value.
[0066] The specific operating steps for step S401 are as follows:
[0067] 1) Sequentially scan and read the one-dimensional main source index array The main star catalog celestial body index number (That is, the main star catalog celestial body number) According to the celestial body index number in the main star catalog In the offset array Internal read index number Corresponding candidate interval In order to identify celestial objects corresponding to multiple candidate star catalogs The first and last index numbers of the corresponding consecutive segments are stored in the one-dimensional candidate source index array;
[0068] 2) Index numbers of objects in the main star catalog Data corresponding to the same index number are pre-selected from the one-dimensional continuous right ascension array, one-dimensional continuous declination array, and one-dimensional continuous observation error radius array of the main star catalog objects (specifically, the main star catalog objects). Equatorial longitude Equatorial latitude and observation error radius );
[0069] 3) Based on the first and last index numbers of the candidate catalog objects, pre-retrieve data corresponding to the same first and last index numbers in batches from the one-dimensional continuous right ascension array, one-dimensional continuous declination array, and one-dimensional continuous observation error radius array of the candidate catalog objects (specifically for each candidate catalog object). Equatorial longitude Equatorial latitude and observation error radius );
[0070] 4) Calculate the spherical angular distance and joint position error between the main catalog object and its corresponding candidate catalog objects sequentially to obtain the logarithmic position correlation likelihood value for each candidate pair; specifically: [The text abruptly ends here, likely due to an incomplete sentence or missing information.] Equatorial longitude and equatorial latitude and celestial objects in each candidate star catalog Equatorial longitude and equatorial latitude Converting to Cartesian space vector representation and substituting into the spherical angular distance formula, the celestial bodies in the main star catalog are calculated. With each candidate star catalog celestial object spherical angular distance The main star represents celestial bodies. Observation error radius and celestial objects in each candidate star catalog Observation error radius Substitute joint position error The formula is used to calculate the celestial bodies in the main star catalog. With each candidate star catalog celestial object joint position error Its calculation expression is: Next, the spherical angular distance and joint position error of each candidate pair are input into a Gaussian kernel function to calculate the position correlation likelihood value. Then, logarithmic processing is performed to obtain the logarithmic position correlation likelihood value. The specific calculation method for the location-related likelihood value is implemented using existing well-known technologies, and will not be elaborated here.
[0071] In step S401, based on the columnar one-dimensional continuous celestial object attribute arrays of the main catalog objects and the candidate catalog objects constructed in step S3, the batch prefetching of equatorial longitude, equatorial latitude, and observation error radius can be performed by traversing and indexing the offset array in a linear manner. The candidate intervals of the offset array are used as "address guides" to further perform batch prefetching of specified data from the columnar one-dimensional continuous celestial object attribute arrays of the main catalog objects and the candidate catalog objects.
[0072] S402. Using the same method as step S401, the physical characteristics of the primary and candidate catalog objects in each candidate pair are pre-fetched sequentially using the offset array to calculate the consistency score of each candidate pair's physical characteristics. The consistency score of each candidate pair's physical characteristics is... The probability density ratio is calculated by feeding the same physical characteristics of objects in the main star catalog and the candidate star catalog into the probability density ratio model; this calculation method is implemented using existing well-known techniques, which will not be elaborated here.
[0073] The specific steps for step S402 are as follows:
[0074] 1) Based on the index number of the main star catalog objects determined in step S401 Data corresponding to the same index number is pre-selected from the one-dimensional continuous array of physical characteristics of celestial bodies in the main star catalog;
[0075] 2) Based on the first and last index numbers of the corresponding candidate star catalog objects determined in step S401, pre-retrieve data corresponding to the same first and last index numbers in batches from the one-dimensional physical feature continuous array of the candidate star catalog objects;
[0076] 3) Calculate the physical feature consistency score for each candidate pair in the order of prefetching in steps 1) and 2).
[0077] For cases involving multiple physical feature data, repeat steps 1) to 3) above to complete the consistency score calculation for each physical feature of each candidate pair.
[0078] S403. Sequentially read the logarithmic positional relevance likelihood value and physical feature consistency score of each candidate pair in step S401, and perform weighted summation to generate an unnormalized logarithmic score array that is the same length as the one-dimensional candidate source index array and is contiguous in memory.
[0079] The weighted cumulative calculation formula is as follows: In the formula, These are the weighting coefficients for the logarithmic positional relevance likelihood values. The weighting coefficients for the physical feature consistency score. For the first The physical feature consistency score is calculated for each physical feature. In actual calculations, the two weighting coefficients are taken as empirical values, for example... =0.9, =0.1.
[0080] After step S4, the newly generated unnormalized logarithm score array For the one-dimensional candidate source index array A new array of equal length and contiguous memory, because it uses a one-dimensional candidate source index array... Since the candidate star catalog objects are recorded in the same order, this unnormalized logarithmic score array is also applicable to batch prefetching the unnormalized logarithmic scores corresponding to the candidate star catalog objects, using the corresponding candidate intervals in the offset array as "address guides".
[0081] S5. Treat all candidate catalog objects corresponding to each main catalog object as a candidate group, and use the offset array in the unnormalized logarithmic score array. The unnormalized logarithm scores corresponding to candidate groups are pre-fetched in the inner batch, and the normalized probabilities of each candidate pair within the candidate group are calculated by performing an in-group log-sum-exp reduction operation. The total confidence level of the candidate group and its member .
[0082] Specifically, for the primary star's celestial bodies The normalized probability of each candidate pair within the corresponding candidate group and the total confidence level of the candidate groups The calculation steps are described below.
[0083] S501, using offset array The unnormalized logarithmic score of the candidate group is pre-fetched in batches.
[0084] Specifically, according to the celestial object index number of the main star catalog In the offset array Internal read index number Corresponding candidate interval Determine the first and last indices of the corresponding candidate catalog objects within the one-dimensional candidate source index array; based on the first and last indices of the candidate catalog objects, in the unnormalized logarithmic score array... The unnormalized logarithm score corresponding to the same first and last index numbers is prefetched in the medium batch.
[0085] S502, Find the maximum unnormalized logarithm score within the candidate group. This serves as an offset to prevent floating-point overflow during subsequent exponentiation.
[0086] S503, Calculate the aggregate likelihood logarithm : , This represents the score of the k-th unnormalized logarithm within the candidate group.
[0087] S504. Obtain the source density of the corresponding covered sky area in the primary star catalog. Source density of the sky area corresponding to the candidate star catalog The background source score is obtained by logarithmic addition of the two. : .
[0088] S505, Calculate the total logarithmic partition function of candidate groups. : .
[0089] S506. Calculate the normalized probability of each candidate pair within the candidate group. : and the total confidence level of the candidate group to which it belongs. : .
[0090] S6. Judge the result of step S5 according to the set threshold and output the candidate pairs with high credibility.
[0091] Specifically, the implementation steps of step S6 are as follows:
[0092] S601. Set a judgment threshold, which consists of the normalized probability threshold of the candidate pair and the total confidence threshold of the candidate group; in this embodiment, the normalized probability threshold of the candidate pair is set to 0.8, and the total confidence threshold of the candidate group is also set to 0.8.
[0093] S602. Judge each candidate pair. Taking this embodiment as an example, when the normalized probability of the candidate pair... ≥0.8, and the total confidence level of its candidate group If the value is ≥0.8, then the candidate relationship of the candidate pair is considered to have high credibility.
[0094] S603. For candidate pairs judged to have high credibility in step S602, output the candidate pairs with high credibility in a way that preserves the logical semantics of the candidate structure, so as to provide quantitative basis for subsequent scientific analysis of cross-validation in astronomy; wherein, the output format of the candidate pair is: Main Star Catalog Celestial Index ID, Candidate Star Catalog Celestial Index ID, , Among them, the primary star catalog object index ID and the candidate star catalog object index ID are obtained through a one-dimensional primary source index array. and one-dimensional candidate source index array The position index of celestial bodies in the original star catalog is obtained by recording them in the catalog.
[0095] To further demonstrate the superiority of the memory-optimized astronomical cross-verification confidence calculation method of the present invention, this method was put into practical application and compared with existing object-based confidence calculation methods. Existing object-based confidence calculation methods specifically use the NWAY method (see: Salvato M, Buchner J, Budavári T, et al. Finding counterparts for all-sky X-ray surveys with NWAY: ABayesian algorithm for cross-matching multiple catalogues [J]. Monthly Notices of the Royal Astronomical Society, 2018, 473: 4937–4955). This method typically organizes candidate result data in an object-based AoS (Array of Structures) format, that is, encapsulating the right ascension, declination, observation error radius, spherical angular distance, and related physical characteristic fields of a single candidate pair into an object record, and then storing and accessing it record by record using DataFrame, tuple, or object array. While this type of representation is convenient for program organization and semantic expression, the different fields are interspersed in memory, and the overhead of copying object metadata, pointers and intermediate results can easily lead to discontinuous memory access and poor cache locality, thus limiting the execution efficiency of large-scale confidence calculation.
[0096] Comparative experiments were conducted on an L20 server (host configuration: Intel Xeon Gold 6530 CPU, CPU architecture: dual-processor (64 cores / 128 threads)), and all experiments used the same multiplexing offset operation on the maxi data, compared with WISE data (total... A confidence level calculation experiment was conducted on the cross-validation results of (a number of candidate pairs) to ensure that the comparative experiment was conducted under a consistent experimental environment. The specific comparative experiment results are shown in Table 1 below.
[0097] Table 1:
[0098] This invention Calculations are performed based on the layout of the index array, offset array, and columnar one-dimensional continuous celestial body attribute array. 341.3 s 265.8 s 7.9 GB 24.53% NWAY Calculation based on AoS data layout 5901.2 s 5502.1 s 11.3 GB 37.95%
[0099] As can be seen from the comparative test results shown in Table 1,
[0100] (1) In terms of improving memory access efficiency and computational execution efficiency, based on steps S1-S2 of the method of this invention, a main star table index array, a candidate star table index array, and an offset array are constructed, so that the cross-validation results originally organized according to candidate groups are reconstructed into a continuous structured linear representation; and in further steps S4-S5, the discrete traversal process based on candidate groups in the traditional method is reorganized into a rule execution mode that combines linear scanning at the candidate pair level and segmented reduction at the candidate group level. Compared with implementation methods such as NWAY that maintain candidate relationships using object pointers, DataFrames, or nested containers, it can effectively avoid frequent address jumps, repeated boundary judgments, and cache misses caused by irregular memory access, enabling the CPU to access memory in a streaming manner and make fuller use of hardware prefetching and multi-level caching, thereby significantly improving memory access locality and execution efficiency; according to the comparative experimental results, the operation time of the method of this invention is shortened by about 17 times, the confidence calculation time is shortened by about 20 times, and the cache miss rate is reduced by more than 13%;
[0101] (2) In terms of reducing data structure redundancy and peak memory overhead, existing object-based Bayesian authentication frameworks usually use AoS or object-oriented containers to maintain candidate pairs and candidate group relationships. This not only requires additional storage of object metadata, index relationships and container management information, but also easily introduces non-computational memory overhead due to object alignment, dynamic expansion and intermediate copies. In contrast, step S3 of the method of this invention adopts a SoA (Structure of Array) layout with columnar field organization, which stores core computational fields such as location features and physical features in a compact one-dimensional array and uses index arrays and offset arrays to express logical relationships, thereby decoupling "computational fields" from "logical organization" and eliminating redundant space occupation caused by object hierarchy. According to the comparative experimental results, the peak memory usage is reduced by about 30% compared with NWAY.
[0102] Furthermore, it is particularly noteworthy that in the comparative experiments described above, the method of this invention maintained a decision consistency rate of 99.9% with the NWAY method. This demonstrates that, without sacrificing the accuracy of scientific computing, the method of this invention effectively solves the problems of inefficient memory access and memory redundancy in astronomical big data computing by modifying the AoS data layout to meet the requirements of high-performance computing and combining it with an offset index array. It also has the advantages of better utilizing cache prefetching performance, reducing data structure redundancy caused by complex nested objects, and improving memory access and computing efficiency under the pressure of large-scale candidate objects.
Claims
1. A memory-optimized method for calculating the confidence level of astronomical cross-validation, characterized in that, The steps are as follows: S1. Obtain the astronomical cross-verification results of the main star catalog and the candidate star catalog, and flatten them into: a one-dimensional main source index array that sequentially records the position index of the main star catalog objects in the original star catalog, and a one-dimensional candidate source index array that synchronously and sequentially records the position index of the candidate objects in the candidate star catalog that form candidate pairs with each main star catalog object in the original star catalog. S2. Construct an offset array to record the candidate interval of each primary catalog object in the one-dimensional primary source index array corresponding to multiple candidate catalog objects in the one-dimensional candidate source index array. S3. Construct multiple columnar one-dimensional continuous celestial body attribute arrays for the main star catalog objects, which record the positional and physical characteristics of the main star catalog objects according to the recording order of the main star catalog objects in the one-dimensional main source index array. Construct multiple columnar one-dimensional continuous celestial body attribute arrays for candidate catalog objects, which record the positional and physical characteristics of the candidate catalog objects according to the recording order of the candidate catalog objects in the one-dimensional candidate source index array; S4. Based on the one-dimensional primary source index array and the one-dimensional candidate source index array, and using the offset array, perform a linear scan on all candidate pairs in sequence to prefetch the celestial properties used to calculate the logarithmic position correlation likelihood value and physical feature consistency score of each candidate pair, and generate an unnormalized logarithmic score array after accumulation. S5. Treat all candidate catalog objects corresponding to each main catalog object as a candidate group, use the offset array to pre-fetch multiple unnormalized logarithm scores corresponding to the candidate group in batches, and perform log-sum-exp reduction operation within the group to calculate the normalized probability of each candidate pair in the candidate group and the total confidence of the candidate group. S6. Judge the result of step S5 according to the set threshold and output the candidate pairs with high credibility.
2. The memory-optimized astronomical cross-validation confidence calculation method according to claim 1, characterized in that, The specific operation of step S2 is as follows: S201. Based on the total number of celestial bodies in the main star catalog. The construction length is offset array and set the offset array sequentially. index number 0~ ; S202, in the offset array The internal sequence records the candidate intervals of each primary catalog object in the one-dimensional primary source index array and multiple candidate catalog objects in the one-dimensional candidate source index array. array elements The corresponding storage in the one-dimensional primary source index array is the first A primary star catalog object in a one-dimensional candidate source index array The first candidate star index astronomical number, array element Corresponding storage one-dimensional primary source index array The Middle A primary star catalog object in a one-dimensional candidate source index array The index number of the first candidate star catalog object in the catalog; S203. Set the sentinel value, which is the total number of all candidate pairs in the astronomical cross-verification results of the main star catalog and the candidate star catalog, and store it in the offset array. The last array middle.
3. The memory-optimized astronomical cross-validation confidence calculation method according to claim 1, characterized in that, In step S3, the positional features include right ascension, declination, and observation error radius, and the physical features are selected from at least one of magnitude, flow rate, and color.
4. The memory-optimized astronomical cross-validation confidence calculation method according to claim 3, characterized in that, The specific steps for step S4 are as follows: S401. Using the offset array, the right ascension, declination, and observation error radius of the primary and candidate star catalog objects in each candidate pair are pre-selected sequentially. The spherical angular distance and joint position error of each candidate pair are calculated to obtain the logarithmic position correlation likelihood value. S402. Using the offset array, pre-extract the physical characteristics of the main catalog objects and candidate catalog objects in each candidate pair in turn to calculate the physical characteristic consistency score of each candidate pair. S403. Sequentially read the logarithmic positional relevance likelihood value and physical feature consistency score of each candidate pair in step S401, and perform weighted summation to generate an unnormalized logarithmic score array that is the same length as the one-dimensional candidate source index array and is contiguous in memory.
5. The memory-optimized astronomical cross-validation confidence calculation method according to claim 4, characterized in that, In step S401, the right ascension, declination, and observation error radius of the primary and candidate catalog objects in each candidate pair are pre-selected sequentially. The method for calculating the logarithmic position correlation likelihood value of each candidate pair is as follows: 1) Sequentially read the index numbers of the main star catalog objects in the one-dimensional main source index array, and read the candidate intervals corresponding to the index numbers in the offset array according to the index numbers of the main star catalog objects, so as to determine the first and last index numbers of the corresponding multiple candidate star catalog objects in the one-dimensional candidate source index array; 2) Based on the index number of the main star catalog objects, pre-select the data corresponding to the same index number from the one-dimensional continuous right ascension array, one-dimensional continuous declination array, and one-dimensional continuous observation error radius array of the main star catalog objects respectively; 3) Based on the first and last index numbers of the corresponding candidate star catalog objects, pre-retrieve the data corresponding to the same first and last index numbers in batches from the one-dimensional continuous right ascension array, one-dimensional continuous declination array, and one-dimensional continuous observation error radius array of the candidate star catalog objects; 4) Following the prefetching order of steps 2) and 3), calculate the spherical angular distance and joint position error between the main catalog object and its corresponding candidate catalog objects in turn, so as to obtain the logarithmic position correlation likelihood value of each candidate pair.
6. The memory-optimized astronomical cross-validation confidence calculation method according to claim 4, characterized in that, In step S402, the method for pre-taking various physical characteristics of the primary and candidate celestial bodies in each candidate pair and calculating the physical characteristic consistency score of each candidate pair is as follows: 1) Based on the index number of the main star catalog celestial body determined in step S401, pre-retrieve the data corresponding to the same index number from the continuous array of one-dimensional physical characteristics of the main star catalog celestial bodies; 2) Based on the first and last index numbers of the corresponding candidate star catalog objects determined in step S401, pre-retrieve data corresponding to the same first and last index numbers in batches from the continuous array of one-dimensional physical characteristics of the candidate star catalog objects; 3) Calculate the physical feature consistency score for each candidate pair in the order of prefetching in steps 1) and 2).
7. The memory-optimized astronomical cross-validation confidence calculation method according to claim 1, characterized in that, For any primary star catalog object, the steps for calculating the normalized probability of each candidate pair within its corresponding candidate group and the total confidence score of the candidate group are as follows: S501, using offset array Unnormalized log score of batch prefetched candidate groups S502, Use the maximum log score within the candidate group as the offset; S503, Calculate the aggregate likelihood logarithm : , The score of the kth unnormalized logarithm within the candidate group; S504. Obtain the source density of the corresponding covered sky area for the main star catalog and candidate star catalog. and The background source score is obtained by logarithmic addition of the two. : ; S505, Calculate the total logarithmic partition function of candidate groups. : ; S506. Calculate the normalized probability of each candidate pair within the candidate group. : and the total confidence level of the candidate group to which it belongs. : .
8. The memory-optimized astronomical cross-validation confidence calculation method according to claim 7, characterized in that, The specific steps of step S501 are as follows: Based on the celestial body index number in the main star catalog... Read the candidate interval corresponding to the index number in the offset array, and determine the first and last index numbers of the corresponding candidate star catalog celestial bodies in the one-dimensional candidate source index array for continuous storage of fragments; Based on the first and last index numbers of the candidate star catalog objects, the unnormalized logarithmic scores corresponding to the same first and last index numbers are pre-retrieved in batches from the unnormalized logarithmic score array.