Method and device for mining outliers of astronomical star catalog, equipment and medium

Abstract

The present disclosure provides a method for mining outliers of an astronomical catalog, comprising: preprocessing astronomical catalog data; performing MC sampling with replacement and KNN algorithm outlier detection on the preprocessed astronomical catalog data to obtain KNN outlier scores of the sampled astronomical catalog data; repeatedly performing step S2 until R / S convergence is achieved to obtain KNN outlier scores of the astronomical catalog data, and then calculating outlier indexes R / S of all data in the astronomical catalog data according to the KNN outlier scores of the astronomical catalog data, regarding data points exceeding an outlier threshold as outliers to obtain outliers in the astronomical catalog data. By combining the advantages of MC sampling, KD tree and KNN algorithm, the present disclosure can effectively mine outliers from large-scale astronomical catalog data, and effectively mine outliers in large-scale catalogs. The present disclosure also provides a device for mining outliers of an astronomical catalog, an electronic device and a computer readable medium.

Description

Technical Field

[0001] This disclosure relates to the field of data processing technology, and in particular to a method, apparatus, device and medium for mining outlier data from astronomical star catalogs. Background Technology

[0002] With the development of astronomical observation technology, people have acquired a large amount of astronomical data in a short period of time. Most of this data follows certain known patterns or laws, but there are often some data points that are different from the norm; these are called outliers. Outlier detection is of crucial significance in astronomy because it is precisely these outliers that often lead to new scientific discoveries or challenges to existing theories.

[0003] Thanks to the continuous development of outlier detection methods, researchers have also conducted some studies and applications in astronomy. Early outlier detection methods were mainly based on statistical principles, assuming that the data follows a certain distribution, and then identifying outlier data points that do not conform to that distribution. Methods such as the interquartile range (IQR) method and the Z-score method are commonly used to find outliers in data. Colleagues such as Feigelson et al. and Wall JV et al. have discussed the application of some commonly used statistical distributions, such as the normal distribution, Poisson distribution, and power-law distribution, in astronomical data analysis.

[0004] With the development of machine learning technology, outlier detection methods based on this technology have been widely used by astronomers. For example, algorithms such as Local Outlier Factor, Isolated Forest, and Support Vector Machine have been proven to perform exceptionally well in high-dimensional data and complex data structures. Baron et al. used the unsupervised Random Forest (RF) algorithm to detect 400 different exotic galaxies with extreme emission-to-line ratios, unusually strong absorption lines, complex velocity structures, and supernova explosions from more than 2.1 million galaxy spectra from The Sloan Digital Sky Survey (SDSS). Shakurova et al. used the Local Outlier Factor (LOF) algorithm based on local density to discover some spectra with anomalous value variations in LAMOST (Large Sky Area Multi-Object Fiber Spectroscopic Telescope) data. Some of these anomalies may be caused by instrument errors, while others may correspond to anomalous celestial objects. Nun et al. proposed an expert ensemble anomaly detection method and applied it to the detection of anomalous celestial objects in astronomical light curves. This method combines five different anomaly detection algorithms, optimizing gating parameters to allow each algorithm to function effectively in its preferred feature subspace, and finally integrating the results of all algorithms for a final prediction. This method was applied to the Massive Compact Halo Objects (MACHO) light curve dataset, discovering some rare celestial objects.

[0005] It is worth noting that astronomical data often has high dimensionality and may contain many different types of observations (such as spectra, light curves, images, etc.). Therefore, outlier detection for such data often requires more complex and sophisticated methods. Deep learning has shown great potential in this regard. Margalef-Bentabol et al. proposed using a Wasserstein Generative Adversarial Network (GAN) model to detect outliers in astronomical images. They first trained the GAN on normal images, then calculated anomaly scores for test images to determine whether an image was abnormal. This method learns complex features directly from pixel space and can detect subtle morphological differences that traditional methods easily miss. These methods can learn the intrinsic structure of data and effectively identify data points that do not conform to the main data. A team proposed an active deep learning-based method that uses convolutional neural networks and combines class balancing and active learning techniques. Through iterative training and manual annotation of partial spectra, the model's performance is gradually improved, leading to the discovery of numerous new emission line star candidates in LAMOST survey data. These methods can learn the intrinsic structure of data and effectively identify data that does not conform to the main data set.

[0006] While the importance of outlier data is widely recognized, several challenges and limitations remain in astronomy. Traditional methods, often based on thresholds or statistical tests, identify outliers, but these are sensitive to assumptions about data distribution and may overlook important information. Furthermore, the processing of high-dimensional data and the data heterogeneity common in astronomy further complicate outlier mining. Summary of the Invention

[0007] (a) Technical problems to be solved

[0008] To address the existing technical problems, this disclosure proposes a method, apparatus, equipment, and medium for mining outlier data in astronomical star catalogs, thereby enabling effective mining of outlier data in large-scale star catalogs.

[0009] (II) Technical Solution

[0010] To achieve the above objectives, this disclosure provides a method for mining outlier data from astronomical star catalogs, including:

[0011] Step S1: Preprocess the astronomical catalog data;

[0012] Step S2: Perform MC sampling with replacement and KNN algorithm outlier detection on the preprocessed astronomical catalog data to obtain the KNN outlier score of the sampled astronomical catalog data;

[0013] Step S3: Repeat step S2 until the outlier index R / S converges to obtain the KNN outlier score of the astronomical catalog data. Then, based on the KNN outlier score of the astronomical catalog data, calculate the outlier index R / S of all data in the astronomical catalog data. Data points that exceed the outlier threshold are considered outliers to obtain the outlier values ​​in the astronomical catalog data.

[0014] In the above scheme, the preprocessing of the astronomical catalog data in step S1 includes: selecting relevant feature dimensions F from the astronomical catalog dataset D according to the characteristics of the astronomical catalog data; performing data normalization or data standardization on the selected feature dimensions F to ensure that each dimension is at a similar scale, thereby obtaining a subset D′ of the astronomical catalog data with a size of N; and defining two vectors R and S with a size of N.

[0015] In the above scheme, the feature dimension F includes at least one of redshift, effective temperature, metallicity, magnitude, and line-of-sight velocity.

[0016] In the above scheme, the step of performing data normalization or data standardization on the selected feature dimension F,

[0017] The data normalization process involves scaling the data in the feature dimension F to a specified range, such as [0, 1], using the following formula:

[0018]

[0019] Where min(F) is the minimum value in the sample, and max(F) is the maximum value in the sample;

[0020] The data standardization process transforms the data in feature dimension F into zero mean and unit variance using the following formula:

[0021]

[0022] Where μ is the sample mean and σ is the sample standard deviation.

[0023] In the above scheme, step S2, which involves performing MC sampling with replacement and KNN algorithm outlier detection on the preprocessed astronomical catalog data to obtain the KNN outlier score of the sampled astronomical catalog data, includes:

[0024] The preprocessed astronomical catalog data is sampled with replacement using MC sampling to obtain a dataset T of size n (n is much smaller than N);

[0025] For each data point in dataset T, recursively select a dimension and find the median on that dimension as the split point to construct the KD tree of dataset T;

[0026] The constructed KD tree is used to find the K (K less than n) nearest neighbor data points of each data point, and the average distance between each data point and its K nearest neighbor data points is calculated. The dataset T is then sorted according to this average distance to obtain dataset T′.

[0027] The sorted index values ​​of dataset T′ are accumulated into vector R, and n is added to the corresponding position in vector S to obtain the KNN outlier scores of the astronomical star catalog data.

[0028] In the above scheme, the step of performing MC sampling with replacement on the preprocessed astronomical catalog data includes:

[0029] Determine the sample size n;

[0030] From a subset D′ of astronomical star catalog data, n data points are randomly selected with replacement to obtain a dataset T of size n (n is much smaller than N).

[0031] In the above scheme, step S3 involves repeating step S2 until R / S converges to obtain the KNN outlier score of the astronomical catalog data. Then, based on the KNN outlier score, the outlier index R / S for all data in the astronomical catalog data is calculated. Data points exceeding the outlier threshold are considered outliers, thus obtaining the outlier values ​​in the astronomical catalog data, including:

[0032] Repeat step S2 until R / S converges to obtain the final vectors R and S. Calculate the outlier index R / S for all data in the astronomical star catalog dataset D.

[0033] By determining an outlier threshold, data points exceeding the outlier threshold are considered outliers, thus obtaining outlier values ​​in the astronomical catalog data.

[0034] Another aspect of this disclosure provides an apparatus for mining outlier data from astronomical catalogues, comprising:

[0035] The data preprocessing module is used to preprocess astronomical catalog data;

[0036] The data detection module is used to perform MC sampling with replacement and KNN algorithm outlier detection on the preprocessed astronomical catalog data to obtain the KNN outlier score of the sampled astronomical catalog data. This process is repeated until the outlier index R / S converges to obtain the KNN outlier score of the astronomical catalog data.

[0037] The data mining module is used to calculate the outlier index R / S of all data in the astronomical catalog data based on the KNN outlier scores of the astronomical catalog data, and to regard data points that exceed the outlier threshold as outliers, thereby obtaining the outlier values ​​in the astronomical catalog data.

[0038] This disclosure also provides an electronic device, including: a processor; and a memory storing a computer-executable program that, when executed by the processor, causes the processor to perform the method for mining outlier data from astronomical catalogs.

[0039] This disclosure also provides a computer-readable medium containing computer-executable instructions that, when executed, implement the method for mining outlier data from the astronomical catalogue.

[0040] (III) Beneficial Effects

[0041] As can be seen from the above technical solutions, the method, apparatus, equipment, and medium for mining outlier data from astronomical star catalogs provided in this disclosure have the following beneficial effects:

[0042] 1. The outlier mining method for astronomical catalogs provided in this disclosure combines the advantages of MC sampling, KD trees, and the KNN algorithm. MC sampling reduces data complexity through random sampling, KD trees can efficiently perform nearest neighbor search and range query for multidimensional spatial data, and the KNN algorithm identifies outlier data points through nearest neighbor distance. The organic combination of these three methods can effectively mine outliers from large-scale astronomical catalog data, including rare and anomalous celestial objects. This is of great significance for discovering new celestial objects and verifying astronomical theories, and achieves effective mining of outlier data in large-scale catalogs.

[0043] 2. The outlier data mining method for astronomical star catalogs provided in this disclosure introduces the MCKD tree method, which can be applied to any dimension and is not limited by the data dimension, thus alleviating the challenges brought about by the high-dimensionality of astronomical star catalog data.

[0044] 3. The outlier data mining method for astronomical star catalogs provided in this disclosure uses MC sampling, KD tree and KNN algorithms, which are simple to calculate and easy to implement, and can be effectively applied to different astronomical star catalog data, such as SDSS, LAMOST and other survey data.

[0045] 4. The outlier data mining method of the astronomical star catalog provided in this publication can discover some unusual celestial object types, such as special variable stars, binary star systems, and active galactic nuclei, providing new samples for astronomical research.

[0046] 5. The outlier data mining method for astronomical star catalogs provided in this disclosure uses MC sampling, KD tree and KNN algorithms, all of which support parallel computing and can process large amounts of data in parallel, greatly reducing the workload of manual screening and improving work efficiency.

[0047] 6. The method for mining outlier data in astronomical star catalogs provided in this disclosure can provide ideas for other data mining tasks, such as abnormal transaction detection and network intrusion detection, and has certain promotional value.

[0048] 7. The method for mining outlier data in astronomical star catalogs provided in this disclosure employs a highly modular algorithm, and each module can be flexibly adjusted to adapt to the needs of different application scenarios. Attached Figure Description

[0049] To gain a more complete understanding of this disclosure and its advantages, reference will now be made to the following description taken in conjunction with the accompanying drawings, wherein:

[0050] Figure 1 A flowchart illustrating a method for mining outlier data from astronomical star catalogs according to an embodiment of the present disclosure is shown.

[0051] Figure 2The illustration shows a schematic diagram of a method for mining outlier data from astronomical star catalogs according to an embodiment of the present disclosure.

[0052] Figure 3 A block diagram of an apparatus for mining outlier data from astronomical star catalogs according to an embodiment of the present disclosure is shown schematically.

[0053] Figure 4 A block diagram of an electronic device for mining outlier data from astronomical star catalogs according to an embodiment of the present disclosure is shown schematically.

[0054] [Figure Labels]:

[0055] S1, S2, S3: Step 300: A device for mining outlier data from astronomical catalogs.

[0056] 301: Data Preprocessing Module

[0057] 302: Data Detection Module

[0058] 303: Data Mining Module

[0059] 400: Electronic Equipment

[0060] 410: Processor

[0061] 420: Memory

[0062] 421: Computer Programs Detailed Implementation

[0063] The embodiments of the present disclosure will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the disclosure. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of the present disclosure for ease of explanation. However, it will be apparent that one or more embodiments may be practiced without these specific details. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concepts of the present disclosure.

[0064] In the process of mining outlier data in large-scale star catalogs, in addition to the challenges faced by existing methods, the inventors of this application have also found that there is currently relatively little analysis and research on outlier data in astronomical star catalogs by astronomers.

[0065] Astronomical catalogues are an important data storage medium in astronomy, containing a wealth of information about celestial bodies, such as their position, brightness, apparent diameter, mass, distance, spectral characteristics, and trajectories. These catalogues provide invaluable data resources for astronomical research, assisting astronomers in important tasks such as celestial body classification, astronomical observation, and analysis of the structure and evolution of the universe.

[0066] This application introduces the Monte Carlo (MC) sampling method, the K-dimensional Tree (KDTree) method, and the K-Nearest Neighbors (KNN) algorithm, and combines the three to propose the MKNN algorithm, which is then used to mine outlier data in astronomical catalogs.

[0067] Motion-Codd (MC) sampling is a statistical method that estimates the answer to complex problems through random simulation. Based on the principle of random sampling, it simulates possible outcomes multiple times and then takes the average or probability to approximate the solution. In science and engineering, MC sampling is widely used in simulation experiments, risk assessment, financial modeling, and high-dimensional data analysis. By introducing randomness, MC sampling can handle complex problems and provide approximate numerical solutions, which is particularly valuable when traditional methods are difficult to apply.

[0068] K-Nearest Neighbors (KNN) is an instance-based learning algorithm used for classification and regression tasks. It works by finding the K nearest neighbors in the existing dataset when a new data point needs to be classified or predicted. The class or value of these nearest neighbors is then used to determine the new data point's class or predicted value. For example, in classification tasks, a new data point is typically assigned to the most common class among its K neighbors. Due to its simplicity and intuitiveness, KNN is widely used in various practical applications.

[0069] McLeod sampling allows for the random selection of samples from a dataset, thereby reducing the complexity of high-dimensional data and addressing data heterogeneity. The KNN algorithm, on the other hand, utilizes the distance relationships between data points to identify outliers; its advantage lies in its ability to locally capture the characteristics of the dataset.

[0070] To effectively mine outlier data in large-scale star catalogs, this disclosure proposes a method for mining outlier data from astronomical star catalogs. For example... Figure 1 As shown, Figure 1 A flowchart illustrating a method for mining outlier data from astronomical catalogs according to an embodiment of the present disclosure is shown, the method comprising:

[0071] Step S1: Preprocess the astronomical catalog data, specifically including:

[0072] Based on the characteristics of the astronomical catalog data, relevant feature dimensions F are selected from the astronomical catalog dataset D, such as redshift, effective temperature, metallicity, magnitude, radial velocity, etc.

[0073] The selected feature dimension F is normalized or standardized to ensure that each dimension is on a similar scale, resulting in a subset D′ of astronomical star catalog data of size N.

[0074] Define two vectors R and S, with size N.

[0075] In this step, the data normalization or standardization processing of the selected feature dimension F includes:

[0076] Data normalization is the process of scaling the data in the feature dimension F to a specified range, such as [0, 1], using the following formula:

[0077]

[0078] Where min(F) is the minimum value in the sample, and max(F) is the maximum value in the sample.

[0079] Data standardization transforms the data in feature dimension F into zero mean and unit variance, typically using the following formula:

[0080]

[0081] Where μ is the sample mean and σ is the sample standard deviation.

[0082] Step S2: Perform MC sampling with replacement and KNN algorithm outlier detection on the preprocessed astronomical catalog data to obtain the KNN outlier score of the sampled astronomical catalog data, specifically including:

[0083] The preprocessed astronomical catalog data is sampled with replacement using MC sampling to obtain a dataset T of size n (n is much smaller than N);

[0084] For each data point in dataset T, recursively select a dimension and find the median on that dimension as the split point to construct the KD tree of dataset T;

[0085] The constructed KD tree is used to find the K (K less than n) nearest neighbor data points of each data point, and the average distance between each data point and its K nearest neighbor data points is calculated. The dataset T is then sorted according to this average distance to obtain dataset T′.

[0086] The sorted index values ​​of dataset T′ are accumulated into vector R, and n is added to the corresponding position in vector S to obtain the KNN outlier scores of the astronomical star catalog data.

[0087] In this step, the MC sampling with replacement of the preprocessed astronomical catalog data includes:

[0088] Determine the sample size n;

[0089] From a subset D′ of astronomical star catalog data, n data points are randomly selected with replacement to obtain a dataset T of size n (n is much smaller than N).

[0090] Step S3: Repeat step S2 until the outlier index R / S converges to obtain the KNN outlier score of the astronomical catalog data. Then, based on the KNN outlier score of the astronomical catalog data, calculate the outlier index R / S of all data in the astronomical catalog data. Data points that exceed the outlier threshold are considered outliers to obtain the outlier values ​​in the astronomical catalog data.

[0091] Repeat step S2 until R / S converges to obtain the final vectors R and S. Calculate the outlier index R / S for all data in the astronomical star catalog dataset D.

[0092] By determining an outlier threshold, data points exceeding the outlier threshold are considered outliers, thus obtaining outlier values ​​in the astronomical catalog data.

[0093] Therefore, the method for mining outlier data in astronomical catalogs provided in this disclosure combines the advantages of MC sampling, KD trees, and KNN algorithms, and can effectively mine outliers from large-scale astronomical catalog data, including rare and anomalous celestial objects. This is of great significance for discovering new celestial objects and verifying astronomical theories, and realizes the effective mining of outlier data in large-scale catalogs.

[0094] based on Figure 1 The flowchart shown is a method for mining outlier data from astronomical star catalogs according to an embodiment of this disclosure. Figure 2 The diagram shows a method for mining outlier data from astronomical star catalogs according to an embodiment of this disclosure.

[0095] like Figure 2 As shown in the embodiments of this disclosure, the method for mining outlier data in astronomical star catalogs mainly includes the following steps:

[0096] Step 1: Normalize or standardize the original astronomical catalog to obtain a dataset D of size N;

[0097] Step 2: Perform MC sampling with replacement on the preprocessed dataset D. Here, we assume that 5 data items are sampled each time, and the nearest neighbor is set to 3, that is, n is 5 and k is 3. The first 5 data items are assumed to be 72, 208, 334, 536 and 932. The average distance between the three nearest neighbors of the 5 data items is obtained by using KD tree and KNN algorithm. Sort them according to their average distance between the three nearest neighbors, and assume that the rankings are 1, 2, 3, 4 and 5 respectively. Add these ranking index values ​​to R, and add 1 to the corresponding position in S.

[0098] Step 3: Execute S2 until the outlier index R / S converges to obtain the KNN outlier score of the astronomical catalog data. Then, based on the KNN outlier score of the astronomical catalog data, calculate the outlier index R / S of all data in the astronomical catalog data. Data points that exceed the outlier threshold are considered outliers to obtain the outlier values ​​in the astronomical catalog data.

[0099] Therefore, the method for mining outlier data in astronomical catalogs provided in this disclosure combines the advantages of MC sampling, KD trees, and KNN algorithms, and can effectively mine outliers from large-scale astronomical catalog data, including rare and anomalous celestial objects. This is of great significance for discovering new celestial objects and verifying astronomical theories, and realizes the effective mining of outlier data in large-scale catalogs.

[0100] based on Figure 1 The flowchart shown is for a method of mining outlier data from astronomical star catalogs according to an embodiment of this disclosure. Figure 2 The diagram shown is a schematic representation of a method for mining outlier data from astronomical star catalogs according to an embodiment of this disclosure. Figure 3 A block diagram of an apparatus for mining outlier data from astronomical star catalogs according to an embodiment of the present disclosure is shown schematically.

[0101] like Figure 3 As shown in the embodiment of this disclosure, the astronomical catalog outlier data mining apparatus 300 includes a data preprocessing module 301, a data detection module 302, and a data mining module 303. The data preprocessing module 301 preprocesses the astronomical catalog data. The data detection module 302 performs outlier detection using MC sampling with replacement and the KNN algorithm on the preprocessed astronomical catalog data, obtaining the KNN outlier score of the sampled astronomical catalog data. This process is repeated until the outlier index R / S converges, yielding the KNN outlier score of the astronomical catalog data. The data mining module 303 calculates the outlier index R / S of all data in the astronomical catalog data based on the KNN outlier score, and considers data points exceeding the outlier threshold as outliers, thus obtaining the outlier values ​​in the astronomical catalog data.

[0102] It should be understood that the data preprocessing module 301, data detection module 302, and data mining module 303 can be implemented in a single module, or any one of these modules can be split into multiple modules. Alternatively, at least some of the functions of one or more of these modules can be combined with at least some of the functions of other modules and implemented in a single module. According to embodiments of this disclosure, at least one of the data preprocessing module 301, data detection module 302, and data mining module 303 can be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or can be implemented in hardware or firmware in any other reasonable manner of integrating or packaging circuitry, or in a suitable combination of software, hardware, and firmware implementations. Alternatively, at least one of the data preprocessing module 301, data detection module 302, and data mining module 303 can be at least partially implemented as a computer program module, which, when run by a computer, can execute the functions of the corresponding module.

[0103] This disclosure also provides an electronic device for mining outlier data from astronomical star catalogs, such as... Figure 4 As shown, Figure 4 A block diagram of an electronic device for mining outlier data from astronomical catalogs according to an embodiment of the present disclosure is shown schematically. The electronic device 400 includes: one or more processors 410; and a memory 420 storing a computer-executable program that, when executed by the processors 410, causes the processors 410 to perform… Figure 1 The method for mining outlier data in the astronomical star catalog shown.

[0104] Specifically, processor 410 may include, for example, a general-purpose microprocessor, an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. Processor 410 may also include onboard memory for caching purposes. Processor 410 may be a single processing unit or multiple processing units for performing different actions of the method flow according to embodiments of this disclosure.

[0105] Memory 420 may be any medium capable of containing, storing, transmitting, propagating, or transmitting instructions. For example, readable storage media may include, but are not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, apparatuses, or propagation media. Specific examples of readable storage media include: magnetic storage devices such as magnetic tape or hard disk drives (HDDs); optical storage devices such as optical discs (CD-ROMs); memories such as random access memory (RAM) or flash memory; and / or wired / wireless communication links.

[0106] The memory 420 may include a computer program 421, which may include code / computer-executable instructions that, when executed by the processor 410, cause the processor 410 to perform a method or any variation thereof according to an embodiment of this disclosure.

[0107] Computer program 421 may be configured to have computer program code, for example, including computer program modules. For example, in an exemplary embodiment, the code in computer program 421 may include at least one program module, such as module 421A, module 421B, ... It should be noted that the division and number of modules are not fixed. Those skilled in the art can use appropriate program modules or combinations of program modules according to the actual situation. When these combinations of program modules are executed by processor 410, processor 410 can perform the method according to the embodiments of this disclosure or any variation thereof.

[0108] This disclosure also provides a computer-readable medium containing computer-executable instructions. This computer-readable medium may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable medium carries one or more programs, which, when executed, implement the method according to the embodiments of this disclosure or any variation thereof.

[0109] According to embodiments of this disclosure, a computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium may be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wired, optical fiber, radio frequency signals, etc., or any suitable combination thereof.

[0110] This disclosure has now been described in detail with reference to the accompanying drawings. Based on the above description, those skilled in the art should have a clear understanding of this disclosure.

[0111] It should be noted that implementations not shown or described in the accompanying drawings or the main text of the specification are all forms known to those skilled in the art and are not described in detail. Furthermore, the definitions of the elements above are not limited to the various specific structures, shapes, or methods mentioned in the embodiments, and those skilled in the art can easily modify or substitute them.

[0112] Although this disclosure has been shown and described with reference to specific exemplary embodiments thereof, various changes in form and detail may be made to this disclosure without departing from the spirit and scope of the disclosure as defined by the appended claims and their equivalents. Therefore, the scope of this disclosure should not be limited to the above-described embodiments, but should be determined not only by the appended claims but also by their equivalents.

Claims

1. A method for mining outlier data from astronomical catalogs, characterized in that, include: Step S1: Preprocess the astronomical catalog data; Step S2: Perform MC sampling with replacement and KNN algorithm outlier detection on the preprocessed astronomical catalog data to obtain the KNN outlier score of the sampled astronomical catalog data; Step S3: Repeat step S2 until the outlier index R / S converges to obtain the KNN outlier score of the astronomical catalog data. Then, based on the KNN outlier score of the astronomical catalog data, calculate the outlier index R / S of all data in the astronomical catalog data. Data points that exceed the outlier threshold are considered outliers to obtain the outlier values ​​in the astronomical catalog data. In step S2, the process of performing MC sampling with replacement and KNN outlier detection on the preprocessed astronomical catalog data to obtain the KNN outlier score of the sampled astronomical catalog data includes: Define two vectors R and S, with size N; The preprocessed astronomical catalog data is sampled with replacement by MC sampling to obtain a dataset T of size n, where n is much smaller than N; For each data point in dataset T, recursively select a dimension and find the median on that dimension as the split point to construct the KD tree of dataset T; The constructed KD-tree is used to find the K nearest neighbor data points of each data point, and the average distance between each data point and its K nearest neighbor data points is calculated. The dataset T is then sorted according to this average distance to obtain dataset T', where K is less than n. The sorted index values ​​of dataset T' are accumulated into vector R, and n is added to the corresponding positions in vector S to obtain the KNN outlier scores of the astronomical catalog data.

2. The method for mining outlier data from astronomical catalogs according to claim 1, characterized in that, The preprocessing of the astronomical catalog data described in step S1 includes: Based on the characteristics of the astronomical star catalog data, relevant feature dimensions F are selected from the astronomical star catalog dataset D; The selected feature dimension F is normalized or standardized to ensure that each dimension is on a similar scale, resulting in a subset D' of astronomical star catalog data of size N.

3. The method for mining outlier data from astronomical catalogs according to claim 2, characterized in that, In the step of performing data normalization or data standardization on the selected feature dimension F The data normalization process scales the data in feature dimension F to a specified range using the following formula: Where min(F) is the minimum value in the sample, max(F) is the maximum value in the sample, and F is the feature dimension, including at least one of redshift, effective temperature, metallicity, magnitude, and line-of-sight velocity; The data standardization process transforms the data in feature dimension F into zero mean and unit variance using the following formula: in, It is the sample mean. It is the sample standard deviation.

4. The method for mining outlier data from astronomical catalogs according to claim 1, characterized in that, The step of performing MC sampling with replacement on the preprocessed astronomical catalog data includes: Determine the sample size n; From a subset D' of astronomical star catalog data, n data points are randomly selected with replacement to obtain a dataset T of size n, where n is much smaller than N.

5. The method for mining outlier data from astronomical catalogs according to claim 4, characterized in that, Step S3 involves repeating step S2 until the outlier index R / S converges, obtaining the KNN outlier score of the astronomical catalog data. Then, based on the KNN outlier score, the outlier index R / S for all data in the astronomical catalog data is calculated. Data points exceeding the outlier threshold are considered outliers, thus obtaining the outlier values ​​in the astronomical catalog data, including: Repeat step S2 until R / S converges to obtain the final vectors R and S. Calculate the outlier index R / S for all data in the astronomical star catalog dataset D. By determining an outlier threshold, data points exceeding the outlier threshold are considered outliers, thus obtaining outlier values ​​in the astronomical catalog data.

6. A device for mining outlier data from astronomical catalogs, characterized in that, include: The data preprocessing module is used to preprocess astronomical catalog data; The data detection module is used to perform MC sampling with replacement and KNN algorithm outlier detection on the preprocessed astronomical catalog data to obtain the KNN outlier score of the sampled astronomical catalog data. This process is repeated until the outlier index R / S converges to obtain the KNN outlier score of the astronomical catalog data. The data mining module is used to calculate the outlier index R / S of all data in the astronomical catalog data based on the KNN outlier score of the astronomical catalog data, and to regard data points that exceed the outlier threshold as outliers, thereby obtaining the outlier values ​​in the astronomical catalog data. The data detection module performs outlier detection using McLeod sampling with replacement and the KNN algorithm on the preprocessed astronomical catalog data, obtaining the KNN outlier scores for the sampled astronomical catalog data, including: Define two vectors R and S, with size N; The preprocessed astronomical catalog data is sampled with replacement by MC sampling to obtain a dataset T of size n, where n is much smaller than N; For each data point in dataset T, recursively select a dimension and find the median on that dimension as the split point to construct the KD tree of dataset T; The constructed KD-tree is used to find the K nearest neighbor data points of each data point, and the average distance between each data point and its K nearest neighbor data points is calculated. The dataset T is then sorted according to this average distance to obtain dataset T', where K is less than n. The sorted index values ​​of dataset T' are accumulated into vector R, and n is added to the corresponding positions in vector S to obtain the KNN outlier scores of the astronomical catalog data.

7. An electronic device, comprising: processor; A memory storing a computer-executable program that, when executed by the processor, causes the processor to perform the method for mining outlier data from astronomical catalogs as described in any one of claims 1-5.

8. A computer-readable medium containing computer-executable instructions, characterized in that, When the computer-executable instructions are executed, the method for mining outlier data from astronomical catalogs as described in any one of claims 1-5 is implemented.

Images