An astronomical data processing method

Abstract

The application discloses an astronomical data processing method, comprising the following steps: processing a fits file to change the storage mode of the fits file from row type to column type; extracting and separately storing metadata, and uninterruptedly storing main data information except the metadata; performing noise reduction processing on a numerical matrix in the main data information, and separately storing noise data; and performing prediction processing on other elements in the numerical matrix except the noise through multiple prediction modes, and recording a prediction mode with the minimum compression rate and corresponding numerical values. The application utilizes the locality and noise existing in astronomical data, applies a numerical prediction strategy to a multi-dimensional numerical matrix, and improves the lossless compression rate through the preprocessing. Firstly, the data arrangement mode of the fits format file is reorganized, and then noise separation and numerical prediction are performed on the numerical data with the largest proportion, so that the effective data is more concentrated, and thus is more suitable for compression.

Description

Technical Field

[0001] The present invention relates to the technical field of data compression, and in particular to an astronomical data processing method. Background Art

[0002] Large-scale astronomical observations are a crucial tool for exploring the mysteries of the universe. However, what's less well-known is that they require maintaining enormous amounts of data, rivaling even those in data centers. The 500-meter Aperture Spherical Radio Telescope in Guizhou Province can generate tens of gigabytes of data per second during observations. This data is stored in the FITs format, and the vast majority of it is represented by single-byte integers.

[0003] As this massive amount of data continues to accumulate, its storage becomes a serious challenge. When data needs to be preserved completely, data compression is the only way to alleviate this storage pressure. However, the development of universal compression algorithms has been slow, and most algorithms repeatedly balance compression rate with throughput.

[0004] Widely used formats all have dedicated compression algorithms, but there are currently few compression schemes for astronomical data, and all have some shortcomings:

[0005] While lossy compression algorithms offer good compression results, they can lose some details in the original data, compromising the data's high accuracy and integrity, impacting subsequent data analysis and scientific discovery, and ultimately reducing the data's research value. Existing compression algorithms for astronomical data, such as HCOMPRESS and RICE, were proposed relatively recently and have been surpassed in compression ratio by newer algorithms. They also lack specificity and suffer from low compression efficiency. Summary of the Invention

[0006] The purpose of the present invention is to overcome the above-mentioned defects or problems in the background technology and provide an astronomical data processing method.

[0007] To achieve the above objectives, the present invention and its preferred embodiments adopt the following technical solutions, but the embodiments are not limited to the following solutions:

[0008] Solution 1: A method for processing astronomical data, comprising the following steps:

[0009] Process the fits file so that it is stored in a column-based format instead of a row-based format.

[0010] Extract and save metadata separately, and store main data information other than metadata continuously;

[0011] Perform noise reduction on the numerical matrix in the main data information and save the noise data separately;

[0012] The elements other than noise in the numerical matrix are predicted using a variety of prediction methods, and the prediction method with the smallest compression rate and the corresponding numerical value are recorded.

[0013] Solution 2: Based on Solution 1, the prediction methods include the following three:

[0014] Do not make predictions, record the original values;

[0015] Mean prediction: the average value of the previously occurring non-noise data is used as the predicted value, and the difference between the actual value and the predicted value is recorded;

[0016] Slope prediction: select the dimension with the smallest change, use the previous value on this dimension as the predicted value, and record the difference between the actual value and the predicted value.

[0017] Solution 3, based on Solution 1, identifies the timestamp data in the file and compresses it by time increments.

[0018] Solution 4 is based on Solution 3. The difference method is used for the part with accuracy to seconds, and the original value is recorded for the millisecond part.

[0019] Solution 5, based on Solution 1, performs simple differential processing on the integer data in the file except for the metadata and data matrix, and records the processed data; and records the original values ​​of the floating-point data or string data in the file except for the metadata and data matrix.

[0020] Solution 6, based on Solution 1, for point i(X i1 , X i2 ,…,X in ) and point j(X j1 , X j2 ,…,X jn ), their values ​​are w i and w j , then in N+1 dimension, the distance between point i and point j is similar to the Euclidean distance, defined as

[0021]

[0022] Define k distance D k (i) is the distance between the kth closest point to point i and point i, and the kth distance area C k (i) is the distance from point i less than or equal to D k (i) The number of points, reachable distance RD k (i,j)=max{D k (i), distance(i,j)};

[0023] Calculate local reachability density

[0024] Calculate local anomaly factor

[0025] Solution 7, based on Solution 1, divides the elements in the numerical matrix except the noise into segments, uses the same prediction method in each segment, and records the prediction method at the starting position of each segment.

[0026] Plan eight, based on plan two, the difference between numbers A and B is defined as BA. Seven bits in the same byte are used to record the difference value, and the remaining bit is used to record the calculation sign, indicating that number B = number A + difference value or number A - difference value. When number A - difference value is negative, add 256 to obtain number B. If number A + difference value is greater than 255, subtract 256 to obtain number B.

[0027] Solution 9, based on Solution 1, implements column-based storage by storing columns of consistent data types contiguously in the storage space and placing the column attributes and length at the beginning of the column.

[0028] From the above description of the present invention and its preferred embodiments, it can be seen that compared with the prior art, the technical solution of the present invention and its preferred embodiments have the following beneficial effects due to the adoption of the following technical means:

[0029] 1. A method for processing astronomical data, comprising the following steps: processing a FITs file to convert its row-based storage to a column-based storage format; extracting and separately storing metadata, while ensuring uninterrupted storage of primary data information other than the metadata; performing noise reduction on the numerical matrices within the primary data information, and separately storing the noise data; predicting the elements of the numerical matrix other than the noise using multiple prediction methods, and recording the prediction method with the lowest compression ratio and its corresponding numerical value. This scheme utilizes the locality and noise characteristics of astronomical data to apply a numerical prediction strategy to the data within the astronomical data, particularly multidimensional numerical matrices, and improves the lossless compression ratio through this preprocessing. First, the FITs format file is parsed, and its data arrangement is reorganized according to the characteristics of a sliding window-based compression algorithm. Then, noise separation and numerical prediction are performed on the numerical data with the largest proportion, thereby concentrating the valid data and making it more suitable for compression. This method targets the most widely used FITs format files and can achieve a higher compression ratio than general compression algorithms. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0031] Figure 1 is a system flow chart of this embodiment;

[0032] Figure 2 Schematic diagram of changing row storage to column storage;

[0033] Figure 3 It is the difference scheme for numerical prediction; DETAILED DESCRIPTION

[0034] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are preferred embodiments of the present invention and should not be regarded as excluding other embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making any creative efforts shall fall within the scope of protection of the present invention.

[0035] In the claims, description and drawings of the present invention, unless otherwise clearly defined, the use of terms such as "first", "second" or "third" is for the purpose of distinguishing different objects rather than for describing a specific order.

[0036] In the claims, specification and the above-mentioned drawings of the present invention, unless otherwise expressly defined, directional words such as the terms "center", "transverse", "longitudinal", "horizontal", "vertical", "top", "bottom", "inside", "outside", "up", "down", "front", "back", "left", "right", "clockwise", "counterclockwise" and the like indicating directions or positional relationships are based on the directions and positional relationships shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific direction or be constructed and operated in a specific direction, and therefore cannot be understood as limiting the specific scope of protection of the present invention.

[0037] In the claims, description and above-mentioned drawings of the present invention, unless otherwise expressly defined, the terms "fixed connection" or "fixed connection" should be understood in a broad sense, that is, any connection method without displacement relationship and relative rotation relationship between the two parties, that is, including non-detachable fixed connection, detachable fixed connection, integral connection and fixed connection through other devices or elements.

[0038] In the claims, description and drawings of the present invention, if the terms "include", "have" and their variations are used, they are intended to mean "including but not limited to".

[0039] An astronomical data processing method is used to process astronomical data so that the compression rate of the astronomical data after being compressed by common compression software or compression algorithm is higher than that of the unprocessed data.

[0040] The astronomical data processing method comprises the following steps:

[0041] refer to Figure 1 :

[0042] Step 1: Process the fits file to change the fits file from row-based storage to column-based storage;

[0043] Specifically, it reads and parses the FITs file format. FITs (Flexible Image Transport System) is a standardized format for data transmission and exchange between observatories worldwide, established by the International Astronomical Union. The file format is parsed using the official format documentation, primarily the data header and data content (data table). The data table specifies table information and the attributes and format of each column.

[0044] Then reorganize the file structure and change the row storage of fits files to column storage; Figure 2 As you can see, the data tables in the fits file are stored sequentially by row, with the attributes of different columns in each row ranging from floating-point numbers to strings. Column-based storage is achieved by storing columns of consistent data types contiguously in the storage space and placing the column attributes and length at the beginning of the column.

[0045] Step 2: Extract and save metadata separately, and store the main data information other than metadata continuously; metadata includes observation time, mode, project name, telescope name, etc.

[0046] Step 3: Identify the timestamp data in the file and compress it by time increments. Specifically, use the differential method to process the part with precision to seconds, and record the original value for the millisecond part.

[0047] Taking drift scan spectral data as an example, the column named "DATE-OBS" in the file is a 24-byte string of the form "2023-03-07T15:04:25.610Z." This data is accurate to the millisecond. For the second portion, this string is converted to a Unix timestamp, representing the number of seconds since January 1, 1970. This integer is 20230307150425. Because dates are strictly ascending, this integer is differentiated: the previous value is subtracted from the next value, the difference is retained, and the value is accumulated to return the original value during restoration. Because the millisecond value varies greatly and is nearly random, it is recorded as a 10-bit value (000-999). When restoring from integers to strings, because the file header already specifies the data format of the column as "YYYY-MM-DDThh:mm:ss.xxxZ", when restoring, you only need to parse the value of each time unit and re-fill it according to this format, without recording additional information.

[0048] Step 4: Perform noise reduction on the numerical matrix in the main data information and save the noise data separately;

[0049] Specifically, for the numerical matrix with the largest proportion in the file (specified in the data header, at least two-dimensional), first identify the noise points in the matrix, use the outlier judgment algorithm to determine the degree of outlier of a certain element relative to other surrounding elements, identify the noise points, and save the noise data separately. Any outlier judgment algorithm can be used to achieve the purpose of noise identification. This embodiment uses the LOF (Local Outlier Factor) algorithm as an example:

[0050] Specifically, for each data point, the distance between it and all other points is calculated, and the distances are sorted from near to far. Based on the fact that the density of other data points around the noise point is low, the distance between each data point and all other points is calculated, and the distances are sorted from near to far. i1 , X i2 ,…,X in ) and point j(X j1 , X j2 ,…,X jn ), their values ​​are w i and w j , then in N+1 dimension, the distance between point i and point j is similar to the Euclidean distance, defined as

[0051]

[0052] Define k distance D k (i) is the distance between point i and the kth closest point to point i; k distance area C k (i), the distance from point i is less than or equal to Dk (i) The number of points, reachable distance RD k (i,j)=max{D k (i), distance(i,j)};

[0053] Calculate local reachability density

[0054] Calculate local anomaly factor

[0055] As a controllable parameter, k in this embodiment uses a value of 32. Finally, the 5% points with the largest local anomaly factors are selected and saved separately in the metadata using the format of (coordinate, value).

[0056] Step 5: Perform prediction processing on the elements in the numerical matrix except the noise through multiple prediction methods, and record the prediction method with the smallest compression rate and the corresponding value.

[0057] Preferably, the elements other than the noise in the numerical matrix are segmented, the same prediction method is used in each segment, and the prediction method is recorded at the starting position of each segment, so that the compression rate is higher through segmentation processing.

[0058] Specifically, predictions are made for all elements in the numerical matrix except the noise in turn, and the difference between the actual value and the predicted value is recorded.

[0059] Taking the total amount of valid data excluding noise as M as an example, the data is divided into segments of different lengths. The same prediction method is used in each segment. The starting position of each segment is represented by a byte to indicate the prediction method of the segment. The prediction methods include the following three:

[0060] (a) No prediction is made, that is, the value is recorded according to the original value (record the original value). In this case, since there is no prediction, the subsequent difference step is not performed;

[0061] (b) Mean prediction, that is, the average value of the previously determined non-noise data is used as the prediction value. Here, the average value of all adjacent points in space is selected, that is, for point i (X i1 , X i2 ,…,X in )Select N points to calculate the average value: (X i1-1 , X i2 ,…,X in ), (X i1 , X i2 -1,…,X in ),…,(X i1 , X i2 ,…,X in-1) If there is noise among them, select the previous point in the same dimension direction. If the valid value does not exist, default to 128 (the value range is 0 - 255, and the middle value 128 can make the difference value obtained in subsequent differences the smallest).

[0062] (c) Slope prediction: Select the dimension direction with the smallest change, and use the previous value on this dimension as the predicted value. Specifically, when about to predict the point k(X k1 , X k2 , …, X kn ) in the space dimension of N, find i and j (i≠j) such that is the smallest. At this time, use the value of the point

[0063] (X k1 , …, X ki-1 , X kj , …, X kn ) as the predicted value.

[0064] For each segment, compare the effects of the three predictions at the same time, and use the prediction strategy with the best compression effect as the prediction strategy for this segment. Of course, in other embodiments, other prediction strategies can also be selected.

[0065] Difference step: Refer to Figure 3 , the difference between number A and number B is defined as B - A. Use seven bits in the same byte to record the difference value, and the remaining one bit records the calculation sign, indicating that number B = number A + difference value or number A - difference value. And in the restoration process, if number A - difference value is negative, add 256 to obtain number B.

[0066] Specifically, when 0≤B - A<128, use the first bit 0 of the byte to indicate that the value of B is A plus another value, and this value is stored in the first 7 bits of the same byte (a number less than 128 can be represented by 7 bits). When -128<B - A<0, similarly use the first bit 1 of the byte to indicate that the value of B is A minus another value, and this value is stored in the first 7 bits of the same byte. When 128<B - A≤255, use the first bit 1 of the byte to indicate that the value of B is A minus another value and add 256 when the result is negative. For example, when the value of B is 200 and the value of A is 0, the stored difference value is 56, and when restoring, B = 0 - 56 + 256 = 200; when -256<B - A<-128, similarly, use the first bit 0 of the byte to indicate that the value of B is A plus another value and subtract 256 when the result is greater than 255. For example, when the value of B is 0 and the value of A is 200, the stored difference value is 56, and when restoring, B = 200 + 56 - 256. The calculation of the difference value here is similar to the modulo operation of a computer, that is, the difference value = (B - A + 256) % 256.

[0067] We define 00000000 to indicate that the two values ​​are identical and the difference is exactly ±128, and 00000001 to indicate that the two values ​​differ by 128 (for values ​​between 0 and 255, adding 128 and subtracting 128 are the same number). This step essentially remaps a range of 256 values ​​to another range of 256 values. However, because the predicted value changes the content of the original sequence, this difference step can make the distribution of values ​​more concentrated, making them more suitable for compression.

[0068] Step 6: Perform simple differential processing on the integer data in the file except for the metadata and data matrix, and record the processed data. For the floating-point data or string data (the timestamp has been processed above, so the string data here does not include the timestamp) except for the metadata and data matrix in the file, record the original value.

[0069] Finally, the processed fits file is compressed using common compression algorithms such as deflate, zstd, and lzma to form compressed packages such as zip, zst, and xz. Compared with the unprocessed fits file, the above method can achieve a higher compression rate and the corresponding noise and other data are not deleted, achieving lossless compression.

[0070] This embodiment leverages the locality and noise characteristics of astronomical data to apply a numerical prediction strategy to astronomical data, particularly multidimensional numerical matrices. This preprocessing improves lossless compression rates. First, the FITs format file is parsed, and its data arrangement is reorganized based on the characteristics of a sliding window-based compression algorithm. Noise separation and numerical prediction are then performed on the numerical data that accounts for the largest proportion, concentrating the valid data and making it more suitable for compression. This invention targets the most widely used FITs format file and can achieve higher compression rates than general compression algorithms.

[0071] Compared with the prior art, the present invention has the following beneficial effects:

[0072] 1. Astronomical data contains a lot of noise, which often interrupts continuous repeating strings that can be recognized by common compression algorithms, significantly affecting the compression of astronomical data. The present invention identifies and separately stores the noise, storing the continuous astronomical data for compression using a common compression algorithm, thereby facilitating compression.

[0073] Adjacent parts of astronomical data often have similar patterns. Conventional compression algorithms, which only look for identical data streams, cannot detect these patterns. For example, the strings abcd and hijk have similar forms. However, the present invention can analyze these patterns through numerical prediction, predicting the strings as a111 and h111, thereby improving the data compression rate.

[0074] 3. The present invention has good compatibility and feasibility. It can be used as an independent middle layer without making major changes to the entire system architecture. In addition, the last compression step does not have strict requirements on the compression algorithm, and an appropriate general compression algorithm can be selected according to performance requirements.

[0075] The above description and embodiments are intended to explain the scope of protection of the present invention, but do not constitute a limitation thereto. Modifications, equivalent substitutions, or other improvements to the embodiments of the present invention or portions thereof that can be obtained by a person of ordinary skill in the art through logical analysis, reasoning, or limited experimentation based on the teachings of the present invention or the above embodiments, combined with common knowledge, ordinary technical knowledge in the field, and / or prior art, should all be included within the scope of protection of the present invention.

Claims

1. A method for processing astronomical data, characterized in that: The following steps are involved: Process the fits file so that it is stored in a column-based format instead of a row-based format. Extract and save metadata separately, and store main data information other than metadata continuously; Identify timestamp data in files and compress them by time increments; The difference method is used for the part with precision to seconds, and the original value is recorded for the millisecond part; Perform noise reduction on the numerical matrix in the main data information and save the noise data separately; Predict the elements other than noise in the numerical matrix using multiple prediction methods, and record the prediction method with the lowest compression rate and the corresponding value; divide the elements other than noise in the numerical matrix into segments, use the same prediction method in each segment, and record the prediction method at the starting position of each segment; Perform simple differential processing on the integer data in the file except for metadata and data matrix, and record the processed data; record the original value of the floating point data or string data in the file except for metadata and data matrix; Finally, the processed files are compressed.

2. The astronomical data processing method according to claim 1, wherein: The prediction methods include the following three: Do not make predictions, record the original values; Mean prediction: the average value of the previously occurring non-noise data is used as the predicted value, and the difference between the actual value and the predicted value is recorded; Slope prediction: select the dimension with the smallest change, use the previous value on this dimension as the predicted value, and record the difference between the actual value and the predicted value.

3. The astronomical data processing method according to claim 1, wherein: For a point i (X i1 , X i2 ,…,X in ) and point j (X j1 , X j2 ,…,X jn ), their values ​​are w i and w j , then in N+1 dimension, the distance between point i and point j is defined as Define k distance D k (i) is the distance between the kth closest point to point i and point i, and the kth distance area C k (i) is the distance from point i less than or equal to D k (i) The number of points, reachable distance RD k (i,j)=max{D k (i), distance(i,j)}; Calculate local reachability density ; Calculate local anomaly factor .

4. The astronomical data processing method according to claim 2, wherein: The difference between numbers A and B is defined as BA. Seven bits of the same byte are used to record the difference value, and the remaining bit is used to record the calculation sign, indicating that number B = number A + difference value or number A - difference value. During the restoration process, if the difference value of number A - difference value is negative, 256 is added to obtain number B. If the difference value of number A + difference value is greater than 255, 256 is subtracted to obtain number B.

5. The astronomical data processing method according to claim 1, wherein: Column-based storage is achieved by storing columns of consistent data types contiguously in the storage space and placing the column attributes and length at the beginning of the column.

Images