Astronomical data processing method

Abstract

The invention discloses an astronomical data processing method, which comprises the following steps of: processing a fit file so as to change line type storage of the fit file into column type storage; metadata is extracted and independently stored, and main data information except the metadata is continuously stored; performing noise reduction processing on a numerical matrix in the main data information, and independently storing noise data; and carrying out prediction processing on other elements except noise in the numerical value matrix through a plurality of prediction modes, and recording the prediction mode with the minimum compression ratio and the numerical value corresponding to the prediction mode with the minimum compression ratio. The characteristics of locality and noise existence in astronomical data are utilized, a numerical value prediction strategy is applied to a multi-dimensional numerical value matrix, and the lossless compression rate is improved through preprocessing. Firstly, a data arrangement mode of a fit format file is reorganized, and then numerical data with the maximum proportion is subjected to noise separation and numerical prediction, so that effective data are more concentrated and are more suitable for being compressed.

Description

Technical Field

[0001] The present invention relates to the technical field of data compression, and in particular to an astronomical data processing method. Background Art

[0002] Large-scale astronomical observations are an important means of exploring the mysteries of the universe, but what is less known is that the amount of data that needs to be maintained for astronomical observations is extremely large, not inferior to that of data centers. The 500-meter Aperture Spherical Radio Telescope in Guizhou Province can generate tens of GB of data per second during observations. All of this data is stored in the FITs format, and most of it is integer data in single byte units.

[0003] As these massive amounts of data continue to accumulate, their storage problems become severe. When data is required to be preserved completely, data compression is the only way to alleviate the pressure of data storage. However, the development of general compression algorithms is slow, and most algorithms repeatedly balance compression rate and throughput.

[0004] Widely used formats have dedicated compression algorithms, but there are currently few compression schemes for astronomical data, and all have some disadvantages:

[0005] If a lossy compression algorithm is used, although it has a good compression effect, some details in the original data may be lost, destroying the high accuracy and integrity of the data, affecting subsequent data analysis and scientific discoveries, and thus reducing the research value of the data. Some existing compression algorithms for astronomical data, such as HCOMPRESS and RICE compression algorithms, were proposed earlier and have been surpassed in compression rate by new compression algorithms in recent years. They are not targeted enough and have low compression efficiency. Summary of the invention

[0006] The purpose of the present invention is to overcome the above-mentioned defects or problems existing in the background technology and provide an astronomical data processing method.

[0007] To achieve the above objectives, the present invention and its preferred embodiments adopt the following technical solutions, but the embodiments are not limited to the following solutions:

[0008] Solution 1, an astronomical data processing method, comprising the following steps:

[0009] Process the fits file so that the fits file is changed from row storage to column storage;

[0010] Extract and save metadata separately, and store main data information other than metadata continuously;

[0011] Perform noise reduction on the numerical matrix in the main data information and save the noise data separately;

[0012] The elements other than noise in the numerical matrix are predicted through multiple prediction methods, and the prediction method with the smallest compression rate and the corresponding numerical value are recorded.

[0013] Solution 2: Based on Solution 1, the prediction methods include the following three:

[0014] Do not make predictions, record the original values;

[0015] Mean prediction, taking the average value of the previously occurring non-noise data as the predicted value, and recording the difference between the actual value and the predicted value;

[0016] Slope prediction: select a dimension with the smallest change, use the previous value on this dimension as the predicted value, and record the difference between the actual value and the predicted value.

[0017] Solution 3, based on Solution 1, identifies the timestamp data in the file and compresses it by time increments.

[0018] Solution 4 is based on Solution 3. The difference method is used for the part with accuracy to seconds, and the original value is recorded for the millisecond part.

[0019] Solution 5, based on Solution 1, performs simple differential processing on the integer data in the file except the metadata and data matrix, and records the processed data; and records the original values ​​of the floating-point data or string data in the file except the metadata and data matrix.

[0020] Solution 6: Based on Solution 1, for point i(X i1 , X i2 , …, X in ) and point j(X j1 , X j2 , …, X jn ), their values ​​are w i and w j , then in N+1 dimension, the distance between point i and point j is similar to the Euclidean distance, defined as

[0021]

[0022] Define k distance D k (i) is the distance between the kth closest point to point i and point i, and the kth distance domain C k (i) is the distance from point i less than or equal to D k (i) The number of points, reachable distance RD k (i,j)=max{D k (i), distance(i,j)};

[0023] Calculate local reachability density

[0024] Calculate the local outlier factor

[0025] Scheme 7, based on scheme 1, divides the elements in the numerical matrix except the noise into segments, uses the same prediction method in each segment, and records the prediction method at the starting position of each segment.

[0026] Plan eight, based on plan two, the difference between number A and number B is defined as BA, and seven bits in the same byte are used to record the difference value, and the remaining one bit is used to record the calculation sign, indicating that number B = number A + difference value or number A - difference value, and when number A - difference value is a negative number, add 256 to obtain number B, if number A + difference value is greater than 255, subtract 256 to obtain number B.

[0027] Solution 9 is based on Solution 1. Columns with consistent data types are stored consecutively in the storage space, and the attributes and length of the column are placed at the beginning of the column to implement column storage.

[0028] From the above description of the present invention and its preferred embodiments, it can be seen that, compared with the prior art, the technical solution of the present invention and its preferred embodiments have the following beneficial effects due to the adoption of the following technical means:

[0029] 1. A method for processing astronomical data, comprising the following steps: processing a fits file so that the fits file is changed from row storage to column storage; extracting and separately saving metadata, and storing the main data information other than metadata uninterruptedly; performing noise reduction processing on the numerical matrix in the main data information, and separately saving the noise data; performing prediction processing on other elements in the numerical matrix except noise through multiple prediction methods, and recording the prediction method with the lowest compression rate and the corresponding numerical value. This scheme utilizes the characteristics of locality and noise in astronomical data, applies a numerical prediction strategy to the data in the astronomical data, especially the multidimensional numerical matrix, and improves the lossless compression rate through this preprocessing. First, the fits format file is parsed, and its data arrangement is reorganized according to the characteristics of the compression algorithm based on the sliding window class, and then the numerical data with the largest proportion is subjected to noise separation and numerical prediction, so that the effective data is more concentrated, so that it is more suitable for compression. The present invention is aimed at the most widely used fits format file, and can obtain a higher compression rate than the general compression algorithm. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0031] Figure 1 is a system flow chart of this embodiment;

[0032] Figure 2 Schematic diagram of changing row storage to column storage;

[0033] Figure 3 It is the difference scheme for numerical prediction; DETAILED DESCRIPTION

[0034] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are preferred embodiments of the present invention and should not be regarded as excluding other embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0035] In the claims, description and the above-mentioned drawings of the present invention, unless otherwise clearly defined, the use of terms such as "first", "second" or "third" etc. are for distinguishing different objects rather than for describing a specific order.

[0036] In the claims, specification and the above-mentioned drawings of the present invention, unless otherwise explicitly defined, directional words, such as the terms "center", "lateral", "longitudinal", "horizontal", "vertical", "top", "bottom", "inside", "outside", "up", "down", "front", "back", "left", "right", "clockwise", "counterclockwise", etc., indicating directions or positional relationships are based on the directions and positional relationships shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific direction or be constructed and operated in a specific direction, and therefore cannot be understood as limiting the specific protection scope of the present invention.

[0037] In the claims, specification and the above drawings of the present invention, unless otherwise clearly defined, if the term "fixed connection" or "fixed connection" is used, it should be understood in a broad sense, that is, any connection method without a displacement relationship and relative rotation relationship between the two, that is to say, including non-detachable fixed connection, detachable fixed connection, integrated connection and fixed connection through other devices or elements.

[0038] In the claims, description and drawings of the present invention, if the terms "include", "have" and their variations are used, they are intended to mean "including but not limited to".

[0039] An astronomical data processing method is used to process astronomical data so that the compression rate of the astronomical data after being compressed by common compression software or compression algorithm is higher than that of the unprocessed data.

[0040] The astronomical data processing method comprises the following steps:

[0041] refer to Figure 1 :

[0042] Step 1: Process the fits file to change the fits file from row storage to column storage;

[0043] Specifically, read the fits file and parse the file format. fits (Flexible Image Transport System) is a unified standard format for data transmission and exchange between observatories around the world determined by the International Astronomical Union. The file format is parsed through the official format document, mainly the data header and data content (data table). The data table standardizes the table information and the attributes and format of each column;

[0044] Then reorganize the file structure, change the row storage of fits files to column storage; Figure 2 It can be seen that the data table of the fits file is stored in rows, and the attributes of different columns in each row range from floating point numbers to strings. Column storage is achieved by storing columns of the same data type continuously in the storage space and placing the attributes and length of the column at the beginning of the column.

[0045] Step 2: Extract and save metadata separately, and store the main data information other than metadata continuously; metadata includes observation time, mode, project name, telescope name, etc.

[0046] Step 3: Identify the timestamp data in the file and compress it by time increments. Specifically, use the differential method to process the part with precision to seconds, and record the original value for the millisecond part.

[0047] Take a drift scan spectral line data as an example. The column named "DATE-OBS" in the file is a 24-byte string in the form of "2023-03-07T15:04:25.610Z". The data is accurate to milliseconds. For the part accurate to seconds, the string is converted into a Unix timestamp, that is, the number of seconds starting from January 1, 1970. This is an integer, 20230307150425. Since the date is strictly increasing, the integer is differentially calculated, that is, the latter value is subtracted from the former value, the difference is retained, and the original value is accumulated when it is restored. For the value of the millisecond part, because it varies greatly and the value is approximately random, 10 bits (000-999) are used to record its value truthfully. For restoring from integer to string, because the file header has already indicated that the data format of the column is "YYYY-MM-DDThh:mm:ss.xxxZ", when restoring, you only need to parse the value of each time unit and re-fill it in this format without recording additional information.

[0048] Step 4: Perform noise reduction on the numerical matrix in the main data information and save the noise data separately;

[0049] Specifically, for the numerical matrix with the largest proportion in the file (specified in the data header, at least two-dimensional), first identify the noise points in the matrix, and use the outlier judgment algorithm to determine the degree of outlier of a certain element relative to other surrounding elements to identify the noise points and save the noise data separately. Any outlier judgment algorithm can be used to achieve the purpose of identifying noise. This embodiment uses the LOF (Local Outlier Factor) algorithm as an example:

[0050] Specifically, the distance between each data point and all other points is calculated and sorted from near to far: Based on the low density of other data points around the noise point, the distance between each data point and all other points is calculated and sorted from near to far. i1 , X i2 , …, X in ) and point j(X j1 , X j2 , …, X jn ), their values ​​are w i and w j , then in N+1 dimension, the distance between point i and point j is similar to the Euclidean distance, defined as

[0051]

[0052] Define k distance D k (i) is the distance between the kth closest point to point i and point i; k distance domain C k (i), the distance from point i is less than or equal to Dk (i) The number of points, reachable distance RD k (i,j)=max{D k (i), distance(i,j)};

[0053] Calculate local reachability density

[0054] Calculate the local outlier factor

[0055] k is a controllable parameter and in this embodiment uses a value of 32. Finally, 5% of the points with the largest local abnormality factors are selected and saved separately in the metadata in the format of (coordinate, value).

[0056] Step 5: Predict the elements in the numerical matrix except the noise using a variety of prediction methods, and record the prediction method with the smallest compression rate and the corresponding numerical value.

[0057] Preferably, the elements other than the noise in the numerical matrix are segmented, the same prediction method is used in each segment, and the prediction method is recorded at the start position of each segment, so that the compression rate is higher through segmentation processing.

[0058] Specifically, predictions are made for the elements in the numerical matrix except the noise in turn, and the difference between the actual value and the predicted value is recorded.

[0059] Taking the total amount of valid data excluding noise as M as an example, the data is divided into segments of different lengths, and the same prediction method is used in each segment. The starting position of each segment uses a byte to represent the prediction method of the segment. The prediction methods include the following three:

[0060] (a) No prediction is made, that is, the value is recorded according to the original value (record the original value). At this time, because there is no prediction, the subsequent difference step is not performed;

[0061] (b) Mean prediction, that is, the average value of the previously determined non-noise data is used as the prediction value. Here, the average value of all adjacent points in space is selected, that is, for point i (X i1 , X i2 , …, X in ) Select N points to calculate the average value: (X i1-1 , X i2 , …, X in ), (X i1 , X i2 -1, ..., X in ),…,(X i1 , X i2 , …, X in-1) If there is noise among them, select the previous point in the same dimension direction. If the valid value does not exist, default to 128 (the value range is 0 - 255, and the middle value 128 can minimize the difference value obtained in subsequent differences).

[0062] (c) Slope prediction: Select the dimension direction with the smallest change, and use the previous value on this dimension as the predicted value. Specifically, when about to predict the point k (X k1 , X k2 , …, X kn ) in the space dimension of N, find i and j (i ≠ j) such that is the smallest. At this time, use the value of the point

[0063] (X k1 , …, X ki-1 , X kj , …, X kn ) as the predicted value.

[0064] Compare the effects of the three predictions for each segment simultaneously, and use the prediction strategy with the best compression effect as the prediction strategy for this segment. Of course, in other embodiments, other prediction strategies can also be selected.

[0065] Difference step: Refer to Figure 3 , the difference between number A and number B is defined as B - A. Use seven bits in the same byte to record the difference value, and the remaining one bit records the calculation sign, indicating that number B = number A + difference value or number A - difference value. And during the restoration process, if number A - difference value is negative, add 256 to obtain number B.

[0066] Specifically, when 0 ≤ B - A < 128, use the first bit 0 of the byte to indicate that the value of B is A plus another value, and this value is stored in the first 7 bits of the same byte (a number less than 128 can be represented by 7 bits). When - 128 < B - A < 0, similarly use the first bit 1 of the byte to indicate that the value of B is A minus another value, and this value is stored in the first 7 bits of the same byte. When 128 < B - A ≤ 255, use the first bit 1 of the byte to indicate that the value of B is A minus another value and add 256 when the result is negative. For example, when the value of B is 200 and the value of A is 0, the stored difference value is 56, and during restoration, B = 0 - 56 + 256 = 200; when - 256 < B - A < - 128, similarly, use the first bit 0 of the byte to indicate that the value of B is A plus another value and subtract 256 when the result is greater than 255. For example, when the value of B is 0 and the value of A is 200, the stored difference value is 56, and during restoration, B = 200 + 56 - 256. The calculation of the difference value here is similar to the modulo operation of a computer, that is, the difference value = (B - A + 256) % 256.

[0067] Define 00000000 to indicate that the two are the same and the difference between them is exactly ±128, and use 00000001 to indicate that the difference between them is 128 (the value is 0-255, plus 128 and minus 128 are the same number). This step essentially remaps a value range of length 256 to another value range of length 256, but because the predicted value changes the content of the original sequence, this difference step can make the distribution of the values ​​more concentrated, making it more suitable for compression.

[0068] Step 6: Perform simple differential processing on the integer data in the file except the metadata and data matrix, and record the processed data. For the floating-point data or string data (the timestamp has been processed above, so the string data here does not include the timestamp) except the metadata and data matrix in the file, record the original value.

[0069] Finally, the processed fits file is compressed by general compression algorithms such as deflate, zstd, and lzma to form compression packages such as zip, zst, and xz. Compared with the unprocessed fits file, the above method can have a higher compression rate and the corresponding noise data is not deleted, achieving lossless compression.

[0070] This embodiment uses the locality and noise in astronomical data to apply a numerical prediction strategy to the data in the astronomical data, especially the multidimensional numerical matrix, and improves the lossless compression rate through this preprocessing. First, the FITs format file is parsed, and its data arrangement is reorganized according to the characteristics of the compression algorithm based on the sliding window class, and then the numerical data with the largest proportion is subjected to noise separation and numerical prediction, so that the valid data is more concentrated and more suitable for compression. The present invention is aimed at the most widely used FITs format file, and can obtain a higher compression rate than the general compression algorithm.

[0071] Compared with the prior art, the present invention has the following beneficial effects:

[0072] 1. There is a lot of noise in astronomical data, which often interrupts the continuous repeated strings that can be recognized by the general compression algorithm, which has a great impact on the compression of astronomical data. The present invention recognizes and saves the noise separately, and stores the continuous astronomical data for compression by the general compression algorithm, thereby facilitating compression.

[0073] 2. Adjacent parts in astronomical data often have similar patterns, while general compression algorithms only look for the same data stream and cannot discover such patterns. For example, the strings abcd and hijk have similar forms. The present invention can parse such patterns through numerical prediction, predicting the above strings into a111 and h111, thereby improving the data compression rate.

[0074] 3. The present invention has good compatibility and feasibility. It can be used as an independent middle layer without making a lot of changes to the architecture of the entire system, and the last compression step has no strict requirements on the compression algorithm, and a suitable general compression algorithm can be selected according to performance requirements.

[0075] The description of the above specification and embodiments is used to explain the protection scope of the present invention, but does not constitute a limitation on the protection scope of the present invention. Through the enlightenment of the present invention or the above embodiments, ordinary technicians in this field can obtain modifications, equivalent substitutions or other improvements to the embodiments of the present invention or part of the technical features thereof through logical analysis, reasoning or limited experiments, which should be included in the protection scope of the present invention.

Claims

1. A method for processing astronomical data, characterized in that: The following steps are involved: Process the fits file so that the fits file is changed from row storage to column storage; Extract and save metadata separately, and store main data information other than metadata continuously; Perform noise reduction on the numerical matrix in the main data information and save the noise data separately; The elements other than noise in the numerical matrix are predicted through multiple prediction methods, and the prediction method with the smallest compression rate and the corresponding numerical value are recorded.

2. An astronomical data processing method as claimed in claim 1, characterized in that: The prediction methods include the following three: Do not make predictions, record the original values; Mean prediction, taking the average value of the previously occurring non-noise data as the predicted value, and recording the difference between the actual value and the predicted value; Slope prediction: select a dimension with the smallest change, use the previous value on this dimension as the predicted value, and record the difference between the actual value and the predicted value.

3. The astronomical data processing method according to claim 1, characterized in that: Identify timestamp data in the file and compress it by time increments.

4. An astronomical data processing method as claimed in claim 3, characterized in that: The part with precision to seconds is processed using the differential method, and the millisecond part records the original value.

5. The astronomical data processing method according to claim 1, characterized in that: The integer data in the file except the metadata and data matrix are processed by simple difference, and the processed data are recorded. The floating point data or string data in the file except the metadata and data matrix are recorded with the original value.

6. The astronomical data processing method according to claim 1, characterized in that: For a point i(X i1 , X i2 , …, X in ) and point j(X j1 , X j2 , …, X jn ), their values ​​are w i and w j , then in N+1 dimension, the distance between point i and point j is defined as Define k distance D k (i) is the distance between the kth closest point to point i and point i, and the kth distance domain C k (i) is the distance from point i less than or equal to D k (i) The number of points, reachable distance RD k (i,j)=max{D k (i), distance(i,j)}; Calculate local reachability density Calculate the local outlier factor 7. The astronomical data processing method according to claim 1, characterized in that: The elements other than noise in the numerical matrix are segmented, the same prediction method is used in each segment, and the prediction method is recorded at the starting position of each segment.

8. An astronomical data processing method as claimed in claim 2, characterized in that: The difference between numbers A and B is defined as BA. Seven bits in the same byte are used to record the difference value, and the remaining bit is used to record the calculation sign, indicating that number B = number A + difference value or number A - difference value. During the restoration process, if the difference value of number A - is a negative number, add 256 to obtain number B. If the difference value of number A + is greater than 255, subtract 256 to obtain number B.

9. The astronomical data processing method according to claim 1, characterized in that: Column storage is achieved by storing columns of consistent data types consecutively in the storage space and placing the column attributes and length at the beginning of the column.

Images