Hierarchical storage management method based on land observation satellite data value

Abstract

The invention provides a hierarchical storage management method based on land observation satellite data value, which belongs to the field of application in a land observation satellite data processing system, and comprises the following steps of: calculating an initial comprehensive weight score during data archiving by creating seven types of core data tables such as a satellite parameter table and a data value parameter table; determining the number of storage days of the online or near-line cluster in combination with a dynamic storage strategy table; obtaining user order update data return times in real time, and recalculating the weight score at regular time to dynamically adjust the storage strategy; and finally, realizing data cleaning in combination with the water and fire wire threshold of the storage device. The problems of low resource utilization efficiency and slow user data return caused by making a storage strategy only based on the number of storage days in the prior art are solved, and the storage resource utilization rate and the user data response efficiency are remarkably improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of land observation satellite data processing system, in particular to a hierarchical storage management method based on land observation satellite data value. BACKGROUND

[0002] The land observation satellite ground processing system currently adopts a three-level storage architecture of online, near-line and offline, and stores and manages the 0-level data and product data generated daily: the newly added data is stored online for a certain number of days and then deleted, and is synchronized to the near-line tape library (stored offline for a certain number of days) and the near-line cluster storage every day; the storage strategy can be configured through the interface to determine the online or near-line storage days corresponding to the satellite type and data level, and the data cleaning is realized in combination with the water line threshold of the storage device. If the user-ordered data has been deleted from the online or near-line cluster storage, it needs to be migrated back to the online storage from the near-line tape library for distribution; if the ordered product has not been produced, the 0-level data needs to be migrated back to the online storage for production.

[0003] The existing storage strategy has obvious defects: only the satellite type and data level are used to determine the online or near-line cluster storage days, without considering the difference in data value, for example, data that is frequently migrated back due to many user orders, or data with high satellite resolution, less cloud cover and high data quality, which are stored with the same period as low-value data, resulting in unreasonable allocation and low utilization of online or near-line cluster storage resources, and the user-ordered data needs to be frequently migrated back from the tape library, which is slow in response.

[0004] Therefore, the present application provides a hierarchical storage management method based on land observation satellite data value. SUMMARY

[0005] The present application provides a hierarchical storage management method based on land observation satellite data value to solve the above technical problems.

[0006] The present application provides a hierarchical storage management method based on land observation satellite data value, comprising:

[0007] Step 1: create a satellite parameter table, a data value parameter table, a comprehensive weight table, a dynamic storage strategy table, a data migration frequency statistics table, a data cleaning water line table and a storage table, wherein the parameters of the data value parameter table include product level, cloud cover, data quality, data migration frequency, satellite image resolution, and record the original value interval, corresponding score, weight percentage and value range of each parameter; the dynamic storage strategy table includes satellite, product level, weight score interval, online storage days, near-line cluster storage days, whether to clean up, product type name, whether to archive to tape library, target tape pool, whether to near-line cluster storage field, and supports dynamic adjustment through interface configuration;

[0008] Step 2: When the Level 0 or product data generated by the ground processing system is archived, first read the XML file of the Level 0 or product data, insert the parsed metadata information into the corresponding Level 0 or product metadata table, then read the entity data path of the Level 0 or product data and insert it into the storage table of the database. Then, periodically read the entity data path of the Level 0 or product data to be archived from the database, archive the entity data to online storage, and then migrate it to nearline cluster storage and nearline tape library respectively. At the same time, read the cloud volume, data quality, number of migrations initially set to 0, resolution, and product level of each scene of Level 0 or product data, and calculate the initial comprehensive weight score of each scene of data in combination with the data value parameter table, and store it in the comprehensive weight table.

[0009] Step 3: Read the dynamic storage strategy table, match the corresponding weight score range according to the weight score of each data in the comprehensive weight table, obtain the online storage days and near-line cluster storage days for the corresponding range, and update the comprehensive weight table.

[0010] Step 4: Obtain user order messages and store them in the user order table. Generate product or level 0 data migration tasks based on the order type and store them in the migration task table. Periodically perform joint queries on the user order table, migration task table, level 0 scene data table, or standard product data table to obtain the satellite code, payload code, product level, scene number or product number, cloud cover, data quality, image resolution, data reception time, order number, order time, username, migration count, online migration count, near-line cluster migration count, and tape library migration count for each scene's ordered data, and store them in the data migration count statistics table.

[0011] Step 5: Periodically read the scene number or product number, migration count, cloud volume, data quality, resolution, and product level of each scene data in the data migration count statistics table, recalculate the comprehensive weight score of each scene data in conjunction with the data value parameter table, and save it to the comprehensive weight table; if the comprehensive weight score changes, repeat step 3 and then execute step 6; if it does not change, directly execute step 6.

[0012] Step 6: Start the data cleanup thread periodically, query the data cleanup fire / water line table to determine if the capacity of the online storage or nearline cluster storage device has reached the cleanup fire / water line threshold; if not, the cleanup thread goes to sleep; if it has, query the dynamic storage policy table to determine if the data needs to be cleaned; if not, the cleanup thread goes to sleep; if it needs to be cleaned, query the storage table and comprehensive weight table to obtain information on cleanable data, clean up data that meets the condition that the difference between the archive time and the current time exceeds the corresponding number of days of online or nearline cluster storage and has been backed up to the nearline tape library, and update the online or nearline cluster data status of the data.

[0013] Preferably, the satellite parameter table includes satellite code, satellite name, sensor code, sensor name, and resolution fields, and records the above information of satellites already in orbit;

[0014] The comprehensive weighting table includes satellite, product level, scene number or product number, weight score, number of days stored online, and number of days stored in nearline clusters.

[0015] The data migration count statistics table includes fields such as satellite code, payload code, product level, scene number or product number, data reception time, order number, order time, username, migration count, online migration count, near-line cluster migration count, tape library migration count, cloud cover, data quality, and satellite image resolution.

[0016] The data cleaning hot-line and water-line tables include ID, hot-line, and water-line fields to distinguish between online and near-line cluster storage, and can be dynamically adjusted through interface configuration;

[0017] The storage table includes the following fields: storage number, satellite code, sensor code, product level, product number, ground station code, orbit circle number, product type, data archiving time, online data path, file size, online data status, nearline cluster archiving time, nearline cluster data path, nearline cluster data status, and storage object number.

[0018] Preferably, the initial comprehensive weight score for each scene of data is calculated by combining the data value parameter table, including: Initial comprehensive weight score = score corresponding to the original cloud amount value × first weight coefficient + score corresponding to the data quality × second weight coefficient + score corresponding to the product level × third weight coefficient + score corresponding to the satellite image resolution × fourth weight coefficient + score corresponding to the number of data migrations × fifth weight coefficient; wherein, the first weight coefficient + second weight coefficient + third weight coefficient + fourth weight coefficient + fifth weight coefficient = 1, and the values ​​of different weight coefficients are determined according to user needs.

[0019] Preferably, the archiving of each satellite is carried out on a corresponding data archiving topic on the Kafka cluster, and the data archiving is implemented in a multi-threaded parallel manner.

[0020] Preferably, obtaining user order messages includes:

[0021] Product order or production order messages generated by the distribution service subsystem are published to a specified topic in the Kafka cluster.

[0022] The data management subsystem retrieves user order messages from the specified topic in real time.

[0023] Preferably, when the land observation satellite is a radar satellite, the score corresponding to the original cloud cover value and the first weight coefficient are removed when calculating the comprehensive weight score in steps 2 and 5.

[0024] Preferably, it also includes: when database access is required, reading the encrypted username and password in the configuration file, decrypting them, connecting to the database, and reading the data path of the level 0 or product data entity to be archived in the storage table.

[0025] Preferably, before reading the encrypted username and password from the configuration file, the method further includes: securely encrypting the username and password, specifically including:

[0026] The original key is randomly divided according to rand(1, n), and the random division is randomly filtered according to rand(1, rand(1, n)) to obtain m1 divisions, where n represents a random number from 2 to 9;

[0027] Combine the first and last fields of each segment, and determine the filtering coordinates of the combined fields from the combination-coordinate filtering lookup table;

[0028] Map the first dynamic change of the username and the second dynamic change of the password of the same user to a unified coordinate system each time the username is modified;

[0029] The permission element attribute set is determined based on the user's user permissions, and a first number of irrelevant elements and a second number of relevant elements are extracted from all permission element attribute sets based on the segment attributes of the segment division.

[0030] Determine whether the filtering coordinates need to be expanded based on the incoherent and coherent elements.

[0031] If not required, determine a two-dimensional representation based on the selected coordinates in the unified coordinate system;

[0032] If necessary, a two-dimensional representation based on extended coordinates is determined in the unified coordinate system;

[0033] The corresponding segments are optimized according to the two-dimensional expression to obtain the security key.

[0034] Compared with the prior art, the beneficial effects of this application are as follows:

[0035] Other features and advantages of the invention will be set forth in the following description, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention may be realized and obtained by means of the structures particularly pointed out in the written description and the accompanying drawings.

[0036] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0037] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0038] Figure 1 This is a flowchart illustrating a hierarchical storage management method based on the value of land observation satellite data in an embodiment of the present invention;

[0039] Figure 2 This is a flowchart illustrating the implementation process of online storage / near-line cluster storage level 0 or product data cleanup for land observation satellites in this embodiment of the invention. Detailed Implementation

[0040] The preferred embodiments of the present invention will be described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.

[0041] This invention provides a hierarchical storage management method based on the value of land observation satellite data, such as... Figure 1 As shown, it includes:

[0042] Step 1: Create a satellite parameter table, a data value parameter table, a comprehensive weight table, a dynamic storage strategy table, a data migration count statistics table, a data cleanup hot and cold line table, and a storage table. The data value parameter table includes parameters such as product level, cloud cover, data quality, data migration count, and satellite image resolution, and records the original value range, corresponding score, weight percentage, and value range for each parameter. The dynamic storage strategy table includes fields for satellite, product level, weight score range, online retention days, near-line cluster retention days, whether cleanup is required, product type name, whether tape library archiving is required, target tape pool, and whether near-line cluster storage is required, and supports dynamic adjustment via interface configuration.

[0043] Step 2: When the Level 0 or product data generated by the ground processing system is archived, first read the XML file of the Level 0 or product data, insert the parsed metadata information into the corresponding Level 0 or product metadata table, then read the entity data path of the Level 0 or product data and insert it into the storage table of the database. Then, periodically read the entity data path of the Level 0 or product data to be archived from the database, archive the entity data to online storage, and then migrate it to nearline cluster storage and nearline tape library respectively. At the same time, read the cloud volume, data quality, number of migrations initially set to 0, resolution, and product level of each scene of Level 0 or product data, and calculate the initial comprehensive weight score of each scene of data in combination with the data value parameter table, and store it in the comprehensive weight table.

[0044] Step 3: Read the dynamic storage strategy table, match the corresponding weight score range according to the weight score of each data in the comprehensive weight table, obtain the online storage days and near-line cluster storage days for the corresponding range, and update the comprehensive weight table.

[0045] Step 4: Obtain user order messages and store them in the user order table. Generate product or level 0 data migration tasks based on the order type and store them in the migration task table. Periodically perform joint queries on the user order table, migration task table, level 0 scene data table, or standard product data table to obtain the satellite code, payload code, product level, scene number or product number, cloud cover, data quality, image resolution, data reception time, order number, order time, username, migration count, online migration count, near-line cluster migration count, and tape library migration count for each scene's ordered data, and store them in the data migration count statistics table.

[0046] Step 5: Periodically read the scene number or product number, migration count, cloud volume, data quality, resolution, and product level of each scene data in the data migration count statistics table, recalculate the comprehensive weight score of each scene data in conjunction with the data value parameter table, and save it to the comprehensive weight table; if the comprehensive weight score changes, repeat step 3 and then execute step 6; if it does not change, directly execute step 6.

[0047] Step 6: Start the data cleanup thread periodically, query the data cleanup fire / water line table to determine if the capacity of the online storage or nearline cluster storage device has reached the cleanup fire / water line threshold; if not, the cleanup thread goes to sleep; if it has, query the dynamic storage policy table to determine if the data needs to be cleaned; if not, the cleanup thread goes to sleep; if it needs to be cleaned, query the storage table and comprehensive weight table to obtain information on cleanable data, clean up data that meets the condition that the difference between the archive time and the current time exceeds the corresponding number of days of online or nearline cluster storage and has been backed up to the nearline tape library, and update the online or nearline cluster data status of the data.

[0048] Preferably, the satellite parameter table includes satellite code, satellite name, sensor code, sensor name, and resolution fields, and records the above information of satellites already in orbit;

[0049] The comprehensive weighting table includes satellite, product level, scene number or product number, weight score, number of days stored online, and number of days stored in nearline clusters.

[0050] The data migration count statistics table includes fields such as satellite code, payload code, product level, scene number or product number, data reception time, order number, order time, username, migration count, online migration count, near-line cluster migration count, tape library migration count, cloud cover, data quality, and satellite image resolution.

[0051] The data cleaning hot-line and water-line tables include ID, hot-line, and water-line fields to distinguish between online and near-line cluster storage, and can be dynamically adjusted through interface configuration;

[0052] The storage table includes the following fields: storage number, satellite code, sensor code, product level, product number, ground station code, orbit circle number, product type, data archiving time, online data path, file size, online data status, nearline cluster archiving time, nearline cluster data path, nearline cluster data status, and storage object number.

[0053] Preferably, the initial comprehensive weight score for each scene of data is calculated by combining the data value parameter table, including: Initial comprehensive weight score = score corresponding to the original cloud amount value × first weight coefficient + score corresponding to the data quality × second weight coefficient + score corresponding to the product level × third weight coefficient + score corresponding to the satellite image resolution × fourth weight coefficient + score corresponding to the number of data migrations × fifth weight coefficient; wherein, the first weight coefficient + second weight coefficient + third weight coefficient + fourth weight coefficient + fifth weight coefficient = 1, and the values ​​of different weight coefficients are determined according to user needs.

[0054] Preferably, the archiving of each satellite is carried out on a corresponding data archiving topic on the Kafka cluster, and the data archiving is implemented in a multi-threaded parallel manner.

[0055] Preferably, obtaining user order messages includes:

[0056] Product order or production order messages generated by the distribution service subsystem are published to a specified topic in the Kafka cluster.

[0057] The data management subsystem retrieves user order messages from the specified topic in real time.

[0058] Preferably, when the land observation satellite is a radar satellite, the score corresponding to the original cloud cover value and the first weight coefficient are removed when calculating the comprehensive weight score in steps 2 and 5.

[0059] In this embodiment, such as Figure 2 The diagram shown is a flowchart of the implementation process for cleaning up Level 0 or product data in online storage / near-line cluster storage for land observation satellites.

[0060] In this embodiment, the satellite parameter table is shown in Table 1:

[0061] Table 1

[0062] satellite id satellite name sensor id sensor name resolution XXXX-01 Land Observation Satellite XXXX PMS-02 Panchromatic Multispectral Sensor 5m RD-02 Radar Observation Satellite RD-02 SAR-01 Synthetic Aperture Radar 3m

[0063] In this embodiment, the rules for the data value parameter table are shown in Table 2:

[0064] Table 2

[0065] Parameter Number Parameter Original Interval Corresponding Score Weight Percentage Value Range (%) 1 cloud cover 0--20 5 25％ 0-100 2 cloud cover 21--100 2 25％ 0-100 3 data quality 6--9 5 25％ 0-9 4 data quality 0--5 2 25％ 0-9 5 restore time 3 or more 5 20％ greater than or equal to 0 6 restore time 0-2 2 20％ greater than or equal to 0 7 product level product 5 15％ 8 product level scene or strip 2 15％ 9 resolution <10m 5 15％ 10 resolution 10m 2 15％

[0066] In this embodiment, the data cleanup water and fire line table is shown in Table 3:

[0067] Table 3

[0068] Storage Type Number fire line water line on-line storage 1 85％ 75％ near-line cluster storage 2 90％ 80％

[0069] In this embodiment, taking XXXX satellite Level 0 data and XXXX satellite L1A product data as examples, the archiving process is detailed as follows:

[0070] Kafka Parallel Topic Configuration: Create a dedicated Kafka topic (data_archive_XXXX) for the XXXX satellite, set up 3 partitions (corresponding to 3 archiving threads), and process an average of 1,000 scenes of data per day with a single thread. Multi-threaded parallel processing can increase this to 3,000 scenes per day, avoiding archiving bottlenecks.

[0071] Level 0 Data Archiving: Read the Level 0 metadata XML from the Kafka topic: / DIS / Level0Data / XXXX / 701 / MYC / XXXX-TMS-20200909-577-136-L00000600456.XML, parse it, and insert it into the XXXX_scene table (fields include sceneid="600456", cloudcover=15%, dataquality=8, etc.).

[0072] Read the Level 0 entity data path: / DIS / Level0Data / XXXX / 701 / MYC / XXXX-MUX-20200909-577-136-L00000600456.DAT, archive it to the online storage path: / ARCHIVE / XXXX / SCENE / TMS / 2020 / 0909 / 701 / MYC-XXXX-MUX-20200909-000701-L00000600456.DAT, and then synchronously migrate it to the near-line cluster storage (path {"bucket":"XXXX","key":"XXXX_L00000600456.DAT","fileSize":678518400}) and the near-line tape library (tape pool XXXX_Scenedata);

[0073] Initial weight calculation: Read restoretimes=0 (first archive), resolution=5m, productlevel=“scene”, and calculate using the Datavaluepar table:

[0074] Weighted score = 15% cloud cover (5 points × 25%) + 8 points quality (5 points × 25%) + 0 migrations (2 points × 20%) + 5m resolution (5 points × 15%) + scene level (2 points × 15%) = 1.25 + 1.25 + 0.4 + 0.75 + 0.3 = 3.95 points, stored in the comprehenweight table.

[0075] In this embodiment, L1A product data archiving:

[0076] Read the L1A metadata XML path from the Kafka topic: / PGS / WORKDIR / PRODUCTS / XXXX / 20240621 / TMS / L1A / XXXX-TMS-W5.8_N55.1_20240620_L1A0000811408.XML, parse it, and insert it into the XXXX_productl1 table;

[0077] Entity data archiving path: Online storage / ARCHIVE / XXXX / LEVEL1A / TMS / 2024 / 0621 / XXXX_TMS_W5.8_N55.1_20240620_L1A0000811408.tar.gz, synchronously migrated to the nearline cluster (path {"bucket":"XXXX","key":"XXXX_TMS_W5.8_N55.1_20240620_L1A0000811408.tar.gz","fileSize":752785704}) and the nearline tape library (tape pool XXXX_Product);

[0078] Initial weight calculation: cloudcover = 10% (5 points × 25%), dataquality = 9 (5 points × 25%), restoretimes = 0 (2 points × 20%), resolution = 5m (5 points × 15%), productlevel = "product" (5 points × 15%), score = 1.25 + 1.25 + 0.4 + 0.75 + 0.75 = 4.4 points.

[0079] Storage policy matching:

[0080] Read the dynamic storage strategy table (dynamicstrategy) and match the number of days to store the data based on the weighted score:

[0081] XXXX satellite level 0 data (3.95 points): Matching satellite = XXXX, Level 0, weight range 3-4 (≤4), corresponding onlinedays = 30 days, nearlinedays = 60 days, updated to comprehenweight table;

[0082] XXXX Satellite L1A Product (4.4 points): Matching Satellite = XXXX, Level 1, Weight Range 4-5, corresponding onlinedays = 180 days, nearlinedays = 360 days, updated to comprehenweight table.

[0083] Statistics on the number of relocations:

[0084] Order Acquisition: The distribution service subsystem generates product orders and publishes them to the Kafka topic productdata_require. The message content is: orderid = "ORD20240621001", username = "user001", productid = "711262", and ordertype = "product migration". The data management subsystem consumes this message, stores it in the Orderinfo table, and generates a migration task taskd = "TASK20240621001" which is stored in the Taskinfo table.

[0085] Migration statistics query: At 2:00 AM every day, perform a joint query on the Orderinfo, Taskinfo, and XXXX_productl1 tables to collect migration information for productid="711262", as shown in Table 4, Migration frequency statistics table:

[0086] Table 4

[0087]

[0088] In this embodiment, the parameter productid = "711262" in table 4 of datarestoresta is read, and the weight is recalculated:

[0089] restoretimes = 3 (≥3, 5 points × 20% = 1 point), other parameters remain unchanged;

[0090] New weighted score = 1.25 (cloud cover) + 1.25 (quality) + 1 (return migration) + 0.75 (resolution) + 0.75 (level) = 5 points;

[0091] Because the score changed from 4.4 to 5: the matching weight range is 4-5 (no change), and onlinedays = 180 days and nearlinedays = 360 days remain unchanged.

[0092] In this embodiment, data cleaning: taking the XXXX satellite L1A product (productid=711262) as an example, the cleaning process is detailed as follows:

[0093] Threshold determination: The cleanup thread starts at 3:00 AM every day and queries the clearconfig table: the online storage used capacity is 88% (exceeding the fire threshold of 85%), which requires further judgment; the near-line cluster storage used capacity is 82% (not exceeding the fire threshold of 90%), which will not be processed for the time being.

[0094] Cleanup strategy verification: Query the dynamicstrategy table: productid = "711262" corresponds to isclean = 1 (cleanup required) and onlinedays = 180 days; Query the storageinfo table: the data was archived on 2024-01-14, the current time is 2024-07-15, the time difference is 182 days (more than 180 days), and it has been backed up to the tape library (tapepoolid = "XXXX_Scenedata").

[0095] Cleanup execution: Delete product data on the online storage path: / archive / XXXX / LEVEL1A / PMS / 2024 / 0114 / 19280 / 711262 / XXXX_PMS_E111.6_N57.6_20240114_L1A0000711262.tar.gz;

[0096] Update the storageinfo table to onlinedatastatus="offline";

[0097] Record the cleanup log (including productid, cleanup time, and cleanup path).

[0098] In this embodiment, taking Level 0 data from radar satellite RD-02 as an example, since radar satellites are not affected by cloud cover, the cloud cover and 25% weight are removed when calculating the weights, and the total weights of other parameters are adjusted to 1 (dataquality = 30%, restoretimes = 25%, productlevel = 20%, resolution = 25%).

[0099] Data parameters: dataquality = 7 (5 points), restoretimes = 2 (2 points), productlevel = "scene" (2 points), resolution = 3m (5 points);

[0100] Weighted score = 5 × 30% + 2 × 25% + 2 × 20% + 5 × 25% = 1.5 + 0.5 + 0.4 + 1.25 = 3.65 points;

[0101] Match RD-02, Level0, and weight range 3-4 in the dynamicstrategy table, corresponding to onlinedays=30 days and nearlinedays=60 days.

[0102] In this embodiment, the weighting table rules are as shown in Table 5:

[0103] Table 5

[0104]

[0105] The beneficial effects of the above technical solution are as follows: This method is applied to the design and implementation of online storage and near-line cluster storage strategies for Level 0 data and product data from land observation satellites in ground processing systems. For each newly generated Level 0 data and product data scene daily, the initial value score for each scene is calculated by comprehensively considering the scores and weights of various parameters such as product level, satellite image resolution, cloud cover (this parameter is not considered for radar satellites), data quality, and data migration count. Based on the dynamic storage strategy table, the online storage and near-line cluster storage retention time for each scene can be obtained. For each scene, as the number of migrations increases, the comprehensive weight score will change, and the online storage and near-line cluster storage time will also change dynamically accordingly. This optimizes the online and near-line data storage strategies, improves the utilization rate of online and near-line cluster storage resources, and further enhances the efficiency of data migration and provision.

[0106] This invention provides a hierarchical storage management method based on the value of land observation satellite data, which further includes: when accessing the database is required, reading the encrypted username and password in the configuration file, decrypting them, connecting to the database, and reading the data path of the level 0 or product data entity to be archived in the storage table.

[0107] Preferably, before reading the encrypted username and password from the configuration file, the method further includes: securely encrypting the username and password, specifically including:

[0108] The original key is randomly divided according to rand(1, n), and the random division is randomly filtered according to rand(1, rand(1, n)) to obtain m1 divisions, where n represents a random number from 2 to 9;

[0109] Combine the first and last fields of each segment, and determine the filtering coordinates of the combined fields from the combination-coordinate filtering lookup table;

[0110] Map the first dynamic change of the username and the second dynamic change of the password of the same user to a unified coordinate system each time the username is modified;

[0111] The permission element attribute set is determined based on the user's user permissions, and a first number of irrelevant elements and a second number of relevant elements are extracted from all permission element attribute sets based on the segment attributes of the segment division.

[0112] Determine whether the filtering coordinates need to be expanded based on the incoherent and coherent elements.

[0113] If not required, determine a two-dimensional representation based on the selected coordinates in the unified coordinate system;

[0114] If necessary, a two-dimensional representation based on extended coordinates is determined in the unified coordinate system;

[0115] The corresponding segments are optimized according to the two-dimensional expression to obtain the security key.

[0116] In this embodiment, the random partitioning of the original key is as follows: let the original key for database access (including username and plaintext password) be K, take n = 5 (n is a random number from 2 to 9), and divide K into 5 fields S1 (user prefix), S2 (user suffix), S3 (password prefix), S4 (password suffix), and S5 (checksum) according to rand(1,5); then, according to rand(1,rand(1,5)) = rand(1,3), randomly select 3 partitions (such as S1, S3, S5) as m1 = 3 core processing segments.

[0117] In this embodiment, the selection of the selection coordinates is determined by combining the first and last fields of each core segment: S1 has the first field as usr and the last field as 01, and the combination is [usr,01]; the preset combination-coordinate selection lookup table is queried, and the combination corresponds to coordinates (2,3); similarly, the combination [pwd,89] of S3 corresponds to coordinates (5,1), and the combination [chk,ff] of S5 corresponds to coordinates (7,4).

[0118] In this embodiment, the dynamic change mapping records the first dynamic change ΔU each time a user changes their username, such as changing the username from admin01 to admin02, ΔU = 01→02; and the second dynamic change ΔP when changing the password, such as increasing the password complexity from mid to high, ΔP = mid→high. The (ΔU, ΔP) is mapped to a unified user-password coordinate system (the horizontal axis represents the username change and the vertical axis represents the password change), resulting in the mapping point (0.2, 0.8).

[0119] The extraction of permission elements and coordinate determination are based on the current user's permissions (e.g., data administrator) to determine the permission element attribute set: {data read / write rights, table modification rights, threshold configuration rights, log viewing rights}. Combined with the authentication segment attributes of the core segment S1, irrelevant elements (table modification rights, threshold configuration rights, first quantity = 2) and relevant elements (data read / write rights, log viewing rights, second quantity = 2) are extracted. Since the number of relevant elements is greater than or equal to the number of irrelevant elements, there is no need to expand the filter coordinates. If there are 3 irrelevant elements and 1 relevant element (3>1), then the filter coordinates (e.g., the original (2,3)) need to be expanded to (2,3,1) (adding a dimension) or (3,4) (adjusting the value).

[0120] The security key generation involves constructing a two-dimensional expression of filtering coordinates and dynamically mapped points in a unified coordinate system: ((2,3),(0.2,0.8)), ((5,1),(0.2,0.8)), ((7,4),(0.2,0.8)). Fields of S1, S3, and S5 are reorganized according to this expression (e.g., S1 is reorganized into usr_01→02_01) to obtain the security key K', which is then stored in the configuration file. When accessing the database, K' is decrypted to obtain the username / password, and the database is connected to read the path of the data to be archived from the storage table. For example, the segment DBU corresponds to the two-dimensional expression ((2,3),(5,1)), and DBU is shifted (each character's ASCII code is increased by 2) to obtain FCW; the segment 025_Pwd corresponds to the two-dimensional expression ((4,6),(5,1)), and characters are replaced (0 is replaced with 1, 5 with 6, P with Q) to obtain 126_Qwd; the final security key is FCW126_Qwd.

[0121] The beneficial effects of the above technical solution are as follows: by performing multi-dimensional encryption processing on the original username and password through random segmentation, coordinate mapping, permission association, and dynamic adaptation, it not only enhances the complexity of encryption by utilizing randomization and coordinate mapping, but also avoids the vulnerabilities of a single encryption method by combining the dynamic nature of user permissions and information changes, thus greatly improving the security of database access authentication, and can flexibly adapt to scenarios of user permission adjustments and account information changes.

[0122] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A hierarchical storage management method based on the value of land observation satellite data, characterized in that, include: Step 1: Create a satellite parameter table, a data value parameter table, a comprehensive weight table, a dynamic storage strategy table, a data migration count statistics table, a data cleanup hot and cold line table, and a storage table. The data value parameter table includes parameters such as product level, cloud cover, data quality, data migration count, and satellite image resolution, and records the original value range, corresponding score, weight percentage, and value range for each parameter. The dynamic storage strategy table includes fields for satellite, product level, weight score range, online retention days, near-line cluster retention days, whether cleanup is required, product type name, whether tape library archiving is required, target tape pool, and whether near-line cluster storage is required, and supports dynamic adjustment via interface configuration. Step 2: When the Level 0 or product data generated by the ground processing system is archived, first read the XML file of the Level 0 or product data, insert the parsed metadata information into the corresponding Level 0 or product metadata table, then read the entity data path of the Level 0 or product data and insert it into the storage table of the database. Then, periodically read the entity data path of the Level 0 or product data to be archived from the database, archive the entity data to online storage, and then migrate it to nearline cluster storage and nearline tape library respectively. At the same time, read the cloud volume, data quality, number of migrations initially set to 0, resolution, and product level of each scene of Level 0 or product data, and calculate the initial comprehensive weight score of each scene of data in combination with the data value parameter table, and store it in the comprehensive weight table. Step 3: Read the dynamic storage strategy table, match the corresponding weight score range according to the weight score of each data in the comprehensive weight table, obtain the online storage days and near-line cluster storage days for the corresponding range, and update the comprehensive weight table. Step 4: Obtain user order messages and store them in the user order table. Generate product or level 0 data migration tasks based on the order type and store them in the migration task table. Periodically perform joint queries on the user order table, migration task table, level 0 scene data table, or standard product data table to obtain the satellite code, payload code, product level, scene number or product number, cloud cover, data quality, image resolution, data reception time, order number, order time, username, migration count, online migration count, near-line cluster migration count, and tape library migration count for each scene's ordered data, and store them in the data migration count statistics table. Step 5: Periodically read the scene number or product number, migration count, cloud volume, data quality, resolution, and product level of each scene data in the data migration count statistics table, recalculate the comprehensive weight score of each scene data in conjunction with the data value parameter table, and save it to the comprehensive weight table; if the comprehensive weight score changes, repeat step 3 and then execute step 6; if it does not change, directly execute step 6. Step 6: Start the data cleanup thread periodically, query the data cleanup fire / water line table to determine if the capacity of the online storage or nearline cluster storage device has reached the cleanup fire / water line threshold; if not, the cleanup thread goes to sleep; if it has, query the dynamic storage policy table to determine if the data needs to be cleaned; if not, the cleanup thread goes to sleep; if it needs to be cleaned, query the storage table and comprehensive weight table to obtain information on cleanable data, clean up data that meets the condition that the difference between the archive time and the current time exceeds the corresponding number of days of online or nearline cluster storage and has been backed up to the nearline tape library, and update the online or nearline cluster data status of the data.

2. The hierarchical storage management method based on the value of land observation satellite data according to claim 1, characterized in that, The satellite parameter table includes satellite code, satellite name, sensor code, sensor name, and resolution fields, and records the above information for satellites already in orbit. The comprehensive weighting table includes satellite, product level, scene number or product number, weight score, number of days stored online, and number of days stored in nearline clusters. The data migration count statistics table includes fields such as satellite code, payload code, product level, scene number or product number, data reception time, order number, order time, username, migration count, online migration count, near-line cluster migration count, tape library migration count, cloud cover, data quality, and satellite image resolution. The data cleaning hot-line and water-line tables include ID, hot-line, and water-line fields to distinguish between online and near-line cluster storage, and can be dynamically adjusted through interface configuration; The storage table includes the following fields: storage number, satellite code, sensor code, product level, product number, ground station code, orbit circle number, product type, data archiving time, online data path, file size, online data status, nearline cluster archiving time, nearline cluster data path, nearline cluster data status, and storage object number.

3. The hierarchical storage management method based on the value of land observation satellite data according to claim 1, characterized in that, The initial comprehensive weight score for each scene of data is calculated based on the data value parameter table, including: Initial comprehensive weight score = Original cloud amount value score × First weight coefficient + Data quality value score × Second weight coefficient + Product level value score × Third weight coefficient + Satellite image resolution value score × Fourth weight coefficient + Data migration count value score × Fifth weight coefficient; where the first weight coefficient + second weight coefficient + third weight coefficient + fourth weight coefficient + fifth weight coefficient = 1, and the values ​​of different weight coefficients are determined according to user needs.

4. The hierarchical storage management method based on the value of land observation satellite data according to claim 1, characterized in that, For each satellite, a corresponding data archiving topic is created on the Kafka cluster, and data archiving is implemented using a multi-threaded parallel approach.

5. The hierarchical storage management method based on the value of land observation satellite data according to claim 1, characterized in that, Retrieve user order messages, including: Product order or production order messages generated by the distribution service subsystem are published to a specified topic in the Kafka cluster. The data management subsystem retrieves user order messages from the specified topic in real time.

6. The hierarchical storage management method based on the value of land observation satellite data according to claim 1, characterized in that, When the land observation satellite is a radar satellite, the score corresponding to the original cloud cover value and the first weight coefficient are removed when calculating the comprehensive weight score in steps 2 and 5.

7. The hierarchical storage management method based on the value of land observation satellite data according to claim 1, characterized in that, Also includes: When database access is required, the encrypted username and password in the configuration file are read, decrypted, and then the database is connected to read the data path of the level 0 or product data entity to be archived in the storage table.

8. The hierarchical storage management method based on the value of land observation satellite data according to claim 7, characterized in that, Before reading the encrypted username and password from the configuration file, the process also includes: securely encrypting the username and password, specifically including: The original key is randomly divided according to rand(1, n), and the random division is randomly filtered according to rand(1, rand(1, n)) to obtain m1 divisions, where n represents a random number from 2 to 9; Combine the first and last fields of each segment, and determine the filtering coordinates of the combined fields from the combination-coordinate filtering lookup table; Map the first dynamic change of the username and the second dynamic change of the password of the same user to a unified coordinate system each time the username is modified; The permission element attribute set is determined based on the user's user permissions, and a first number of irrelevant elements and a second number of relevant elements are extracted from all permission element attribute sets based on the segment attributes of the segment division. Determine whether the filtering coordinates need to be expanded based on the incoherent and coherent elements. If not required, determine a two-dimensional representation based on the selected coordinates in the unified coordinate system; If necessary, a two-dimensional representation based on extended coordinates is determined in the unified coordinate system; The corresponding segments are optimized according to the two-dimensional expression to obtain the security key.

Images