Space observation data dynamic storage method and system

Abstract

This application provides a method and system for dynamic storage of space observation data, belonging to the field of dynamic data storage technology. The method involves writing data files into the original data layer; obtaining clock correction values ​​for each data source identifier at the corresponding observation time and writing these correction values ​​into the metadata log layer; when a spatiotemporal range retrieval request is received, reading the matching clock correction values ​​from the metadata log layer to obtain the corrected observation time; performing spatiotemporal range matching based on the corrected observation time and the calibrated spatial coverage to filter out data files that meet the retrieval request; and appending the new clock correction value and its version identifier to the metadata log layer when an updated clock correction value is obtained for the same data source identifier. Using this solution, continuous incremental updates of clock correction values ​​can be supported while maintaining the immutable storage of the original space observation data, ensuring the consistency and accuracy of spatiotemporal matching results.

Description

Technical Field

[0001] This application relates to the field of dynamic data storage technology, and more specifically, to a method and system for dynamic storage of space observation data. Background Technology

[0002] Dynamic data storage refers to the process of transforming static, hot, and heterogeneous data sets that reside in fixed media into a hierarchical data service form that can flow flexibly according to access frequency, latency requirements, and cost constraints through a series of adaptive management processes such as real-time data heat monitoring and storage hierarchy mapping. This process enables the transformation of information assets from passively fixed storage to actively supplying storage on demand.

[0003] In the existing dynamic storage of space observation data, the updated precise clock correction is usually applied by overwriting the time stamps or reprocessing the original data files after the fact. This operation mode destroys the integrity and immutability of the original observation data. Moreover, in the scenario of multi-source observation data fusion, the dynamic update frequency and version evolution path of the clock corrections from different data sources are different. This leads to inconsistent spatiotemporal alignment results when the same original data is spatiotemporally retrieved at different times due to the different clock correction versions used. Therefore, how to support the continuous incremental update of the clock correction after the fact while maintaining the immutable storage of the original space observation data has become a technical challenge for the industry. Summary of the Invention

[0004] This application provides a method and system for dynamic storage of space observation data, which can support continuous incrementing of post-hoc clock corrections while maintaining the immutable storage of the original space observation data.

[0005] In a first aspect, this application provides a method for dynamically storing space observation data, comprising the following steps: The system receives data files from multiple space observation sources and writes these data files into the raw data layer of the storage system. The data files contain initial observation time stamps and data source identifiers. Obtain the clock correction for each data source identifier at the corresponding observation time. The clock correction includes clock model parameters. Write the clock correction into the metadata log layer of the storage system and keep the initial observation time label in the original data layer unchanged. When a spatiotemporal range retrieval request is received, the matching clock correction amount is read from the metadata log layer, and the clock model parameters therein are used to correct the sampling point time labels of the corresponding data files in the original data layer point by point to obtain the corrected observation time. Based on the corrected observation time, the data files in the original data layer are matched in time and space range to filter out the data files that meet the search request, and a version identifier of the applied clock correction is attached when the list is returned. When an updated clock correction is obtained for the same data source identifier, the new clock correction and its version identifier are appended to the metadata log layer.

[0006] In some embodiments, clock corrections for each data source identifier at the corresponding observation time are obtained, wherein the clock corrections include clock model parameters, specifically including: Based on the data source identifier, query the preset space observation platform clock error service directory to determine the clock error data acquisition interface and clock error data release frequency corresponding to the data source identifier; Based on the observation time indicated by the initial observation time tag, the clock difference data acquisition interface request is obtained and the effective clock difference correction information covering the observation time is extracted. The effective clock difference correction information includes clock difference polynomial model coefficients or clock difference ephemeris sampling point sequence. The effective clock error correction information is converted into clock error model parameters suitable for unified calculation of time base, which are used as the clock error correction amount for each data source at the corresponding observation time.

[0007] In some embodiments, writing the clock correction amount into the metadata log layer of the storage system while keeping the initial observation time stamp in the original data layer unchanged specifically includes: Construct a metadata log record that includes the data source identifier, the timestamp range of the initial observation time label, and the clock correction amount; The metadata log records are submitted to the storage system's metadata log layer, which is independent of the original data layer, in an append-only write mode, and a logical association index is established between the metadata log records and the corresponding immutable data objects in the original data layer. When responding to a time retrieval request for the initial observation timestamp, the clock correction amount in the metadata log layer is read through the logical association index and the corrected observation time is dynamically calculated, while the initial observation timestamp stored in the original data layer remains unchanged.

[0008] In some embodiments, when a spatiotemporal range retrieval request is received, reading the matching clock correction from the metadata log layer specifically includes: Extract the target time range constraints from the spatiotemporal range retrieval request, and determine the retrieval period covered by the target time range constraints; In the metadata log layer, metadata log records whose observation time range intersects with the retrieval period are selected as candidate metadata log records; Clock error model parameters are extracted from the candidate metadata log records, and the extracted clock error model parameters are determined as clock error correction amounts that match the corresponding data files.

[0009] In some embodiments, the clock difference model parameters are used to correct the sampling point time labels of the corresponding data files in the original data layer point by point, and the corrected observation time specifically includes: Based on the associations recorded in the candidate metadata log, locate the corresponding immutable data object in the original data layer, and read the sampling point time tag sequence within the immutable data object; For each sampling point in the time-stamped sequence of the sampling points, the clock error compensation value is calculated using the clock error model parameters; The corrected observation time of the data file is generated by subtracting the corresponding clock error compensation value from the initial time label of each sampling point.

[0010] In some embodiments, performing spatiotemporal range matching on data files in the original data layer based on the corrected observation time to filter out data files that meet the retrieval request specifically includes: The corrected observation time is matched with the target time range constraint in the spatiotemporal range retrieval request to determine whether the data file meets the retrieval request in the time dimension. For data files that meet the retrieval request in the time dimension, the spatial coverage information of the original records of the data files is synchronously calibrated based on the corrected observation time and the orbit and attitude metadata pre-archived by the corresponding data source to obtain the corrected spatial coverage range; combined with the target spatial range constraint conditions in the spatiotemporal range retrieval request, the corrected spatial coverage range is used to perform spatial matching determination. Data files that simultaneously meet both time and spatial dimension matching requirements are identified as data files that satisfy the retrieval request.

[0011] In some embodiments, when an updated clock correction is obtained for the same data source identifier, writing the new clock correction and its version identifier to the metadata log layer in an append-only manner specifically includes: Obtain a new clock error correction amount corresponding to the same data source identifier, and generate a new version identifier for the new clock error correction amount that is different from the existing version identifier; The new clock error correction is used to convert into new clock error model parameters, and a new metadata log record is constructed. The new metadata log record is submitted to the metadata log layer in an append-only manner.

[0012] In some embodiments, the initial observation timestamp refers to the original timestamp recorded by the space observation platform in the header of the data file, which is embedded in the data file by relying on its onboard clock or local frequency source when collecting observation data.

[0013] In some embodiments, the data source identifier refers to a globally unique encoded string consisting of one or more of the following: space observation platform number, payload number, beam number, or channel number.

[0014] Secondly, this application provides a dynamic storage system for space observation data, comprising: The receiving module is used to receive data files from multiple space observation sources and write the data files into the raw data layer of the storage system. The data files include initial observation time stamps and data source identifiers. The processing module is used to obtain the clock error correction amount for each data source identifier at the corresponding observation time. The clock error correction amount includes clock error model parameters. The clock error correction amount is written into the metadata log layer of the storage system, and the initial observation time label in the original data layer remains unchanged. The processing module is also used to read the matching clock correction amount in the metadata log layer when a spatiotemporal range retrieval request is received, and use the clock error model parameters therein to correct the sampling point time label of the corresponding data file in the original data layer point by point to obtain the corrected observation time. The processing module is also used to perform spatiotemporal range matching on the data files in the original data layer based on the corrected observation time, filter out the data files that meet the retrieval request, and attach the version identifier of the applied clock correction amount when returning the list; The execution module is used to append the new clock correction and its version identifier to the metadata log layer when an updated clock correction is obtained for the same data source identifier.

[0015] The technical solutions provided by the embodiments disclosed in this application have the following beneficial effects: The spatial observation data dynamic storage method and system provided in this application firstly, by writing the received spatial observation source data file into the original data layer of the storage system while keeping its initial observation time label unchanged, the original observation data and its subsequently generated time correction information are physically isolated and logically decoupled. This transforms the invasive overwriting risk of traditional post-correction to the original data into an independent append operation to the metadata log layer, providing a solid data foundation for the dynamic evolution of clock correction in an immutable storage architecture. Subsequently, upon receiving a spatiotemporal range retrieval request, the matching clock correction is read from the metadata log layer. The clock correction model parameters are used to correct the sampling point time labels of the corresponding data files in the original data layer point by point to obtain the corrected observation time. Based on the corrected observation time, the spatial coverage of the data is synchronously calibrated. This process integrates the static original time label, original spatial parameters, and dynamic correction model parameters in real time during the retrieval calculation, rather than pre-fixing them in the data file. This allows the same original data to generate accurate and consistent spatiotemporal alignment results at different time points based on different clock correction versions, while completely avoiding the original data overwriting risk. The initial observation addresses the issues of information loss and redundant storage across multiple versions. Furthermore, by attaching a version identifier of the applied clock correction to the returned search results list, the version information of the correction used in each search calculation is explicitly anchored in the search output. This transforms the need for retrospective retrieval of search conditions from vague human recollection or external document records into auditable metadata credentials carried within the search results themselves, providing precise version tracing evidence for reproducing search results across time and versions. Moreover, when an updated clock correction is obtained for the same data source identifier, the new correction is appended to the metadata log layer. The clock correction and its version identifier are used to transform the update behavior of the clock correction from modifying existing data to appending only to the log sequence. This ensures that the version evolution path of the clock correction is completely and immutably recorded within the storage system, providing the ability to correct the time based on any historical version correction for subsequent retrieval requests. At the same time, it avoids the data rewriting input / output overhead and data consistency maintenance complexity caused by version updates. In summary, this scheme can support continuous incremental updates of the clock correction after the fact while maintaining the immutable storage of the original spatial observation data. Attached Figure Description

[0016] Figure 1 This is a flowchart illustrating a method for dynamically storing space observation data according to some embodiments of this application; Figure 2 This is a schematic flowchart illustrating the process of obtaining clock correction amounts according to some embodiments of this application; Figure 3This is a flowchart illustrating the process of determining the retrieval period according to some embodiments of this application; Figure 4 This is a schematic diagram of the structure of a dynamic storage system for space observation data according to some embodiments of this application; Figure 5 This is an internal structural diagram of a computer device that implements a method for dynamically storing space observation data according to some embodiments of this application. Detailed Implementation

[0017] To better understand the technical solutions in this embodiment, the technical solutions in this embodiment will be described in detail below with reference to the accompanying drawings and specific implementation methods.

[0018] refer to Figure 1 The figure is a flowchart illustrating a method for dynamically storing space observation data according to some embodiments of this application. The method for dynamically storing space observation data mainly includes the following steps: In step 101, data files from multiple space observation sources are received and written into the raw data layer of the storage system. The data files contain initial observation time stamps and data source identifiers.

[0019] In practice, raw observation data files transmitted from different space observation platforms can be continuously received through standardized data interfaces deployed at ground receiving stations or data relay satellites. These space observation platforms include Earth observation remote sensing satellites, astronomical observation satellites, space science detectors, and ground-based space target monitoring equipment. The data files they generate generally adopt scientific data formats defined by the International Astronomical Union or international remote sensing standards organizations, such as the Flexible Image Transmission System format commonly used in astronomy or the geotagged image file format commonly used in remote sensing. During the reception process, frame synchronization and integrity checks are first performed on the arriving data stream. The initial observation time stamp field and data source identifier field embedded in the file are located and parsed based on the format identifier keywords in the file header. The initial observation time stamp is used to record the first sample of the data file. The observation time, as indicated by the onboard clock or the detector's local clock, is typically stored in the numerical field corresponding to the date observation keyword, time system keyword, or time reference keyword in the file header. The data source identifier is used to uniquely identify the specific space observation payload that generated the data file and can be extracted based on a combination of the satellite's international identification number and the sensor identifier. Subsequently, the verified and parsed original data file, along with its extracted initial observation time tag and data source identifier, is encapsulated into an immutable data object and written to the original data layer logical storage bucket pre-configured by the storage system via an object storage protocol. The write operation employs a compliance retention strategy or object locking mechanism that prohibits modification after a one-time write, ensuring that the data file content, initial observation time tag, and data source identifier are all in an unchangeable state after being written to the original data layer.

[0020] It should be noted that the raw data layer mentioned in this application refers to a logical storage partition or bucket in the storage system specifically used to store raw space observation data files that have not undergone any post-time base correction and are persisted in the form of immutable objects. Its underlying structure can be composed of a distributed object storage engine or a log-structured file system that supports write-once-read-many semantics. The purpose is to provide a unique and constant raw data anchor point for all subsequent time alignment operations based on different post-time clock versions. The initial observation timestamp refers to the original timestamp recorded by the space observation platform when collecting observation data, relying on its onboard clock or local frequency source and embedded in the header of the data file. This timestamp is affected by factors such as frequency drift, aging, and relativistic effects of the onboard clock. There is an inherent deviation between the data source and the ground-based high-precision time reference. The data source identifier is a globally unique coded string composed of one or more of the following: space observation platform number, payload number, beam number, or channel number. It is used to accurately distinguish and index the source of observation data in cross-platform, multi-payload joint spatiotemporal retrieval scenarios. The process of writing the data file into the original data layer is not a simple format conversion or data copying. Instead, it delivers the complete original observation data file containing the initial observation time label and the data source identifier as an immutable whole to the bottom layer of the layered storage architecture of the storage system. This provides an invariant basis for the independent incremental appending of clock correction in the subsequent metadata log layer and the dynamic time correction during the retrieval stage.

[0021] In step 102, the clock correction amount for each data source identifier at the corresponding observation time is obtained. The clock correction amount includes clock model parameters. The clock correction amount is written into the metadata log layer of the storage system, while keeping the initial observation time label in the original data layer unchanged.

[0022] In some embodiments, reference Figure 2 As shown in the figure, this is a schematic diagram of the process for obtaining clock error corrections according to some embodiments of this application. Obtaining the clock error correction for each data source identifier at the corresponding observation time, wherein the clock error correction includes clock error model parameters, can be achieved by the following steps: First, in step 1021, the preset space observation platform clock error service directory is queried according to the data source identifier to determine the clock error data acquisition interface and clock error data release frequency corresponding to the data source identifier; Then, in step 1022, based on the observation time indicated by the initial observation time tag, the clock difference data acquisition interface request is obtained from the clock difference data and the effective clock difference correction information covering the observation time is extracted. The effective clock difference correction information includes clock difference polynomial model coefficients or clock difference ephemeris sampling point sequence. Finally, in step 1023, the effective clock error correction information is converted into clock error model parameters suitable for unified calculation of time base, which are used as the clock error correction amount of each data source at the corresponding observation time.

[0023] In specific implementation, the clock error service directory of the space observation platform is queried according to the data source identifier to determine the clock error data acquisition interface and clock error data release frequency corresponding to the data source identifier. This can be achieved in the following way: For example, a space observation platform clock error service directory table is pre-built in the metadata management module of the storage system. This directory table uses the space observation platform number field in the data source identifier as the primary index key, and stores the name of the clock error data release organization associated with the platform, the network address of the clock error data service endpoint, the clock error product type identifier, and the routine release cycle parameters. When the received data source identifier enters this step, the platform number field is parsed first, and the key-value matching query of the space observation platform clock error service directory table is performed using the platform number. If the query is successful, the corresponding clock error data acquisition interface address and release cycle parameters are read, and the clock error data acquisition interface address is used as the clock error data acquisition interface for this time, and the release cycle parameters are used as the clock error data release frequency for this time. If the query does not find a match, the clock error service is returned to the storage system management plane as unregistered and the current clock error correction acquisition process is terminated.

[0024] It should be noted that the clock bias data acquisition interface and clock bias data release frequency mentioned in this application refer to the network service endpoint address and its corresponding data update cycle parameters used to establish a connection with the clock bias data release service of the external space observation platform and obtain clock bias product data, which are used to provide the storage system with a channel and opportunity to obtain post-event precise clock bias correction information.

[0025] In specific implementation, based on the observation time indicated by the initial observation time stamp, the effective clock error correction information covering that observation time is requested and extracted from the clock error data acquisition interface. This effective clock error correction information includes clock error polynomial model coefficients or a clock error epoch sampling point sequence. This can be implemented in the following ways: for example, a data acquisition request is sent to the determined clock error data acquisition interface, the request message containing the target epoch parameters converted from the initial observation time stamp and the desired data return format identifier; in the clock error product file returned by the clock error data acquisition interface, the clock error correction information is organized in the form of an epoch sampling point list or a piecewise polynomial coefficient set; for cases where the returned content is an epoch sampling point list, the list is traversed to find the timestamp immediately adjacent to the initial observation time stamp. The first epoch sampling point before the initial observation time and the second epoch sampling point immediately after the initial observation time are used together as effective clock error correction information covering the observation time. For cases where the returned content is a set of piecewise polynomial coefficients, the constant term coefficient, linear term coefficient, and quadratic term coefficient corresponding to the effective time interval to which the initial observation time belongs are extracted from the set of piecewise polynomial coefficients, and this set of coefficients is used as effective clock error correction information covering the observation time. During the extraction process, the clock error product version identifier and issuing organization identifier corresponding to this request are simultaneously recorded as candidate components for generating subsequent version identifiers.

[0026] It should be noted that the effective clock error correction information mentioned in this application refers to the clock error description data extracted from the clock error product file returned from the external clock error service interface, which can cover the observation time indicated by the initial observation time tag in the time dimension.

[0027] In specific implementation, the effective clock error correction information is converted into clock error model parameters suitable for unified calculation of time bases. The clock error correction amount for each data source at the corresponding observation time can be implemented in the following way: For example, when the extracted effective clock error correction information consists of the first epoch sampling point and the second epoch sampling point and their respective clock error values, the timestamp of the first epoch sampling point is used as the model reference origin. The clock error difference and time interval between the second epoch sampling point and the first epoch sampling point are calculated. The reciprocal of this time interval is used as the first-order rate of change. A linear interpolation model is constructed with the offset of the time variable relative to the model reference origin as input and the clock error estimate as output. The clock error value of the first epoch sampling point is used as a constant term parameter, and the first-order rate of change is used as a first-order term parameter. If the space observation platform is marked as a precise time-frequency reference type in the clock error service catalog, multiple adjacent epoch sampling points before and after the first epoch sampling point are further extracted, and a quadratic time-frequency reference is generated through piecewise parabolic fitting. The parameters are ultimately used as the constant term parameters, linear term parameters, and quadratic term parameters as the transformed clock error model parameters. When the extracted effective clock error correction information is a set of piecewise polynomial coefficients, the constant term coefficients, linear term coefficients, and quadratic term coefficients in the set are directly used as the clock error model parameters, and the effective start time corresponding to the piecewise polynomial is used as the model reference origin. The transformed clock error model parameters, together with the data source identifier and the initial observation time tag, are encapsulated into a clock error correction record, which is used as the clock error correction amount of the data source identifier at the corresponding observation time. As a preferred embodiment, when encapsulating the clock error correction record, the clock error product version identifier on which this transformation is based can be written at the same time so that the clock error correction version used can be accurately traced during subsequent retrieval. In other embodiments, for application scenarios that only require coarse-grained time alignment, the clock error model parameters may only include constant term parameters and linear term parameters to reduce the storage overhead of the metadata log layer. This application does not limit this.

[0028] It should be noted that the clock error model parameters mentioned in this application refer to a set of mathematical descriptive coefficients used to calculate clock error compensation values ​​with time variables as input.

[0029] In some embodiments, writing the clock correction amount into the metadata log layer of the storage system while keeping the initial observation timestamps in the original data layer unchanged can be achieved by the following steps: Construct a metadata log record that includes the data source identifier, the timestamp range of the initial observation time label, and the clock correction amount; The metadata log records are submitted to the storage system's metadata log layer, which is independent of the original data layer, in an append-only write mode, and a logical association index is established between the metadata log records and the corresponding immutable data objects in the original data layer. When responding to a time retrieval request for the initial observation timestamp, the clock correction amount in the metadata log layer is read through the logical association index and the corrected observation time is dynamically calculated, while the initial observation timestamp stored in the original data layer remains unchanged.

[0030] In specific implementation, constructing a metadata log record containing the data source identifier, the timestamp range of the initial observation timestamp, and the clock correction can be achieved in the following way: First, determine the object unique identifier of the original data file associated with this write operation in the original data layer. This object unique identifier has been allocated and returned by the storage system when the data file is encapsulated as an immutable data object. Then, extract the space observation platform number and payload number fields from the data source identifier, and extract the observation start time and observation end time from the initial observation timestamp. The observation start time is the initial observation timestamp itself, and the observation end time can be calculated based on the number of sampling points and the sampling interval field in the data file header. The initial observation time stamp range is defined; the space observation platform number, the payload number, the observation start time, the observation end time, the object unique identifier, and the clock difference model parameters obtained in the preceding steps are used as the field values ​​of the metadata log record, and assembled according to a preset metadata log record structure. The metadata log record structure sequentially includes a data source identifier field, an observation time range start field, an observation time range end field, an associated object identifier field, a clock difference model parameter field, and a record write time field; during the assembly process, the clock difference model parameters are serialized into fixed-length or variable-length byte strings and filled into the clock difference model parameter field, and the wall clock time of the current storage system is written as the value of the record write time field.

[0031] It should be noted that the clock correction amount mentioned in this application refers to a data unit that is identified by a specific data source at a specific observation time, and encapsulates the clock error model parameters and their associated version information. It is used as a basic object for independent storage and management in the metadata log layer, thereby providing an accurate and traceable correction basis for dynamic time correction during retrieval. The metadata log record refers to a structured data entry written in an append manner in the metadata log layer of the storage system, which is used to completely record the identifier of a data file, the observation time range, the associated clock error model parameters and their version information.

[0032] In specific implementation, the metadata log records are submitted to the metadata log layer of the storage system, which is independent of the original data layer, in an append-only write mode. The logical association index between the metadata log records and the corresponding immutable data objects in the original data layer can be established in the following way: For example, the storage system allocates an independent log structured storage area for the metadata log layer on the physical storage medium. This storage area is isolated from the object storage bucket where the original data layer is located in terms of namespace and input / output paths. After the metadata log record is assembled, it is appended as a new log entry to the end of the log stream file named after the data source identifier. The log stream file is stored in the metadata log layer partitioned by data source identifier, with each data source identifier corresponding to one or more log segment files. The writing operation of the new log entry only involves moving the tail pointer of the currently active log segment file forward and counting... According to the persistence method, it does not involve overwriting or deleting existing log entries. Simultaneously with the append write completion, the storage system, based on the object unique identifier carried in the metadata log record, establishes an index entry in the metadata index structure pointing from the object unique identifier to the physical offset position of the log record in the log stream file. This metadata index structure can be implemented using a key-value storage engine, with the object unique identifier as the key and the log stream file identifier and offset range as the value. This index entry is the logical association index between the metadata log record and the corresponding immutable data object in the original data layer. Through this logical association index, the corresponding data object can be quickly located during retrieval. In other embodiments, the logical association index can also be directly embedded in the extended attributes of the original data layer object, implemented by recording a pointer to the log stream offset in the object metadata; this application does not limit this approach.

[0033] In specific implementation, when responding to a time retrieval request for the initial observation time tag, the clock correction amount in the metadata log layer is read through the logical association index, and the corrected observation time is dynamically calculated. The initial observation time tag stored in the original data layer remains unchanged. This can be achieved in the following way: when the storage system receives a time retrieval request from a user or upper-layer application, it first parses the target time range and target data source identifier contained in the time retrieval request; it determines the corresponding log stream file of the metadata log layer based on the target data source identifier, and searches for a list of associated object identifiers in the metadata index structure based on the target time range. The search process prioritizes range scanning using a secondary index sorted by the observation time range, filtering out a set of candidate object unique identifiers where the observation time range intersects with the target time range; for each candidate object unique identifier in the set... Each object has a unique identifier. The corresponding clock error correction record is read from the metadata log layer through the logical association index, and the clock error model parameters are extracted. Then, without reading or modifying the content of the corresponding data file in the original data layer, the clock error model parameters are used to perform point-by-point dynamic correction calculation on the sampling point time stamps of the data file. That is, for each sampling point of the data file, the clock error compensation value is calculated based on its time offset relative to the initial observation time stamp and the clock error model parameters. The clock error compensation value is subtracted from the initial observation time stamp to obtain the corrected observation time. Then, it is determined whether the corrected observation time falls within the target time range. If so, the data file is included in the search results list. Throughout the dynamic calculation process, the initial observation time stamp stored in the original data layer always maintains its original value when it was written, without any overwriting or modification, and is only used as the basic input for dynamic correction calculation.

[0034] It should be noted that the logical association index described in this application refers to an index mapping structure in the storage system used to associate immutable data objects in the original data layer with corresponding clock correction records in the metadata log layer, which is used to quickly locate and read the corresponding clock correction based on the data object identifier during retrieval.

[0035] In step 103, when a spatiotemporal range retrieval request is received, the matching clock correction amount in the metadata log layer is read, and the clock model parameters therein are used to correct the sampling point time labels of the corresponding data files in the original data layer point by point to obtain the corrected observation time.

[0036] In some embodiments, when a spatiotemporal range retrieval request is received, reading the matching clock correction from the metadata log layer can be achieved using the following steps: Extract the target time range constraints from the spatiotemporal range retrieval request, and determine the retrieval period covered by the target time range constraints; In the metadata log layer, metadata log records whose observation time range intersects with the retrieval period are selected as candidate metadata log records; Clock error model parameters are extracted from the candidate metadata log records, and the extracted clock error model parameters are determined as clock error correction amounts that match the corresponding data files.

[0037] For specific implementation, refer to Figure 3 As shown in the figure, this is a flowchart illustrating the process of determining the retrieval period in some embodiments of this application. Extracting the target time range constraints from the spatiotemporal range retrieval request and determining the retrieval period covered by the target time range constraints can be achieved in the following way: For example, when the storage system receives the spatiotemporal range retrieval request, it first performs syntax parsing on the retrieval request, separating the spatial condition field and the time condition field. The time condition field includes the target observation start time and target observation end time specified by the user. If the user only specifies a single time, that single time is treated as both the start and end time. The closed interval or semi-open interval defined by the target observation start time and target observation end time is determined as the retrieval period covered by the target time range constraints. This retrieval period is then standardized using a time base format uniformly adopted within the storage system, so that subsequent interval intersection operations can be performed with the observation time range recorded in the metadata log layer.

[0038] It should be noted that the target time range constraint mentioned in this application refers to the time interval limit specified by the user through the spatiotemporal range retrieval request, which is used to define the effective boundary of this retrieval in the time dimension; the retrieval period refers to a specific time interval under the unified time base inside the storage system obtained after parsing and standardizing the target time range constraint, which is used to perform efficient interval intersection operation with the observation time range of each record in the metadata log layer.

[0039] In specific implementation, in the metadata log layer, metadata log records whose observation time range intersects with the retrieval period are selected as candidate metadata log records. This can be achieved in the following way: for example, based on the target data source identifier carried in the spatiotemporal range retrieval request, locate the log stream file or log partition in the metadata log layer corresponding to the target data source identifier; if the target data source identifier is empty or contains a wildcard, perform multi-path concurrent scanning on all log stream files in the metadata log layer; during the scanning process, for each metadata log record, read the data from its observation time range start field and observation time range end field. The data file associated with the metadata log record is determined to match the retrieval request in the time dimension if the value of the starting field of the observation time range is less than or equal to the end time of the retrieval period, and the value of the ending field of the observation time range is greater than or equal to the starting time of the retrieval period. The metadata log record is then marked as a candidate metadata log record and added to the candidate set. In other embodiments, the intersection determination can also be extended to a weighted matching that considers the overlap length of the time range, using the overlap duration as the sorting criterion. This application does not limit this.

[0040] It should be noted that the candidate metadata log records mentioned in this application refer to metadata log entries that have undergone preliminary screening in the metadata log layer, and whose observation time range overlaps with the search period, thus being considered to have potential matching possibilities in the time dimension.

[0041] In specific implementation, extracting clock bias model parameters from the candidate metadata log records and determining the extracted clock bias model parameters as clock bias corrections matching the corresponding data files can be achieved in the following way: For each record in the candidate metadata log record set, based on the storage offset or record identifier of the record in the log stream file, the complete content of the record is read from the persistent storage medium of the metadata log layer; the clock bias model parameter field is parsed from the read complete content. The byte string stored in this field has been serialized into structured data containing constant term parameters, linear term parameters, and optional quadratic term parameters. The byte string is deserialized to restore the clock bias model parameters that can be directly used in the calculation; subsequently, a memory mapping relationship is established between the deserialized clock bias model parameters and the corresponding data source identifier, observation time range, and associated object unique identifier, and this mapping relationship is cached as a clock bias correction matching the data file pointed to by the object unique identifier.

[0042] In some embodiments, the clock error model parameters are used to correct the sampling point time stamps of the corresponding data files in the original data layer point by point to obtain the corrected observation time. This can be achieved by the following steps: Based on the associations recorded in the candidate metadata log, locate the corresponding immutable data object in the original data layer, and read the sampling point time tag sequence within the immutable data object; For each sampling point in the time-stamped sequence of the sampling points, the clock error compensation value is calculated using the clock error model parameters; The corrected observation time of the data file is generated by subtracting the corresponding clock error compensation value from the initial time label of each sampling point.

[0043] In specific implementation, locating the corresponding immutable data object in the original data layer based on the association relationships recorded in the candidate metadata log records and reading the sampling point time stamp sequence within the immutable data object can be achieved in the following way: For each record in the candidate metadata log record set, extract the object unique identifier from its associated object identifier field. This object unique identifier has been allocated by the storage system and bound to the data file when the data file is encapsulated as an immutable data object. Use the object unique identifier to initiate an object read request to the object storage bucket of the original data layer. The read request locates the corresponding immutable data object based on the object unique identifier. Based on the physical storage location of the object in the underlying storage medium, when reading the content of the immutable data object, first obtain its file header information, parse the sampling point number field and sampling interval field from the header information, and determine the total number of sampling points contained in the data file based on the sampling point number field; then, based on the initial observation time tag and the sampling interval field, calculate the initial time tag corresponding to each sampling point in sequence. The calculation method is to start from the initial observation time tag, expand the arithmetic sequence with the value of the sampling interval field as the step size, thereby generating a sequence containing the initial time tags of all sampling points, and read this sequence as the sampling point time tag sequence.

[0044] It should be noted that the immutable data object mentioned in this application refers to a data file unit stored in the original data layer in a one-time write and permanent read-only manner, which is used to ensure that the space observation data and its original time label will not be overwritten or modified by any subsequent operation after being written; the sampling point time label sequence refers to an ordered set containing the initial observation time labels of all sampling points in the data file, which is extracted from the immutable data object or calculated based on its header information.

[0045] In specific implementation, for each sampling point in the sampling point time label sequence, the clock error compensation value calculated using the clock error model parameters can be achieved in the following way: For example, the clock error model parameters corresponding to the current data file are obtained in memory. These clock error model parameters include constant term parameters, linear term parameters, and optional quadratic term parameters. Simultaneously, the model reference origin associated with the clock error model parameters is obtained. For each sampling point in the sampling point time label sequence, the time offset of the initial time label of that sampling point relative to the model reference origin is first calculated. This time offset is the time difference between the initial time label of that sampling point and the model reference origin. Subsequently, the clock error compensation is calculated using the clock error model parameters. Specifically, the time offset is substituted into the constant term parameters, linear term parameters, and quadratic term parameters... In the polynomial mapping relationship jointly defined by the constant term parameter and the quadratic term parameter, the constant term parameter is first used as the baseline compensation amount. Then, the linear term parameter is multiplied by the time offset to obtain the linear compensation component. If a quadratic term parameter exists, the quadratic term parameter is multiplied by the square of the time offset to obtain the quadratic compensation component. The baseline compensation amount, the linear compensation component, and the optional quadratic compensation component are accumulated, and the accumulated result is the clock error compensation value corresponding to the sampling point. Preferably, if the clock error model parameters only include the constant term parameter and the linear term parameter, then only the sum of the baseline compensation amount and the linear compensation component is calculated as the clock error compensation value. In the entire calculation process, for all sampling points in the same data file, the clock error model parameters remain unchanged, and only the time offset changes with different sampling points, thereby realizing point-by-point independent correction for each sampling point.

[0046] It should be noted that the clock error compensation value mentioned in this application refers to a time correction value calculated for a single sampling point in the data file using its corresponding clock error model parameters and the time offset of the sampling point, which is used to correct the initial observation time label of the sampling point.

[0047] In practice, the corrected observation time of the data file is generated by subtracting the corresponding clock error compensation value from the initial time label of each sampling point. This can be achieved in the following way: For each sampling point in the time label sequence, after calculating the clock error compensation value of that sampling point, the value of the initial time label of that sampling point is subtracted from the value of the clock error compensation value. The result of the subtraction operation is the corrected observation time of that sampling point after time base correction. The above subtraction operation is performed on all sampling points in the data file in their original order in the original data file to obtain a set that is one-to-one with the original sampling points. The corrected observation times are then arranged in the original sampling point order to form the corrected observation times of the data file. It should be noted that the corrected observation times fully reflect the actual observation times of each sampling point in the data file under the high-precision ground time reference after the clock error correction is applied. The process of generating the corrected observation times is completed only in the memory or temporary computing space of the storage system, without involving any write-back or update operations on the immutable data objects in the original data layer, thereby ensuring that the data content in the original data layer always maintains the original state of the initial observation time stamp.

[0048] It should be noted that the corrected observation time mentioned in this application refers to a new ordered set of time points generated by subtracting the corresponding clock error compensation value from the initial time labels of all sampling points in the data file. This set is used to characterize the actual observation time sequence of the data file under a high-precision ground time reference after the data file has undergone post-hoc precise clock error correction.

[0049] In step 104, the data files in the original data layer are matched for spatiotemporal range based on the corrected observation time, and the data files that meet the retrieval request are selected. The version identifier of the applied clock correction is attached when the list is returned.

[0050] In some embodiments, the spatiotemporal range matching of data files in the original data layer based on the corrected observation time, and the selection of data files that meet the retrieval request, can be achieved by the following steps: The corrected observation time is matched with the target time range constraint in the spatiotemporal range retrieval request to determine whether the data file meets the retrieval request in the time dimension. For data files that meet the retrieval request in the time dimension, the spatial coverage information of the original records of the data files is synchronously calibrated based on the corrected observation time and the orbit and attitude metadata pre-archived by the corresponding data source to obtain the corrected spatial coverage range; combined with the target spatial range constraint conditions in the spatiotemporal range retrieval request, the corrected spatial coverage range is used to perform spatial matching determination. Data files that simultaneously meet both time and spatial dimension matching requirements are identified as data files that satisfy the retrieval request.

[0051] In specific implementation, matching the corrected observation time with the target time range constraint in the spatiotemporal range retrieval request to determine whether the data file meets the retrieval request in the time dimension can be achieved in the following way: For example, for a data file that has completed point-by-point correction calculation, obtain the first corrected observation time and the last corrected observation time in its corrected observation time, and use them as the effective coverage start time and effective coverage end time of the data file in the time dimension, respectively; perform interval intersection judgment between the time interval formed by the effective coverage start time and the effective coverage end time and the retrieval period defined by the target time range constraint; if the effective coverage start time is less than or equal to the end time of the retrieval period and the effective coverage end time is greater than or equal to the start time of the retrieval period, then it is determined that the data file meets the retrieval request in the time dimension; otherwise, it is determined that the data file does not meet the retrieval request in the time dimension and is removed from the candidate data file set; in other embodiments, other methods can also be used, which are not limited here.

[0052] It should be noted that the search request mentioned in this application refers to a data query instruction initiated by a user or upper-layer application that includes constraints such as the target time range and the target spatial range.

[0053] In specific implementation, for data files that meet the retrieval request in the time dimension, the spatial coverage information of the data file is first synchronously calibrated in conjunction with the target spatial range constraints in the spatiotemporal range retrieval request. Spatial matching determination using the calibrated spatial coverage information can be achieved in the following ways: For example, the target spatial range constraints are parsed from the spatial condition field of the spatiotemporal range retrieval request. These constraints can be represented as a rectangular celestial region, a circular cone region, or a polygonal coverage area in the celestial coordinate system. Simultaneously, the original spatial coverage information of the data file is extracted from its file header or metadata description field. This original spatial coverage information includes a description of the coverage area of ​​the data file's observation field of view in space, such as the coordinates of the four corner boundaries, the center pointing coordinates combined with the field of view angle, or the vertex sequence of the polygonal coverage area. Based on the corrected observation time corresponding to the data file and the pre-archived orbit and attitude metadata of the corresponding observation platform, the extracted spatial coverage information is then analyzed. The original spatial coverage information is geometrically calibrated to obtain corrected spatial coverage information. The spatial region described by the corrected spatial coverage information is then compared with the spatial region described by the target spatial range constraint to determine geometric intersection or inclusion. Specifically, if the target spatial range constraint is a rectangular sky region and the corrected spatial coverage information consists of calibrated corner boundary coordinates, it is determined whether the two rectangular regions overlap. If the target spatial range constraint is a circular cone region and the corrected spatial coverage information consists of calibrated center pointing coordinates and the field of view angle, the sum of the center distance and radius of the two circular regions is compared to determine the intersection relationship. If the corrected spatial coverage information is a calibrated polygon vertex sequence, a polygon intersection detection algorithm is used to determine whether the two spatial regions intersect. If an intersection exists, the data file is determined to satisfy the retrieval request in the spatial dimension; if no intersection exists, the data file is determined not to satisfy the retrieval request in the spatial dimension.

[0054] It should be noted that the spatial matching determination described in this application refers to the process of determining whether there is an intersection between the spatial coverage information obtained by the corrected observation time synchronization calibration and the target spatial range constraints in the retrieval request through geometric calculation.

[0055] In specific implementation, determining data files that simultaneously satisfy both time and spatial dimension matching as data files satisfying the retrieval request can be achieved in the following way: For example, after sequentially completing the time dimension matching and spatial dimension matching determination, the storage system maintains a matching status flag for each data file. The matching status flag contains two Boolean values: time matching result and spatial matching result. For the currently processed data file, if its time matching result is satisfied and its spatial matching result is also satisfied, then the data file is marked as a hit data file, and its corresponding object unique identifier, data source identifier, and clock difference product version identifier extracted from the candidate metadata log record are added to the retrieval result set. If either the time matching result or the spatial matching result is not satisfied, then the data file is skipped and the next data file in the candidate set is processed. When all data files corresponding to records in the candidate metadata log record set have completed the above two-dimensional matching determination, all data files included in the retrieval result set are the finally determined data files that satisfy the retrieval request. In other embodiments, other methods can also be used, which are not limited here.

[0056] It should be noted that the data file that satisfies the retrieval request mentioned in this application refers to the hit data file whose calibrated time coverage and calibrated spatial coverage are confirmed to match the constraints of the user's retrieval request after time dimension matching and spatial dimension matching, and is used to form the final data list returned to the user.

[0057] As a preferred embodiment, attaching the version identifier of the applied clock correction amount when returning the list can be implemented in the following way, for example: after determining the data file that meets the search request as the hit data file and adding it to the search result set, traversing each hit data file in the search result set, and extracting the associated clock correction product version identifier from the previously cached clock correction amount mapping relationship according to the object unique identifier corresponding to the hit data file; combining the file name or object unique identifier of the hit data file with the clock correction product version identifier into a list record, the list record containing at least data The file identifier field and version identifier field, if the hit data file is also associated with a data source identifier, calibrated observation time range, or calibrated spatial coverage summary information, can be filled into the corresponding extended fields of the list record to enhance the usability of the returned results; the list records of all hit data files are summarized into a structured return list document, which can be organized using a key-value pair structure or a table structure, and the return list document is returned to the request initiator as the response payload of the spatiotemporal range retrieval request; in other embodiments, other methods can also be used to achieve this, which are not limited here.

[0058] It should be noted that the version identifier mentioned in this application refers to the tag information used to uniquely identify a specific version of clock correction, which is used to distinguish multiple clock correction versions obtained or calculated at different times for the same data source, so that when the search results are returned, users can clearly know the clock correction version on which the result depends, and realize the traceability and reproducibility of the results.

[0059] In step 105, when an updated clock correction is obtained for the same data source identifier, the new clock correction and its version identifier are written to the metadata log layer in an append-only manner.

[0060] In some embodiments, when an updated clock correction is obtained for the same data source identifier, writing the new clock correction and its version identifier to the metadata log layer in an append-only manner can be achieved using the following steps: Obtain a new clock error correction amount corresponding to the same data source identifier, and generate a new version identifier for the new clock error correction amount that is different from the existing version identifier; The new clock error correction is used to convert into new clock error model parameters, and a new metadata log record is constructed. The new metadata log record is submitted to the metadata log layer in an append-only manner.

[0061] In specific implementation, obtaining a new clock correction value corresponding to the same data source identifier and generating a new version identifier for the new clock correction value that is different from the existing version identifier can be achieved in the following way: For example, when it is detected that the clock product of a certain data source identifier has been updated and the effective time range of the new version product covers the time period to which the initial observation time tag of the archived data file belongs, the updated version of the clock product file is requested from the corresponding clock data acquisition interface; the effective clock correction information covering the initial observation time tag is extracted from the updated version of the clock product file in accordance with the method described in step 102, and the effective clock correction information is converted into new clock model parameters, thereby obtaining a new clock correction value corresponding to the same data source identifier. The updated clock product file is used to generate a new version identifier. Simultaneously, the version number or product generation timestamp is extracted from the metadata of the updated version of the clock product file. This version number or timestamp is then combined with the data source identifier and the current clock time in the storage system to generate a globally unique new version identifier that differs from existing version identifiers. If the updated version of the clock product file does not carry a specific version number, an incrementing sequence number is generated based on the product's release date and batch number as the new version identifier. In other embodiments, the new version identifier can also directly use the cryptographic hash value of the updated version of the clock product file as a unique identifier to ensure the tamper-proof and verifiable nature of the version identifier; this application does not limit this.

[0062] In specific implementation, the conversion of the new clock bias correction into new clock bias model parameters and the construction of a new metadata log record can be achieved in the following way: For example, the new clock bias model parameters are associated with the data source identifier, the timestamp range of the initial observation time tag, and the object unique identifier corresponding to the data file in the original data layer. A new metadata log record is assembled according to the preset structure of the metadata log record. The structure of the new metadata log record is consistent with the structure of the old version metadata log record written in step 102, and includes, in sequence, the data source identifier field, the observation time range start field, the observation time range end field, the associated object identifier field, the clock bias model parameter field, the record write time field, and the version identifier field. During the assembly process, the new clock bias model parameters are serialized into byte strings and filled into the clock bias model parameter field, the current storage system clock time is used as the value of the record write time field, and the new version identifier is filled into the version identifier field. The object unique identifier in the new metadata log record is the same as the object unique identifier in the old version metadata log record, thereby indicating that both the new and old log records point to the same immutable data object in the original data layer.

[0063] In specific implementation, submitting the new metadata log record to the metadata log layer via append writing can be achieved in the following way: After the new metadata log record is assembled and passes structure verification, the storage system appends the record as a new log entry to the end of the currently active log segment file of the log stream file named after the data source identifier; during the writing process, the storage system first serializes the content of the log entry into a byte stream, appends it to the end of the currently active log segment file, then updates the end pointer of the log segment file and the record count in the file metadata, and persists the write operation to the underlying storage medium to ensure data reliability; throughout the append writing process, the old metadata already existing in the log stream file is not updated. The log records retain their original storage location and content completely unchanged, without being overwritten, deleted, or modified, thus enabling the coexistence of old and new version clock corrections in the metadata log layer. Preferably, at the same time as the append write is completed, the storage system updates the metadata index structure, adding or updating an index entry pointing to the new log entry to the existing index entry pointing from the object's unique identifier to the log record's physical offset, or creating a new index entry while retaining the old index entry for version-specific retrieval. In this way, subsequent spatiotemporal range retrieval requests can locate the corresponding version's metadata log record through the metadata index structure based on the version identifier or default version selection strategy, thereby enabling the application of different version clock corrections to the same data file.

[0064] It should be noted that the appending method described in this application refers to a writing strategy in which new data is written to the end of the log file without overwriting or deleting existing log entries. This strategy is used to ensure that the metadata log layer can completely record the clock correction amount of all versions in the form of an immutable log stream.

[0065] In another aspect, in some embodiments, this application provides a dynamic storage system for space observation data, referring to... Figure 4 The figure is a schematic diagram of the structure of a dynamic storage system for space observation data according to some embodiments of this application. The dynamic storage system 200 for space observation data includes a receiving module 201, a processing module 202, and an execution module 203, which are described below: The receiving module 201 in this application is mainly used to receive data files from multiple space observation sources and write the data files into the raw data layer of the storage system. The data files include initial observation time stamps and data source identifiers. Processing module 202, in this application, is mainly used to obtain the clock error correction amount for each data source identifier at the corresponding observation time. The clock error correction amount includes clock error model parameters. The clock error correction amount is written into the metadata log layer of the storage system, and the initial observation time label in the original data layer remains unchanged. In addition, the processing module 202 in this application is also used to read the matching clock correction amount in the metadata log layer when a spatiotemporal range retrieval request is received, and use the clock error model parameters therein to correct the sampling point time label of the corresponding data file in the original data layer point by point to obtain the corrected observation time. In addition, the processing module 202 in this application is also used to perform spatiotemporal range matching on the data files in the original data layer based on the corrected observation time, filter out the data files that meet the retrieval request, and attach the version identifier of the applied clock correction amount when returning the list; The execution module 203 in this application is mainly used to write the new clock error correction and its version identifier to the metadata log layer in an append manner when an updated clock error correction is obtained for the same data source identifier.

[0066] In addition, this application also provides a computer device, which includes a memory and a processor. The memory stores code, and the processor is configured to acquire the code and execute the above-described dynamic storage method for space observation data.

[0067] In some embodiments, reference Figure 5 This figure is an internal structural diagram of a computer device implementing a dynamic storage method for space observation data according to some embodiments of this application. The dynamic storage method for space observation data in the above embodiments can be implemented through... Figure 5 The computer device shown is used to implement this, and the computer device 300 includes at least one processor 301, a communication bus 302, a memory 303, and at least one communication interface 304.

[0068] The processor 301 may be a general-purpose central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more devices used to control the execution of the dynamic storage method for space observation data in this application.

[0069] The communication bus 302 is used to transmit information between the aforementioned components.

[0070] Memory 303 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital versatile optical discs, Blu-ray discs, etc.), magnetic disks or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto. Memory 303 may exist independently and be connected to processor 301 via communication bus 302. Memory 303 may also be integrated with processor 301.

[0071] The memory 303 stores program code for executing the scheme of this application, and its execution is controlled by the processor 301. The processor 301 executes the program code stored in the memory 303. The program code may include one or more software modules. In the above embodiments, the dynamic storage method for space observation data can be implemented by the processor 301 and one or more software modules in the program code in the memory 303.

[0072] Communication interface 304 uses any transceiver-like device to communicate with other devices or communication networks, such as Ethernet, radio access network (RAN), wireless local area networks (WLAN), etc.

[0073] In a specific implementation, as one example, a computer device may include multiple processors, each of which may be a single-core (single-CPU) processor or a multi-core (multi-CPU) processor. Here, a processor may refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions).

[0074] The aforementioned computer device can be a general-purpose computer device or a special-purpose computer device. In specific implementations, the computer device may be a desktop computer, a portable computer, a network server, a handheld digital assistant (PDA), a mobile phone, a tablet computer, a wireless terminal device, a communication device, or an embedded device. This application does not limit the type of computer device.

[0075] In addition, this application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the above-described method for dynamically storing space observation data.

[0076] In summary, the space observation data dynamic storage method and system disclosed in this application involves receiving data files from multiple space observation sources and writing these data files into the raw data layer of the storage system. The data files contain initial observation time stamps and data source identifiers. Clock corrections for each data source identifier at the corresponding observation time are obtained. These clock corrections include clock error model parameters. The clock corrections are then written into the metadata log layer of the storage system, while maintaining the initial observation time stamps in the raw data layer. When a spatiotemporal range retrieval request is received, the matching clock corrections are read from the metadata log layer. The clock error model parameters are used to correct the sampling point time labels of the corresponding data files in the original data layer point by point to obtain the corrected observation time. Based on the corrected observation time, the data files in the original data layer are matched in spatiotemporal range to filter out the data files that meet the retrieval request, and the version identifier of the applied clock error correction is attached when returning the list. When an updated clock error correction is obtained for the same data source identifier, the new clock error correction and its version identifier are written to the metadata log layer in an append manner. While maintaining the immutable storage of the original spatial observation data, continuous incremental updates of the ex-post clock error correction are supported.

[0077] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0078] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A method for dynamically storing space observation data, characterized in that, Includes the following steps: The system receives data files from multiple space observation sources and writes these data files into the raw data layer of the storage system. The data files contain initial observation time stamps and data source identifiers. Obtain the clock correction for each data source identifier at the corresponding observation time. The clock correction includes clock model parameters. Write the clock correction into the metadata log layer of the storage system and keep the initial observation time label in the original data layer unchanged. When a spatiotemporal range retrieval request is received, the matching clock correction amount is read from the metadata log layer, and the clock model parameters therein are used to correct the sampling point time labels of the corresponding data files in the original data layer point by point to obtain the corrected observation time. Based on the corrected observation time, the data files in the original data layer are matched in time and space range to filter out the data files that meet the search request, and a version identifier of the applied clock correction is attached when the list is returned. When an updated clock correction is obtained for the same data source identifier, the new clock correction and its version identifier are appended to the metadata log layer.

2. The method as described in claim 1, characterized in that, Obtain the clock error correction for each data source identifier at the corresponding observation time. The clock error correction includes clock error model parameters, specifically: Based on the data source identifier, query the preset space observation platform clock error service directory to determine the clock error data acquisition interface and clock error data release frequency corresponding to the data source identifier; Based on the observation time indicated by the initial observation time tag, the clock difference data acquisition interface request is obtained and the effective clock difference correction information covering the observation time is extracted. The effective clock difference correction information includes clock difference polynomial model coefficients or clock difference ephemeris sampling point sequence. The effective clock error correction information is converted into clock error model parameters suitable for unified calculation of time base, which are used as the clock error correction amount for each data source at the corresponding observation time.

3. The method as described in claim 1, characterized in that, Writing the clock correction to the metadata log layer of the storage system while keeping the initial observation time stamps in the original data layer unchanged specifically includes: Construct a metadata log record that includes the data source identifier, the timestamp range of the initial observation time label, and the clock correction amount; The metadata log records are submitted to the storage system's metadata log layer, which is independent of the original data layer, in an append-only write mode, and a logical association index is established between the metadata log records and the corresponding immutable data objects in the original data layer. When responding to a time retrieval request for the initial observation timestamp, the clock correction amount in the metadata log layer is read through the logical association index and the corrected observation time is dynamically calculated, while the initial observation timestamp stored in the original data layer remains unchanged.

4. The method as described in claim 1, characterized in that, When a spatiotemporal range retrieval request is received, the matching clock correction values ​​read from the metadata log layer specifically include: Extract the target time range constraints from the spatiotemporal range retrieval request, and determine the retrieval period covered by the target time range constraints; In the metadata log layer, metadata log records whose observation time range intersects with the retrieval period are selected as candidate metadata log records; Clock error model parameters are extracted from the candidate metadata log records, and the extracted clock error model parameters are determined as clock error correction amounts that match the corresponding data files.

5. The method as described in claim 4, characterized in that, The clock error model parameters are used to correct the time stamps of the sampling points in the corresponding data files within the original data layer point by point, resulting in the corrected observation times, which specifically include: Based on the associations recorded in the candidate metadata log, locate the corresponding immutable data object in the original data layer, and read the sampling point time tag sequence within the immutable data object; For each sampling point in the time-stamped sequence of the sampling points, the clock error compensation value is calculated using the clock error model parameters; The corrected observation time of the data file is generated by subtracting the corresponding clock error compensation value from the initial time label of each sampling point.

6. The method as described in claim 1, characterized in that, Based on the corrected observation time, spatiotemporal range matching is performed on the data files in the original data layer to filter out data files that meet the retrieval request, specifically including: The corrected observation time is matched with the target time range constraint in the spatiotemporal range retrieval request to determine whether the data file meets the retrieval request in the time dimension. For data files that meet the retrieval request in the time dimension, the spatial coverage information of the original records of the data files is synchronously calibrated based on the corrected observation time and the orbit and attitude metadata pre-archived by the corresponding data source to obtain the corrected spatial coverage range; combined with the target spatial range constraint conditions in the spatiotemporal range retrieval request, the corrected spatial coverage range is used to perform spatial matching determination. Data files that simultaneously meet both time and spatial dimension matching requirements are identified as data files that satisfy the retrieval request.

7. The method as described in claim 1, characterized in that, When an updated clock correction is obtained for the same data source identifier, the new clock correction and its version identifier are appended to the metadata log layer. This includes: Obtain a new clock error correction amount corresponding to the same data source identifier, and generate a new version identifier for the new clock error correction amount that is different from the existing version identifier; The new clock error correction is used to convert into new clock error model parameters, and a new metadata log record is constructed. The new metadata log record is submitted to the metadata log layer in an append-only manner.

8. The method as described in claim 1, characterized in that, The initial observation timestamp refers to the original timestamp recorded by the space observation platform during the acquisition of observation data, relying on its onboard clock or local frequency source, and embedded in the header of the data file.

9. The method as described in claim 1, characterized in that, The data source identifier refers to a globally unique encoded string consisting of one or more of the following: space observation platform number, payload number, beam number, or channel number.

10. A dynamic storage system for space observation data, characterized in that, include: The receiving module is used to receive data files from multiple space observation sources and write the data files into the raw data layer of the storage system. The data files include initial observation time stamps and data source identifiers. The processing module is used to obtain the clock error correction amount for each data source identifier at the corresponding observation time. The clock error correction amount includes clock error model parameters. The clock error correction amount is written into the metadata log layer of the storage system, and the initial observation time label in the original data layer remains unchanged. The processing module is also used to read the matching clock correction amount in the metadata log layer when a spatiotemporal range retrieval request is received, and use the clock error model parameters therein to correct the sampling point time label of the corresponding data file in the original data layer point by point to obtain the corrected observation time. The processing module is also used to perform spatiotemporal range matching on the data files in the original data layer based on the corrected observation time, filter out the data files that meet the retrieval request, and attach the version identifier of the applied clock correction amount when returning the list; The execution module is used to append the new clock correction and its version identifier to the metadata log layer when an updated clock correction is obtained for the same data source identifier.

Images