A scientific satellite data processing system with cross-APID hybrid dependencies

Abstract

The invention belongs to the field of scientific satellite data processing, and discloses a scientific satellite data processing system with cross-APID mixed dependency, including: a raw data preprocessing module, used for preprocessing multi-source data and generating level 0 products; an engineering parameter template and dependency matrix parsing module, used for parsing level 0 products according to the scientific satellite mission engineering parameter template of the parameter parsing configuration file and the parameter dependency matrix of the cross-APID source package; a Redis cache library sharing architecture module, used for sharing the parameter values ​​of each data product across the APID source package using the Redis cache; a cross-APID dependency parameter scheduling module, used for scheduling the processing order according to different data types and dependency situations; a data product formatting generation module, used for generating products of various types and levels based on product definition specifications after all parameters are processed and cached.

Description

Technical Field

[0001] The present application belongs to the field of scientific satellite data processing, and in particular, to a scientific satellite data processing system with cross-APID hybrid dependencies. Background Art

[0002] As the data processing requirements of modern scientific satellites become increasingly complex, the demand for data fusion and analysis between satellite multi-payloads and multi-tasks is gradually increasing. In scientific satellite engineering missions, parameter analysis and the production of scientific data products usually rely on parameters in multiple APID (application process identification) source packages. These parameters may originate from the same data processing unit or across data processing units (different production modules, different processing batches, and different cycles), and there are complex dependencies between them. This cross-APID and cross-cycle dependency increases the complexity of data processing.

[0003] Traditional parameter parsing methods face many challenges in dealing with these cross-APID and cross-cycle dependencies. For example, traditional methods rely on predefined parsing rules or sequential processing methods, which makes it difficult to flexibly deal with the dependencies between multiple data sources, and there is redundancy in the data processing of different cycles. This limitation makes the data processing process more complicated and lengthy, increasing the computational burden of the system.

[0004] Therefore, there is an urgent need for an automated algorithm that can efficiently handle cross-APID and cross-cycle dependencies to ensure the consistency, efficiency, and accuracy of scientific data processing and product production. Summary of the invention

[0005] The purpose of the present invention is to overcome the defects of the prior art, propose a scientific satellite data processing system with cross-APID hybrid dependencies, and successfully apply it to the Sino-French Astronomical Satellite (SVOM) engineering mission, ensuring the correct and efficient processing of SVOM satellite downlink data.

[0006] In order to achieve the above object, the present invention proposes a scientific satellite data processing system with cross-APID hybrid dependency, comprising:

[0007] The raw data preprocessing module is used to preprocess multi-source data and generate level 0 products;

[0008] The engineering parameter template and dependency matrix parsing module is used to parse the level 0 product according to the scientific satellite mission engineering parameter template of the parameter parsing configuration file and the parameter dependency matrix across the APID source package;

[0009] Redis cache library sharing architecture module, used to use Redis cache to share the parameter values ​​of various data products across APID source packages;

[0010] The cross-APID dependency parameter scheduling module is used to schedule the processing order according to different data types and dependencies;

[0011] The data product formatting generation module is used to generate products of various types and levels based on the product definition specifications after all parameters are processed and cached.

[0012] Preferably, the pretreatment comprises:

[0013] When the original data transmission file is received from the ground station, the virtual channel separation and source packet extraction of the single-track data are first performed to generate the corresponding 0C-level source packet data product, and the multi-track data are spliced ​​and deduplicated, and the data jump statistics and fault tolerance are performed to generate the 0D-level source packet data product. The 0C-level source packet data product includes scientific data and engineering data.

[0014] Preferably, the scientific satellite mission engineering parameter template is used to describe the name, unit, byte size, starting and ending position in the data packet, and calculation formula of each telemetry parameter inside the engineering product; the parameter dependency matrix across APID source packages is used to describe the parameter dependencies in different data products, the rows of the matrix represent the parameter codes of the target products, and the columns of the matrix represent the APID source package numbers of one or more other dependent parameters, the dependent parameter codes, the parsing order, and the parsing method formulas.

[0015] Preferably, the engineering parameter template and dependency matrix parsing module preferentially processes the source package where the dependency parameters are located, and traverses and performs parsing calculations on each parameter until all parameters are processed.

[0016] Preferably, the cache shares the parameter values ​​of various data products across APID source packages in the following manner: including a cache key value and a parameter value, wherein the cache key value includes a time code, an APID source package identifier, and a parameter code; the cache expiration time is set based on the on-board storage capacity of the scientific satellite mission to be processed, and a random delay of 0 to a set second is added to stagger the time points when a large amount of cached data becomes invalid, thereby preventing the system from processing a large amount of cached invalid data in the same time period, and ensuring the smooth operation of the system.

[0017] Preferably, the Redis cache library shared architecture module is also used to clear expired parameter cache according to a set time.

[0018] Preferably, the processing process of the cross-APID dependent parameter scheduling module includes:

[0019] Step 1) Set the waiting threshold for level 1 processing of engineering parameters. When it is less than the threshold and the parallel engineering parameter processing in the orbital data is completed, trigger the parameter processing flow of scientific data. When the parameter X to be calculated depends on the parameters of other APID source packages, go to step 2), otherwise go to step 3);

[0020] Step 2) Get the data packet time TS of the current parameter, query the Redis cache according to TS and the dependent parameter code, get the time code T0 closest to TS, when |TS-T0|≤TH is satisfied, where TH represents the set threshold; substitute the dependent parameter value obtained by the query into the calculation formula of parameter X, output the parameter value of parameter X, and go to step 4); when |TS-T0|≤TH is not satisfied, use the default value for calculation and go to step 4);

[0021] Step 3) Calculate the parameter X and perform time processing;

[0022] Step 4) Cache the parsed parameter values.

[0023] Preferably, the data product formatting generation module is also used to verify the generated product format to ensure the integrity and accuracy of the data.

[0024] Compared with the prior art, the advantages of the present invention are:

[0025] 1. The cross-APID hybrid dependency processing system proposed in this invention designs a hierarchical framework for 0C, 0D, and 1-level scientific data products, and proposes an engineering parameter configuration template and a dependency configuration template, which can effectively handle the complex parameter dependency relationships between different cycles and different source packages, and automatically processes through configuration files and cache mechanisms, thereby improving the consistency and processing efficiency of data parsing, parameter solution, and product generation processes;

[0026] 2. By using Redis cache, the time overhead of repeated data reading and calculation is reduced when processing cross-module dependencies, the data processing process is optimized, and the efficiency of solving cross-APID dependency parameters and data consistency are ensured. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 It is a module diagram of the cross-APID hybrid dependent parameter scientific data processing system of the present invention;

[0028] Figure 2 It is an example of a project parameter template;

[0029] Figure 3 This is a flowchart of the processing of the original data source packet of the digital transmission using the SVOM satellite as an example (level 0 processing);

[0030] Figure 4This is a flow chart of cross-APID and cross-type mixed dependent parameter processing (level 1 processing) taking the SVOM satellite as an example. DETAILED DESCRIPTION

[0031] The purpose of the present invention is to provide an efficient cross-APID hybrid dependency parameter processing system to solve the problems of cross-circle and cross-APID dependency and key engineering parameter sharing in scientific satellite downlink data processing. It can efficiently manage dependent parameters in a multi-source package and multi-task processing environment, avoid repeated calculations and improve data processing speed, and ensure the consistency and accuracy of data processing.

[0032] like Figure 1 As shown, the present invention proposes a cross-APID hybrid dependency data processing system, and the modular design scheme is as follows:

[0033] 1) Raw data preprocessing module, which preprocesses multi-source data and generates level 0 products, including virtual channel separation, source packet extraction, source packet sorting, and data verification of single-track data; splicing, sorting and deduplication of multi-track data, statistics and fault tolerance of data jumps, etc.

[0034] 2) Engineering parameter template and dependency matrix parsing module: parses the pre-built scientific satellite mission engineering parameter template and the parameter dependency matrix across APID source packages, and automatically identifies and classifies dependent parameters based on the dependency matrix and parsing rules;

[0035] 3) Redis cache library sharing architecture module: The Redis cache library is used to store key engineering parameters, support data sharing between different products, and adopt a key-value pair structure of time code + parameter code to effectively manage the life cycle of cache data, avoid repeated calculations, and improve data access speed;

[0036] 4) Cross-APID dependency parameter scheduling module: Schedule the processing order of level 1 engineering and scientific data according to different data types (engineering, science) and dependency conditions (whether it is a dependent source package); give priority to triggering the processing flow of level 1 engineering data, first read the source package 0D-level engineering data product file to be processed and perform parameter solution, parse the configuration file and parse the dependency matrix according to the engineering parameters, schedule and traverse the calculation of each parameter; for parameters with cross-APID dependencies in parameter calculation, obtain the parameter values ​​of cross-APID dependencies from the Redis cache and then calculate, and directly calculate the parameters without cross-APID dependencies; after the processing of level 1 engineering parameters is completed (within the threshold range), trigger the production of level 1 scientific data products, obtain the data packet time TS of the current parameter, query the Redis cache according to TS and the dependent parameter code, obtain the engineering parameter value closest to TS and calculate it.

[0037] 5) Data product formatting generation module: After all parameters are parsed and cached, various types and levels of products are generated based on the product definition specifications.

[0038] Through the collaborative work of these modules, the present invention can realize an integrated solution for cross-APID cross-circle parameter dependency processing, key engineering parameter sharing, and data product formatting generation. The algorithm has been effectively verified in the Sino-French Astronomical Satellite (SVOM) engineering mission, ensuring the efficient and correct processing of SVOM satellite level 0 and level 1 products.

[0039] The technical solution of the present invention is described in detail below with reference to the accompanying drawings and embodiments.

[0040] Example

[0041] An embodiment of the present invention provides a cross-APID hybrid dependency data processing system, including:

[0042] The raw data preprocessing module is used to preprocess multi-source data and generate level 0 products;

[0043] The engineering parameter template and dependency matrix parsing module is used to parse the level 0 product according to the scientific satellite mission engineering parameter template of the parameter parsing configuration file and the parameter dependency matrix across the APID source package;

[0044] Redis cache library sharing architecture module, used to use Redis cache to share the parameter values ​​of various data products across APID source packages;

[0045] The cross-APID dependency parameter scheduling module is used to schedule the processing order according to different data types and dependencies;

[0046] The data product formatting generation module is used to generate products of various types and levels based on the product definition specifications after all parameters are processed and cached.

[0047] The following are the details:

[0048] 1. Engineering parameter template analysis and dependency matrix construction

[0049] Project parameter template definition:

[0050] The engineering XML template is used to describe the structure, location, and calculation method of each parameter in the telemetry data packet. Taking the SVOM satellite engineering mission as an example, the template contains the APID of the data packet, the name, unit, byte size, start and end position of each telemetry parameter in the data packet, and the calculation formula (including the internal dependencies of the engineering product, that is, the mutual dependencies within the same data processing unit and the same virtual channel downlink file). Figure 2 Shown is an example of an engineering parameter template.

[0051] When parsing XML, first extract the data packet information, then obtain the original data at the corresponding position, and calculate according to the formula. This template clarifies the calculation information of each parameter, which is convenient for subsequent processing and maintenance of engineering parameters.

[0052] Dependency matrix definition: The parameter dependency matrix is ​​used to describe the dependency relationship between key parameters in different types of data products (mainly for different virtual channel downlink files, i.e. different data processing units). Taking the SVOM satellite engineering mission as an example, the GRM L1a science data product depends on multiple parameters in the HK1 engineering data product, including payload temperature, orbit status, etc. The dependency matrix can intuitively and concisely show the dependency relationship between the parameters in one product and the parameters of other types of products, and provide input for the parameter processing sequence.

[0053] The rows of the matrix represent the parameters in the target product, and the columns represent the dependent APID source packages and parameter codes. The parsing order column indicates the order of calculation of the target product parameters. The data of the same parsing order are processed in parallel. The parsing method column gives the specific calculation formula. The specific matrix design is shown in Table 1:

[0054] Table 1 Dependency parameter matrix diagram

[0055]

[0056] In this example, GRM_TIMES and GRM_TEMP3 can be processed in parallel at the same time in the first order, while GRM_V_main depends on the calculation of GRM_TIMES, so GRM_V_main is processed in the second order.

[0057] Configuration file loading: The information of the project parameter template and dependency matrix are written into the configuration file. When the system starts, it will load the configuration file and parse and store it in the memory for subsequent parameter parsing.

[0058] 2. Construction and optimization of Redis shared cache library

[0059] ·Cache library design: Use Redis cache library to store key engineering parameters. Each cache key-value pair consists of a time code, an APID source package identifier, and a parameter code to ensure that data across APIDs can be effectively shared between different modules and tasks. The structural design of the Redis cache library follows the principles of uniqueness and efficiency. For example, a typical key can be expressed as "20240822093000_APID0633_HK1_TEMP", where the time code identifies the time when the data is generated, the APID source package identifies the source of the data, and the parameter code indicates the specific parameter code.

[0060] Table 2 Redis cache diagram

[0061]

[0062] Optimization of cache mechanism: Taking into account the situation of on-demand data on the satellite and a certain degree of redundancy, the cache data is set to have a validity period of 10 days (864,000 seconds) (which can be adjusted according to the specific satellite storage capacity design). At the same time, a random delay of 0 to 1,000 seconds is added to stagger the time points when a large amount of cache data becomes invalid, preventing the system from processing a large amount of cache invalid data in the same time period, and ensuring the stability of the system operation.

[0063] 3. Data processing and scheduling

[0064] Raw data preprocessing: When the system receives the raw data files from the ground station, it first performs virtual channel separation and source packet extraction of single-track data, and sorts and removes duplicates of the current track data to generate the corresponding 0C-level source packet data products (including science and engineering). Secondly, it performs splicing, sorting, duplicate removal, data jump statistics and fault tolerance of multi-track data on the basis of 0C to generate 0D-level source packet data products.

[0065] Cross-APID dependent parameter scheduling:

[0066] The processing order is scheduled according to different data types (engineering, science) and dependency situations (whether it is a dependent source package): the processing flow of level 1 engineering data is triggered first, first reading the 0D-level engineering data product file of the source package to be processed and performing parameter calculations, parsing the configuration file and the dependency matrix according to the engineering parameters, scheduling and traversing the calculation of each parameter, and giving priority to the source package where the dependent parameters are located.

[0067] After the Level 1 engineering parameter processing is completed (within the threshold range), the production of Level 1 scientific data products is triggered. For parameters with cross-APID dependencies in the calculation of scientific product parameters, the cross-APID dependent parameter values ​​are queried from the Redis cache based on the current parameter time and the dependent parameter code, and the calculation is performed after that. For parameters without cross-APID dependencies, they are calculated directly.

[0068] · Cache priority strategy and scheduling: The system first queries the value of the dependent parameter from the Redis cache library. If the corresponding data exists in the cache, it is directly read and substituted into the calculation formula for analysis; if there is no dependent parameter, the system directly calculates according to the analysis rules in the configuration file.

[0069] Time backtracking query and exception handling: If the dependent parameter is not found in the cache, the system starts the backtracking query mechanism and backtracks second by second. The maximum backtracking time is usually set to 8 seconds (the threshold can be adjusted according to the characteristics of the data). If no matching value is found in the end, the system uses the preset default value for calculation and records the exception.

[0070] 4. Parameter calculation and product formatting generation

[0071] ·Storage and update of analysis results: After the parameter analysis is completed, the system writes the analysis results to the Redis cache for subsequent task calls to avoid repeated calculations. For example, when generating HK1-level products, the dependent parameter analysis results will be immediately stored in the cache library to ensure that the GRM L1a product can quickly obtain data when it depends on these parameters.

[0072] Data product formatting generation and verification: After all parameters are parsed and cached, the system generates the final scientific data product file (such as FITS file, etc.) according to the predefined product format. The generated product is format-verified to ensure the integrity and accuracy of the data, and is stored or distributed according to the mission requirements.

[0073] The specific processing flow is as follows Figure 3 and Figure 4 As shown:

[0074] Step 1) When the system receives the original data transmission file from the ground station, it first performs virtual channel separation and source packet extraction of single-track data to generate corresponding 0C-level source packet data products (including science and engineering 2 categories), and then performs multi-track data splicing, sorting and repetition removal, data jump statistics and fault tolerance on the basis of 0C, generates 0D-level source packet data products, and first triggers the production process of 1-level engineering data products: read the 0D-level engineering data product file to be processed, the engineering parameter template and the dependency matrix parsing module, give priority to the source package where the dependency parameters are located, and traverse the parsing calculation of each parameter until all parameters are processed:

[0075] Step 2) Set the waiting threshold for level 1 processing of engineering parameters. When it is less than the threshold and the parallel engineering parameter processing in the orbital data is completed, trigger the parameter processing flow of scientific data. When the parameter X to be calculated depends on the parameters of other APID source packages, go to step 3), otherwise go to step 4);

[0076] Step 3) Get the data packet time TS of the current parameter, query the Redis cache according to TS and the dependent parameter code, get the time code T0 closest to TS, when |TS-T0|≤TH is satisfied, where TH represents the set threshold; substitute the dependent parameter value obtained by the query into the calculation formula of parameter X, output the parameter value of parameter X, and go to step 5); when |TS-T0|≤TH is not satisfied, use the default value for calculation and go to step 5);

[0077] Step 4) Calculate the parameter X and perform time processing;

[0078] Step 5) Cache the parsed parameter values;

[0079] Step 6) Format and produce corresponding data products according to the format definition specifications.

[0080] The focus of the present invention is to propose a data processing method for cross-APID hybrid dependencies. The core of this method is to effectively manage and schedule the dependency parameters between different APID source packages in different circles, realize data sharing and cache management, and effectively support the hybrid dependency and fusion processing of multi-source data of scientific satellites.

[0081] It should be pointed out in particular that the present invention does not involve a specific payload data processing algorithm, that is, the present invention does not propose a new calculation formula or processing method for how to analyze the observation data of a specific payload or how to calculate specific scientific parameters.

[0082] The cross-APID hybrid dependency parameter processing algorithm provided by the present invention realizes efficient multi-source data hybrid dependency fusion processing by designing engineering parameter templates, dependency matrices, data scheduling processes and cache mechanisms. Specifically, it is manifested as follows:

[0083] 1) Scientific product classification definition: Based on the characteristics of satellite and payload scientific data, different levels of product design frameworks are constructed, laying a solid foundation for subsequent processing procedures and system development, ensuring data compatibility and scalability between products of different levels, thereby improving the flexibility and adaptability of the system.

[0084] 2) Consistency and accuracy guarantee: In view of the dependencies between different source packages, the system ensures that the Redis cache mechanism is used in parameter solution to reduce the delay of cross-module data reading, maintain data consistency, and improve the accuracy of solution.

[0085] 3) Process optimization and resource utilization: The system uses Redis cache and automatic scheduling mechanism to significantly optimize the data processing process, reduce resource consumption under complex dependencies, and improve processing efficiency.

[0086] 4) Flexible data processing capabilities: The system supports updates to dependent configurations and can flexibly process and generate various scientific data products to meet the diverse needs of satellite engineering.

[0087] Compared with the existing technology, the present invention has achieved a breakthrough in the automation of cross-APID data processing, can effectively support complex data processing tasks in satellite engineering, and provide an efficient and reliable technical solution for data production and application of scientific satellites.

[0088] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit the present invention. Although the present invention is described in detail with reference to the embodiments, it should be understood by those skilled in the art that any modification or equivalent replacement of the technical solutions of the present invention does not depart from the spirit and scope of the technical solutions of the present invention and should be included in the scope of the claims of the present invention.

Claims

1. A scientific satellite data processing system with cross-APID hybrid dependencies, characterized in that: include: The raw data preprocessing module is used to preprocess multi-source data and generate level 0 products; The engineering parameter template and dependency matrix parsing module is used to parse the level 0 product according to the scientific satellite mission engineering parameter template of the parameter parsing configuration file and the parameter dependency matrix across the APID source package; Redis cache library sharing architecture module, used to use Redis cache to share the parameter values ​​of various data products across APID source packages; The cross-APID dependency parameter scheduling module is used to schedule the processing order according to different data types and dependencies; and The data product formatting generation module is used to generate various types and levels of products based on the product definition specifications after all parameters are processed and cached; The pre-processing comprises: When receiving the original data transmission file from the ground station, first perform virtual channel separation and source packet extraction of single-track data to generate the corresponding 0C-level source packet data product, perform splicing and deduplication of multi-track data, statistics and fault tolerance of data jumps, and generate 0D-level source packet data products. The 0C-level source packet data products include scientific data and engineering data; The engineering parameter template and dependency matrix parsing module preferentially processes the source package where the dependency parameters are located, and traverses and performs parsing calculations on each parameter until all parameters are processed; The processing process of the cross-APID dependent parameter scheduling module includes: Step 1) Set the waiting threshold for level 1 processing of engineering parameters. When it is less than the threshold and the parallel engineering parameter processing in the orbital data is completed, trigger the parameter processing flow of scientific data. When the parameter X to be calculated depends on the parameters of other APID source packages, go to step 2), otherwise go to step 3); Step 2) Get the data packet time TS of the current parameter, query the Redis cache according to TS and the dependent parameter code, get the time code T0 closest to TS, when |TS-T0|≤TH is satisfied, where TH represents the set threshold; substitute the dependent parameter value obtained by the query into the calculation formula of parameter X, output the parameter value of parameter X, and go to step 4); when |TS-T0|≤TH is not satisfied, use the default value for calculation and go to step 4); Step 3) Calculate the parameter X and perform time processing; Step 4) Cache the parsed parameter values.

2. The scientific satellite data processing system with cross-APID hybrid dependencies according to claim 1 is characterized in that: The scientific satellite mission engineering parameter template is used to describe the name, unit, byte size, starting and ending position in the data packet, and calculation formula of each telemetry parameter within the engineering product; the parameter dependency matrix across APID source packages is used to describe the parameter dependencies in different data products. The rows of the matrix represent the parameter codes of the target products, and the columns of the matrix represent the APID source package numbers of one or more other dependent parameters, the dependent parameter codes, the parsing order, and the parsing method formulas.

3. The scientific satellite data processing system with cross-APID hybrid dependencies according to claim 1, characterized in that: The cache shares the parameter values ​​of various data products across APID source packages in the following manner: including cache key values ​​and parameter values, wherein the cache key values ​​include time codes, APID source package identifiers, and parameter codes; the cache expiration time is set based on the on-board storage capacity of the scientific satellite mission to be processed, and a random delay of 0 to a set second is added to stagger the time points when a large amount of cached data becomes invalid, thereby preventing the system from processing a large amount of cached invalid data in the same time period, and ensuring the smooth operation of the system.

4. The scientific satellite data processing system with cross-APID hybrid dependencies according to claim 1, characterized in that: The Redis cache library shared architecture module is also used to clear expired parameter cache according to a set time.

5. The scientific satellite data processing system with cross-APID hybrid dependencies according to claim 1, characterized in that: The data product formatting generation module is also used to verify the generated product format to ensure the integrity and accuracy of the data.

Images