High-speed data acquisition and storage methods for large-sample spacecraft system simulation

Abstract

This invention relates to a high-speed data acquisition and storage method, electronic device, and storage medium for large-sample spacecraft system simulation. The method includes the following steps: S1, Data Aggregation: Real-time acquisition of data from the large-sample simulation process, and offline synchronization of data from other databases; S2, High-Speed ​​Data Storage Management: Generating database operations for multiple SQL dialects through a multi-SQL dialect engine framework, and managing the aggregated data in an integrated manner to form a distributed data resource pool; S3, Data Exchange: Defining data exchange services through unified data encapsulation and interface definitions, and performing high-speed data querying and sharing based on information pressure and automatic data synchronization mechanisms. This invention enables high-speed data acquisition for large-sample spacecraft system simulation processes, supporting data mining, evaluation, and decision support in large-sample spacecraft system simulation experiments.

Description

Technical Field

[0001] This invention relates to the field of spacecraft system simulation technology, and in particular to a high-speed data acquisition and storage method for large-sample spacecraft system simulation. Background Technology

[0002] In information-based environments, spacecraft system simulation experiments involve numerous and diverse spacecraft types, with complex interaction methods. The experimental environment includes not only digital spacecraft models but also physical spacecraft payloads, mission planning systems, and image processing systems. Therefore, to effectively evaluate and analyze the results of spacecraft system simulation experiments, it is necessary to uniformly collect, store, manage, share, and utilize experimental design data, massive amounts of experimental process data of different types, and evaluation result data in real time. After the simulation, it is also necessary to quickly query various process and design data for comprehensive evaluation and analysis.

[0003] Currently, the acquisition and storage of large-sample spacecraft system simulation experimental data face the following challenges:

[0004] Firstly, during the data acquisition process of the massive sample spacecraft system simulation experiment, a large amount of model interaction data, entity state data, and key event data are collected in real time and classified and stored according to different scenarios. Specifically, time-series data such as satellite orbit predictions need to be stored in the database in real time according to the simulation step size; discrete event data such as detection events and adjudication results need to be monitored in real time and promptly stored. Simultaneously, data needs to be aggregated from other types of databases and external systems through data synchronization and import. Therefore, efficient and stable data acquisition is required to collect and store real-time data from each sample, as well as structured and unstructured data from external systems.

[0005] Secondly, after completing the collection, connection and synchronization of various types of data, it is necessary to build a fast and effective large-scale data processing platform to solve the problem of unified storage management and shared use of massive data, so as to better serve large-scale data processing scenarios such as simulation and comprehensive evaluation, decision support, and scenario analysis.

[0006] Therefore, designing an efficient and reliable method for high-speed acquisition and storage of large-sample spacecraft system simulation experimental data has become an urgent problem to be solved. Summary of the Invention

[0007] To address the technical problems existing in the prior art, the present invention aims to provide a high-speed data acquisition and storage method for large-sample spacecraft system simulation, integrating orbit calculation, spatial correlation and profile visualization analysis into one, providing an effective means for three-dimensional profile visualization and correlation analysis of space targets.

[0008] To achieve the above-mentioned objectives, this invention provides a high-speed data acquisition and storage method for large-sample spacecraft system simulation, comprising the following steps:

[0009] Step S1, Data Collection: Real-time acquisition of data from large-sample simulation and simulation processes, as well as offline connection and synchronization of data from other databases;

[0010] Step S2, High-speed data storage management: Generate database operations for multiple SQL dialects through a multi-SQL dialect engine framework, perform integrated management of the collected data, and form a distributed data resource pool;

[0011] Step S3, Data Exchange: Through unified data encapsulation and interface definition, data exchange services are provided, and high-speed data query and sharing are achieved based on information pressure mechanism and automatic data synchronization mechanism.

[0012] According to a technical solution of the present invention, in step S1, the data of the large-sample simulation and deduction process is collected in real time and optimized by a data collection service process optimization method; wherein, the data collection service process optimization method specifically includes:

[0013] A. Conduct an in-depth performance evaluation of the existing physical machines to determine their limits, and configure a tuning thread pool with a fixed upper limit based on the evaluation results. The tuning thread pool shall be used without affecting other services deployed on the same machine, and the data collection process shall be processed within the limits of the tuning thread pool.

[0014] B. Use persistent methods within the consumer thread to aggregate data;

[0015] C. Automatically destroy the consumer threads of the data acquisition service based on the thread destruction conditions met by the simulation process data;

[0016] The thread destruction conditions are divided into two categories: normal reception of the simulation end flag and abnormal reception of the simulation end flag.

[0017] When the simulation ends normally, all threads of the acquisition service are blocked and waited for a preset time. If the preset time is exceeded, the resources held by all threads of the acquisition service are released and the threads are rolled back to the tuning thread pool. The consumer thread is destroyed according to the preset maximum timeout time of the thread in the tuning thread pool.

[0018] If the simulation end flag is not received normally, and the corresponding situation is that the simulation task is paused, only a step counter with very low overhead is maintained in the consumer thread, and the other thread resources of the collection service are put back into the corresponding tuning thread pool.

[0019] If the simulation end flag is not received normally, and the corresponding situation is that the model exits abnormally or the simulation engine exits abnormally, a step counter is maintained in the consumer thread. The number of steps in the newly received process data is compared with the number of steps in the step counter. If the two do not match, the number of steps in the step counter is cleared to zero and the state of the consumer thread is restored to normal. Otherwise, the thread of the data acquisition service is restored to the running state and the sample acquisition work of this simulation is continued.

[0020] D. A new daemon process is automatically created upon normal startup of the data collection service. The data collection service and the daemon process keep each other alive. The daemon process monitors whether the data collection service is working properly. If it exits abnormally, it is automatically restarted. The data collection service also monitors the daemon process. If the daemon process exits abnormally, it is automatically restarted.

[0021] According to a technical solution of the present invention, the data collection using a persistent method within the consumer thread specifically includes:

[0022] Based on the received simulation experiment start request, perform a persistent memory check of the consumption topic to determine whether there is a consumption thread for the topic corresponding to the simulation experiment start request that has been started.

[0023] If there is no consumer thread that corresponds to the topic of the simulation experiment start request, a new consumer thread is created to listen to the topic of the simulation experiment start request, and the simulation experiment identifier of the simulation experiment start request is set on the newly created consumer thread.

[0024] If there is a consumer thread for the topic corresponding to the simulation experiment start request, insert the simulation experiment identifier of the simulation experiment start request into the priority queue of the consumer thread for the corresponding topic.

[0025] The consumer thread consumes data in a loop, associating the received data with the simulation experiment identifier;

[0026] Based on the end marker received by the simulation engine from the consumer thread, the next simulation experiment marker to be stored is retrieved from the priority queue, and the subsequently received data is associated with the new simulation experiment marker.

[0027] According to a technical solution of the present invention, in step S1, the real-time collected data is optimized for data entry into the database through a data entry optimization mechanism, the data entry optimization mechanism including:

[0028] a. Design of the pre-processing stage for data entry: The collected real-time data is sent to the pre-processing thread pool. The pre-processing thread in the pre-processing thread pool performs preliminary parsing on the received data. The preliminary parsing only parses the port and does not parse the specific content of the protocol. According to the preset port mapping table, the port of the received data is pre-judged. If the port of the received data is related to the evaluation function or the playback function, the specific protocol content of the received data is parsed. Otherwise, it is directly packaged and entered into the database.

[0029] b. Set up a database connection pool, which is used to configure standard database connections and / or non-standard database connections according to the data entry requests in the data acquisition service;

[0030] c. Set up a storage thread pool for multi-threaded data storage, and write the data output by the preprocessing thread pool into the database in batches using multiple threads.

[0031] According to a technical solution of the present invention, in step S2, the multi-SQL dialect engine framework includes:

[0032] A unified SQL compiler for multi-engine databases, used to convert between standard SQL statements and different SQL dialects, and generate database operation execution plans;

[0033] The SQL engine integrates multiple database engines and is used to receive the database operation execution plan and perform corresponding database operations.

[0034] The storage layer is used to store data from different databases; the database type of the storage layer corresponds to the database type of the SQL engine.

[0035] According to a technical solution of the present invention, in step S2, the compilation process of the unified SQL compiler for multi-engine databases specifically includes:

[0036] Step S211: For database operations targeting different databases, generate a unified SQL statement, perform lexical analysis on the unified SQL statement, and convert it into a word sequence; perform syntax checking on the unified SQL statement based on metadata information and multi-storage information, and construct an abstract syntax tree based on the word sequence based on the result of passing the syntax check.

[0037] Step S212: Perform constant propagation optimization on the abstract syntax tree;

[0038] Step S213: Determine whether the unified SQL statement is a stored procedure. If yes, proceed to step S214; otherwise, proceed to step S217.

[0039] Step S214: Perform stored procedure semantic analysis on the unified SQL statement; the stored procedure semantic analysis includes semantic analysis of various dialect SQL and control flow semantic analysis;

[0040] Step S215: Based on the results of control flow semantic analysis, optimize the control flow of the unified SQL statement;

[0041] Step S216: Perform cursor parallelization on the unified SQL statement that has completed control flow optimization to generate a physical execution plan;

[0042] Step S217: Perform semantic analysis of various dialects on the unified SQL statement;

[0043] Step S218: Optimize the logical execution plan and transform the subqueries of the unified SQL statement that has completed semantic analysis of various dialect SQL statements, and generate a physical execution plan.

[0044] According to a technical solution of the present invention, in step S3, during the data query process, an information pressure mechanism is set, specifically including:

[0045] Step S31: Generate SQL commands based on the received data exchange instructions;

[0046] Step S32: Compile the SQL instructions using the multi-engine unified SQL compiler to generate a database execution plan;

[0047] Step S33: Optimize and identify database operation functions in the database execution plan, and perform optimal operation matching on the identified database operation functions;

[0048] Step S34: Based on the optimal operation matching result, optimize the database execution plan and replace the operation function with the optimal operation function;

[0049] Step S35: Return the database operation function result according to the optimal operation function.

[0050] According to a technical solution of the present invention, in step S3, before data exchange, multi-database data synchronization based on an automatic data synchronization mechanism is further included, specifically including:

[0051] Step S301: Collect and monitor the Binlog database using Flink CDC, and push the data change operations to the Kafka message queue in real time;

[0052] Step S302: Based on the data change operations in the Kafka message queue, clean, transform, adapt the format, and filter the data, and asynchronously synchronize the processed data to other databases or storage engines.

[0053] According to one aspect of the present invention, an electronic device is provided, comprising: one or more processors, one or more memories, and one or more computer programs; wherein the processor is connected to the memory, and the one or more computer programs are stored in the memory; when the electronic device is running, the processor executes the one or more computer programs stored in the memory, so that the electronic device performs the above-described high-speed data acquisition and storage method for large-sample spacecraft system simulation.

[0054] According to one aspect of the present invention, a computer-readable storage medium is provided for storing computer instructions, which, when executed by a processor, implement the above-described high-speed data acquisition and storage method for large-sample spacecraft system simulation.

[0055] Compared with the prior art, the present invention has the following beneficial effects:

[0056] This invention achieves the collection of data from large-sample simulation and other databases through data aggregation. It generates database operations for various SQL dialects using a multi-SQL dialect engine framework, enabling unified management of multiple storage engines and forming a distributed data resource pool. This supports integrated management of various data types, including structured, unstructured, and graph data. Through unified data encapsulation and interface definition, it provides data exchange services and facilitates high-speed querying and sharing of aggregated data via information compression and automatic data synchronization mechanisms. This invention is suitable for high-speed data acquisition in large-sample spacecraft system simulation processes and is compatible with efficient sharing, storage, and management methods and platforms for various types of massive data. It provides reliable support for data mining, evaluation, and decision support in large-sample spacecraft system simulation experiments. Attached Figure Description

[0057] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly described below. Obviously, the drawings described below are merely some embodiments of the present invention, and those skilled in the art can obtain other drawings based on these drawings without creative effort.

[0058] Figure 1 A flowchart of a high-speed data acquisition and storage method for large-sample spacecraft system simulation according to an embodiment of the present invention is shown;

[0059] Figure 2A schematic diagram of the data aggregation process according to an embodiment of the present invention is shown;

[0060] Figure 3 A schematic diagram of a multi-SQL dialect engine framework according to an embodiment of the present invention is shown;

[0061] Figure 4 A schematic diagram of a unified SQL compiler architecture for multi-storage engine databases is shown in an embodiment of the present invention. Detailed Implementation

[0062] The description of the embodiments in this specification should be taken in conjunction with the accompanying drawings, which should form part of the complete specification. In the drawings, the shape or thickness of the embodiments may be exaggerated and may be indicated in a simplified or convenient manner. Furthermore, parts of the various structures in the drawings will be described separately; it is worth noting that elements not shown in the figures or not described in words are in a form known to those skilled in the art.

[0063] The descriptions of the embodiments herein, including any references to directions and orientations, are for ease of description only and should not be construed as limiting the scope of the invention. The following description of preferred embodiments involves combinations of features, which may exist independently or in combination; the invention is not particularly limited to the preferred embodiments. The scope of the invention is defined by the claims.

[0064] like Figure 1 As shown, the present invention provides a high-speed data acquisition and storage method for large-sample spacecraft system simulation, comprising the following steps:

[0065] Step S1, Data Collection: Real-time acquisition of data from large-sample simulation and simulation processes, as well as offline connection and synchronization of data from other databases.

[0066] (1) Real-time acquisition of large sample spacecraft system simulation data

[0067] In the real-time data acquisition process of large-sample spacecraft system simulations, each simulation step involves numerous model output parameters. Furthermore, during ultra-real-time simulations with massive sample sizes, each parameter outputs data concurrently for extended periods, resulting in a massive dataset generated per second within the simulation engine. This can easily lead to problems such as data corruption, loss, service crashes, and inability to store data in real-time. Therefore, it is urgent to solve the challenge of real-time acquisition and storage of massive, high-concurrency, large-sample simulation data, and to achieve efficient and stable data acquisition.

[0068] 1) Problems with data disorder and loss during the collection process.

[0069] Because the simulation engine uses the message middleware Kafka as the output method for simulation process data, it immediately starts a thread upon receiving a request to start a simulation experiment, listening to the corresponding topic in different consumer groups to receive simulation data. Kafka's characteristic is that message consumption is tailored to different consumer groups. When frequently starting simulation experiments or conducting large-sample simulation experiments, the simulation engine may reuse the same consumer topic. Newly started consumer threads are highly likely to collect data from the previous simulation process, leading to data corruption. Furthermore, while a message queue clearing operation can be performed each time a consumer thread starts to prevent data corruption, the simulation may have already started by then, potentially resulting in data loss during the current simulation process.

[0070] To address the above issues, data aggregation is performed using persistent methods within the consumer thread, specifically including:

[0071] Based on the received simulation experiment start request, perform a persistent memory check of the consumption topic to determine whether there is a consumption thread for the topic corresponding to the simulation experiment start request that has been started.

[0072] If there is no consumer thread that corresponds to the topic of the simulation experiment start request, a new consumer thread is created to listen to the topic of the simulation experiment start request, and the simulation experiment identifier of the simulation experiment start request is set on the newly created consumer thread.

[0073] If there is a consumer thread for the topic corresponding to the simulation experiment start request, insert the simulation experiment identifier of the simulation experiment start request into the priority queue of the consumer thread for the corresponding topic.

[0074] The consumer thread consumes data in a loop, associating the received data with the simulation experiment identifier;

[0075] Based on the end marker received by the simulation engine from the consumer thread, the next simulation experiment marker to be stored is retrieved from the priority queue, and the subsequently received data is associated with the new simulation experiment marker.

[0076] Upon receiving a simulation start request, the system first checks the resident (pooled) memory of the consumer topic to determine if a thread is already listening to it. If no thread is currently listening to the topic within the running consumer threads, a new consumer thread is created to listen to that topic, and a unique identifier for the current simulation process data is set. If a thread is found to be listening to the topic within the running consumer threads, the unique identifier for the current simulation process data is inserted into the thread's priority queue.

[0077] The unique identifier that has been inserted into the priority queue may correspond to a consumer thread that has not yet completed its previous consumption. By inserting the unique identifier into the priority queue, the thread can consume in a loop. If the thread encounters a simulation engine production completion identifier, it will retrieve the next simulation experiment identifier to be stored from the priority queue and store the subsequent data in Kafka into that simulation experiment identifier.

[0078] The above thread reuse techniques ensure zero blocking during the consumption process and prevent data corruption and loss during experiments.

[0079] 2) The data collection service is prone to crashing.

[0080] The original data acquisition system would launch a new thread to process the simulation data each time it received a simulation start signal. This frequent thread creation in high-concurrency scenarios such as large samples, multiple schemes, and multiple operators failed to consider the maximum processing capacity of the physical machine. Furthermore, the thread termination status was unclear, and expired threads were not automatically destroyed. These factors led to excessive resource usage and a high risk of service crashes.

[0081] To address the aforementioned issues, real-time data acquisition is performed on large-sample simulation and deduction process data, and optimization is achieved through a data acquisition service process optimization method. This optimization method specifically includes:

[0082] A. Conduct an in-depth performance evaluation of the existing physical machines to determine their limits, and configure a tuning thread pool with a fixed upper limit based on the evaluation results. The tuning thread pool is used without affecting other services deployed on the same machine. The data collection process is handled within the limits of the tuning thread pool, and no new threads are created directly.

[0083] B. Data collection using resident methods within the consumer thread: When a simulation signal is received, a pre-judgment is performed. If the topic to be monitored is already being processed by a thread, the simulation experiment flag is added to the existing thread's pending priority queue, and the thread can then process it itself.

[0084] C. Automatically destroy the threads of the data acquisition service according to the thread destruction conditions met by the simulation process data; wherein, the thread destruction conditions are divided into two categories: 1. Normally receiving the simulation end flag; 2. Not normally receiving the simulation end flag.

[0085] When the simulation end flag is received normally, the simulation engine may reuse the topic to push data for the next simulation process. Therefore, all threads of the acquisition service are blocked and waited for a preset waiting time (e.g., 10 minutes). If the waiting time exceeds the preset time, the resources held by all threads of the acquisition service are released and the threads are rolled back to the tuning thread pool. The consumer thread is destroyed according to the preset maximum timeout time of the thread in the tuning thread pool.

[0086] If the simulation end marker is not received normally, the situation becomes much more complex, with possible causes including: the simulation task being paused, the model exiting abnormally, or the simulation engine exiting abnormally. Based on the simulation engine's ability to reuse consumed topics and the ordered nature of simulation process data, a low-overhead simulation step counter is maintained within each thread.

[0087] If the simulation end flag is not received normally, and the corresponding situation is a paused simulation task, only a low-overhead step counter is maintained in the consumer thread, and all other thread resources of the acquisition service are returned to the corresponding tuning thread pool. In the event of a paused simulation task, the acquisition thread will inevitably experience thread blocking, but in this case, the acquisition service will only maintain a low-overhead step counter and return all other resources to the corresponding pool, without consuming additional resources. As the simulation task continues and the consumer thread listens for the acquired data, the acquisition service continues data acquisition by calling thread resources from the tuning thread pool.

[0088] If the simulation end marker is not received normally, and the corresponding situation is either a model exit or a simulation engine exit, a step counter is maintained in the consumer thread. The number of steps in the newly received process data is compared with the number of steps in the step counter. If they do not match, the step counter is cleared, and the consumer thread is restored to normal. Otherwise, all threads of the data acquisition service are restored to running state, and the sample acquisition work for this simulation continues. When the above situation occurs, the threads of the data acquisition service also maintain minimal resource consumption. When the topic is reactivated, the number of steps in the simulation experiment is compared with the step counter maintained in the thread. If the number of steps in the newly received process data does not match the counter, it is considered that the next simulation has started, the thread state automatically returns to normal, and the next simulation sample data acquisition begins.

[0089] D. A new daemon process is automatically created upon normal startup of the data collection service. The data collection service and the daemon process keep each other alive. The daemon process monitors whether the data collection service is working properly. If it exits abnormally, it will be automatically restarted. The data collection service also monitors the daemon process. If the daemon process exits abnormally, it will be automatically restarted.

[0090] Through the above process, resources become controllable and services are less prone to crashes.

[0091] 3) Real-time inbound processing is not possible, resulting in inbound delays.

[0092] During the simulation process, the data acquisition service collected a lot of irrelevant data, resulting in the collection of too much useless data. In addition, since many existing data storage drivers are developed by various manufacturers, database connections use non-standard configurations and cannot use existing standardized database connection pooling technologies, resulting in the need for database connection adaptation. At the same time, the single-threaded collection, parsing and saving process can cause serious lag in the acquisition service or data loss during the storage of large amounts of data at one time.

[0093] To address the aforementioned issues, a data import optimization mechanism is implemented to optimize the import of real-time collected data. This data import optimization mechanism includes:

[0094] a. Design of the preprocessing stage for data entry: The collected real-time data is sent to the preprocessing thread pool. The preprocessing thread in the preprocessing thread pool performs preliminary parsing on the received data. The preliminary parsing only parses the port and does not parse the specific content of the protocol. According to the preset port mapping table, the port of the received data is pre-judged. If the port of the received data is related to the evaluation function or the playback function, the specific protocol content of the received data is parsed. Otherwise, it is directly packaged and entered into the database.

[0095] After data collection, the data enters a preprocessing thread pool to perform preliminary parsing of the received process data. Only the ports are parsed, not the specific content of the protocol. A configurable port mapping table is established to pre-judge the ports. Only ports involved in evaluation or playback are parsed for specific protocol content; otherwise, no parsing is performed, which greatly accelerates the processing speed.

[0096] b. Set up a database connection pool, which is used to configure standard database connections and / or non-standard database connections according to the data entry requests in the data acquisition service;

[0097] For non-standardized data storage drivers, a new database connection pool is designed to supplement and improve the standard database connection pool, enabling multi-threaded sharing of database connections. This effectively solves the problem that the system starts a database connection every time it receives a simulation start signal, resulting in a huge waste of resources and a serious impact on startup speed.

[0098] c. Set up a storage thread pool for multi-threaded data storage. The data output from the preprocessing thread pool is written to the database in batches using multiple threads. By using the storage thread pool to break down the massive amount of data to be stored into smaller parts and writing them to the database in batches using multiple threads, storage efficiency is greatly improved.

[0099] (2) Offline data import

[0100] For data that does not require real-time acquisition in spacecraft system simulation experiments, the necessary spacecraft performance parameters, remote sensing images, scene text data, etc., are mainly accessed and synchronized in batches from external databases through offline data synchronization, and then stored in the corresponding relational database, file system, and graph system. Further details are omitted here.

[0101] Step S2, High-speed data storage management: Generate database operations for multiple SQL dialects through a multi-SQL dialect engine framework, perform integrated management of the collected data, and form a distributed data resource pool;

[0102] A multi-SQL dialect engine framework is used to manage multiple storage engines in a unified manner, forming a distributed data resource pool that supports integrated management of structured, unstructured, and graph data.

[0103] In the current spacecraft system simulation process, structured data, semi-structured data, and unstructured data are stored in different databases. Structured data is mainly stored in databases of different simulation experimental environments such as MySQL, Oracle, DM, and Kingbase. Semi-structured data in row-column format such as json / xml / csv / xls / xlsx, as well as unstructured data in formats such as documents, BMP images, audio, and video, are stored in a distributed file system, which uses a set of rules to name, store, and retrieve files.

[0104] When performing specific database operations, different databases extend standardized SQL statements to meet specific needs; these extensions can be called SQL dialects. In conducting spacecraft system simulation experiments and evaluation analyses, to achieve unified management and SQL access across multiple database types, a unified SQL parsing and model mapping mechanism needs to be established to support the use of multiple database dialects.

[0105] To address this, the present invention provides an engine framework that supports multiple SQL dialects, enabling unified management and access to databases with multiple dialect storage engines.

[0106] The multi-SQL dialect engine framework mainly consists of three layers. The top layer is a unified SQL compiler for multiple engines, capable of converting standard SQL statements between different SQL dialects, generating database operation execution plans, and achieving unified SQL language compilation. The second layer is the SQL engine, which integrates multiple different database engines such as DM and Kingbase. It can receive the database operation execution plans generated by the unified SQL compiler and perform corresponding database operations. The third layer is the storage layer, used to store data from different databases. The database type of the storage layer corresponds to the database type of the SQL engine.

[0107] A unified SQL compiler for multi-engine databases, the compilation process is as follows:

[0108] Step S211: For database operations targeting different databases, generate a unified SQL statement, perform lexical analysis on the unified SQL statement, and convert it into a word sequence; perform syntax checking on the unified SQL statement based on metadata information and multi-storage information, and construct an abstract syntax tree based on the word sequence based on the result of passing the syntax check.

[0109] Step S212: Perform constant propagation optimization on the abstract syntax tree;

[0110] Step S213: Determine whether the unified SQL statement is a stored procedure. If yes, proceed to step S214; otherwise, proceed to step S217.

[0111] Step S214: Perform stored procedure semantic analysis on the unified SQL statement; the stored procedure semantic analysis includes semantic analysis of SQL in various dialects and control flow semantic analysis;

[0112] Step S215: Based on the results of control flow semantic analysis, optimize the control flow of the unified SQL statement;

[0113] Step S216: Perform cursor parallelization on the unified SQL statement that has completed control flow optimization to generate a physical execution plan;

[0114] Step S217: Perform semantic analysis of various dialects on the input unified SQL statement;

[0115] Step S218: Optimize the logical execution plan and transform the subqueries of the unified SQL statement that has completed semantic analysis of various dialect SQL statements, and generate a physical execution plan.

[0116] Lexical and syntactic analysis, implemented through the lexer and parser in the SQL compiler, is the process of converting SQL character sequences into token sequences. The lexer typically exists as a function, invoked by the parser. The parser uses meta-information and multi-storage abstraction information to perform syntax checks on the SQL and constructs an abstract syntax tree composed of the input tokens.

[0117] Ordinary semantic analysis examines the context-dependent properties and types of structurally correct source code. Semantic analysis checks for semantic errors and collects type information for the code generation phase. Type checking within semantic analysis verifies whether each operator has operands allowed by the language specification; if not, the compiler reports an error. However, stored procedures also require control flow semantic analysis, transforming them into several execution blocks executable in a distributed system, and using intermediate representations to record the dependencies between these blocks. Semantic analysis of the intermediate representation involves analyzing and recording its attributes, including parameters, variables, and types. This transforms the stored procedure into several execution blocks executable in a distributed system. When executing a stored procedure, a stack-based function execution framework is first created using the generated execution blocks and their dependencies. The attributes of the intermediate representation are allocated in the stack of the function execution framework; then, the execution blocks are executed one by one using the function execution framework. If there is an execution block for a standard SQL statement, the process analyzes whether the intermediate representation of the standard SQL statement uses variables from other intermediate representations. If so, the variable's name and position in the standard SQL statement are recorded. When the execution block of the standard SQL statement is reached, the variable's value is first obtained, replaced with a constant, and then stored in the standard SQL statement execution block. Finally, it is sent to the distributed system's computing engine for execution.

[0118] Logical execution plan optimization includes CSE, Column prune, Partition prune, and PPD. Unoptimized code often has poor execution efficiency. For example, some common subqueries need to be deleted (CSE); unused columns don't need to be read during table scans (Column prune); data from filtered partitions doesn't need to be read at all (Partition prune); some filtering conditions, especially those in implicit join WHERE clauses, should be pushed down to the table scan process as much as possible (PPD), so the engine can improve query efficiency through indexes or table scan filtering; constant propagation calculates values ​​that can be precisely determined at compile time, avoiding redundant calculations at runtime.

[0119] Control flow-based stored procedure optimization: Compared to ordinary SQL, stored procedures have complex control flow, function calls, loops, and other logic. Optimizing these execution blocks differs from optimizing traditional SQL logic execution plans and requires control flow-based optimization methods. Control flow optimization includes dead code removal (code that the control flow cannot reach); removal of redundant computations; necessary function inwards to reduce parameter passing overhead; and deep optimization of loops in the code, such as loop invariant hoisting and parallelization.

[0120] Physical execution plan generation is the process of generating a physical execution plan from an optimized logical execution plan, which can be executed in a distributed environment.

[0121] Step S3, Data Exchange: Through unified data encapsulation and interface definition, data exchange services are provided, and high-speed data query and sharing are achieved based on information pressure mechanism and automatic data synchronization mechanism.

[0122] In step S3, during the data query process, an information push-down mechanism is set up, specifically including:

[0123] Step S31: Generate SQL commands based on the received data exchange instructions;

[0124] Step S32: Compile the SQL instructions using the multi-engine unified SQL compiler to generate a database execution plan;

[0125] Step S33: Optimize and identify database operation functions in the database execution plan, and perform optimal operation matching on the identified database operation functions;

[0126] Step S34: Based on the optimal operation matching result, optimize the database execution plan and replace the operation function with the optimal operation function;

[0127] Step S35: Return the database operation function result according to the optimal operation function.

[0128] In step S3, before data exchange, multi-database data synchronization based on an automatic data synchronization mechanism is also included, specifically including:

[0129] Step S301: Collect and monitor the Binlog database using Flink CDC, and push the data change operations to the Kafka message queue in real time;

[0130] Different databases or services can achieve data consistency by listening to data change operations in the Kafka message queue.

[0131] Step S302: Based on the data change operations in the Kafka message queue, clean, transform, adapt the format, and filter the data, and asynchronously synchronize the processed data to other databases or storage engines.

[0132] After achieving unified storage for different types of databases and supporting unified SQL queries, to better support the high-speed data query needs of various evaluation and analysis in spacecraft system simulation experiments, the structured data managed by the database can be encapsulated into data products, including databases, tables, datasets, and data records at various granularities. After encapsulation, data services can be published, allowing spacecraft system simulation experiment evaluation and analysis tools to obtain the required data content.

[0133] When performing data queries for spacecraft system simulation and evaluation, the amount of stored data is enormous, requiring rapid retrieval of necessary indicators and data from the database to meet increasingly demanding real-time evaluation requirements. This invention designs an information compression mechanism and an automatic data synchronization mechanism to improve the efficiency of data querying and access.

[0134] When performing data queries, different storage engines and databases have their own characteristics and advantages. After SQL compilation, as much query information as possible can be parsed out and pushed down to the storage for analysis. For example, HBase supports fast retrieval by Rowkey, so the Filter conditions in the SQL are identified as HBase Rowkey conditions and pushed down to the storage for fast retrieval by Rowkey. Another example is Elasticsearch, which supports fast aggregation. Its underlying storage aggregation operations are more efficient than the upper-layer computation engine. In this case, the aggregation (GroupBy) information in the SQL can be pushed down to Elasticsearch to generate the corresponding aggregation operations. These aggregation operations are then performed in advance during data retrieval, thereby improving data query efficiency.

[0135] Therefore, in the process of conducting large-sample spacecraft system simulation experiments and evaluations, it is also necessary to store the same data in different formats to meet the needs of different scenarios. When users query, the corresponding storage format will be selected according to the application scenario of the SQL, thereby improving the overall query efficiency.

[0136] By collecting and monitoring Binlog logs in the Flink CDC database, changes made to the data are pushed to the Kafka message queue in real time. Simultaneously, after cleaning, transforming, formatting, and filtering the data, it is asynchronously synchronized to other databases or storage engines. For example, during spacecraft system simulation experiments and evaluations, the collected simulation data is stored in a MongoDB database. To improve query efficiency, some frequently used fields, such as spacecraft details, can be synchronized to more efficient search engines like Elasticsearch. Data queries are then performed in Elasticsearch using techniques such as word segmentation and pre-indexing, inverted indexes, sharding and parallel processing, and columnar storage. If some frequently used fields are not synchronized to Elasticsearch, the indexes of related fields are queried in Elasticsearch, linking the indexes to the experimental data stored in the MongoDB database. This method significantly improves the speed of data retrieval and querying.

[0137] According to one aspect of the present invention, an electronic device is provided, comprising: one or more processors, one or more memories, and one or more computer programs; wherein the processor is connected to the memory, and the one or more computer programs are stored in the memory; when the electronic device is running, the processor executes the one or more computer programs stored in the memory, so that the electronic device performs a high-speed data acquisition and storage method for large-sample spacecraft system simulation as described in any of the above technical solutions.

[0138] The processor can be a central processing unit (CPU), but it can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0139] According to one aspect of the present invention, a computer-readable storage medium is provided for storing computer instructions, which, when executed by a processor, implement a high-speed data acquisition and storage method for large-sample spacecraft system simulation as described in any of the above technical solutions.

[0140] Computer-readable storage media can include any medium capable of storing or transmitting information. Examples of computer-readable storage media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, fiber optic media, radio frequency (RF) links, and so on. Code segments can be downloaded via computer networks such as the Internet and intranets.

[0141] This invention relates to a high-speed data acquisition and storage method, electronic device, and storage medium for large-sample spacecraft system simulation. The method includes the following steps: Step S1, Data Aggregation: Real-time acquisition of data from the large-sample simulation process, and offline synchronization of data from other databases; Step S2, High-Speed ​​Data Storage Management: Generating database operations for multiple SQL dialects through a multi-SQL dialect engine framework, and managing the aggregated data in an integrated manner to form a distributed data resource pool; Step S3, Data Exchange: Defining data exchange services through unified data encapsulation and interface definitions, and performing high-speed data query and sharing based on information pressure and automatic data synchronization mechanisms. This invention addresses the real-time acquisition and offline integration of simulation process data with other external data in large-sample spacecraft system simulation experiments, supporting integrated storage of structured, semi-structured, and unstructured data, as well as efficient and fast unified querying.

[0142] Furthermore, it should be noted that the present invention can be provided as a method, apparatus, or computer program product. Therefore, embodiments of the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Moreover, embodiments of the present invention can take the form of a computer program product implemented on one or more computer-usable storage media containing computer-usable program code.

[0143] Embodiments of the present invention are described with reference to flowchart illustrations and / or block diagrams of methods, terminal devices (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0144] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The functions specified in one or more boxes. These computer program instructions may also be loaded onto a computer or other programmable data processing terminal equipment to cause a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0145] It should also be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.

[0146] Finally, it should be noted that the above description represents a preferred embodiment of the present invention. It should be pointed out that although preferred embodiments have been described, those skilled in the art, once they understand the basic inventive concept of the present invention, can make various improvements and modifications without departing from the principles described herein. These improvements and modifications should also be considered within the scope of protection of the present invention. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the embodiments of the present invention.

Claims

1. A high-speed data acquisition and storage method for large-sample spacecraft system simulation, characterized in that, Includes the following steps: Step S1, Data Collection: Real-time acquisition of data from large-sample simulation and simulation processes, as well as offline connection and synchronization of data from other databases; Step S2, High-speed data storage management: Generate database operations for multiple SQL dialects through a multi-SQL dialect engine framework, perform integrated management of the collected data, and form a distributed data resource pool; Step S3, Data Exchange: Through unified data encapsulation and interface definition, data exchange services are provided, and high-speed data query and sharing are achieved based on information pressure mechanism and automatic data synchronization mechanism.

2. The high-speed data acquisition and storage method for large-sample spacecraft system simulation according to claim 1, characterized in that, In step S1, the large-sample simulation process data is collected in real time and optimized using a data collection service process optimization method; wherein, the data collection service process optimization method specifically includes: A. Conduct an in-depth performance evaluation of the existing physical machines to determine their limits, and configure a tuning thread pool with a fixed upper limit based on the evaluation results. The tuning thread pool shall be used without affecting other services deployed on the same machine, and the data collection process shall be processed within the limits of the tuning thread pool. B. Use persistent methods within the consumer thread to aggregate data; C. Automatically destroy the threads of the data acquisition service based on the thread destruction conditions met by the simulation process data; The thread destruction conditions are divided into two categories: normal reception of the simulation end flag and abnormal reception of the simulation end flag. When the simulation ends normally, all threads of the acquisition service are blocked and waited for a preset time. If the preset time is exceeded, the resources held by all threads of the acquisition service are released and the threads are rolled back to the tuning thread pool. If a thread exceeds the preset maximum timeout in the tuning thread pool, the thread is destroyed. If the simulation end flag is not received normally and the corresponding situation is that the simulation task is paused, only a low-overhead step counter is maintained in the consumer thread, and the other thread resources of the acquisition service are put back into the corresponding tuning thread pool. If the simulation end flag is not received normally, and the corresponding situation is that the model or simulation engine exits abnormally, a step counter is maintained in the consumer thread. The number of steps in the newly received process data is compared with the number of steps in the step counter. If the two do not match, the number of steps in the step counter is cleared to zero and the state of the consumer thread is restored to normal. Otherwise, all threads of the data acquisition service are restored to the running state and the sample acquisition work of this simulation is continued. D. A new daemon process is automatically created upon normal startup of the data collection service. The data collection service and the daemon process keep each other alive. The daemon process monitors whether the data collection service is working properly. If it exits abnormally, it is automatically restarted. The data collection service also monitors the daemon process. If the daemon process exits abnormally, it is automatically restarted.

3. The high-speed data acquisition and storage method for large-sample spacecraft system simulation according to claim 2, characterized in that, The method of using persistent methods within the consumer thread for data aggregation specifically includes: Based on the received simulation experiment start request, perform a persistent memory check of the consumption topic to determine whether there is a consumption thread for the topic corresponding to the simulation experiment start request that has been started. If there is no consumer thread that corresponds to the topic of the simulation experiment start request, a new consumer thread is created to listen to the topic of the simulation experiment start request, and the simulation experiment identifier of the simulation experiment start request is set on the newly created consumer thread. If there is a consumer thread for the topic corresponding to the simulation experiment start request, insert the simulation experiment identifier of the simulation experiment start request into the priority queue of the consumer thread for the corresponding topic. The consumer thread consumes data in a loop, associating the received data with the simulation experiment identifier; Based on the end marker received by the simulation engine from the consumer thread, the next simulation experiment marker to be stored is retrieved from the priority queue, and the subsequently received data is associated with the new simulation experiment marker.

4. The high-speed data acquisition and storage method for large-sample spacecraft system simulation according to claim 2, characterized in that, In step S1, the real-time collected data is optimized for storage through a data storage optimization mechanism, which includes: a. Design of the pre-processing stage for data entry: The collected real-time data is sent to the pre-processing thread pool. The pre-processing thread in the pre-processing thread pool performs preliminary parsing on the received data. The preliminary parsing only parses the port and does not parse the specific content of the protocol. According to the preset port mapping table, the port of the received data is pre-judged. If the port of the received data is related to the evaluation function or the playback function, the specific protocol content of the received data is parsed. Otherwise, it is directly packaged and entered into the database. b. Set up a database connection pool, which is used to configure standard database connections and / or non-standard database connections according to the data entry requests in the data acquisition service; c. Set up a storage thread pool for multi-threaded data storage, and write the data output by the preprocessing thread pool into the database in batches using multiple threads.

5. The high-speed data acquisition and storage method for large-sample spacecraft system simulation according to claim 1, characterized in that, In step S2, the multi-SQL dialect engine framework includes: A unified SQL compiler for multi-engine databases, used to convert between standard SQL statements and different SQL dialects, and generate database operation execution plans; The SQL engine integrates multiple database engines and is used to receive the database operation execution plan and perform corresponding database operations. The storage layer is used to store data from different databases; the database type of the storage layer corresponds to the database type of the SQL engine.

6. The high-speed data acquisition and storage method for large-sample spacecraft system simulation according to claim 5, characterized in that, In step S2, the compilation process of the unified SQL compiler for multi-engine databases specifically includes: Step S211: For database operations targeting different databases, generate a unified SQL statement, perform lexical analysis on the unified SQL statement, and convert it into a word sequence; perform syntax checking on the unified SQL statement based on metadata information and multi-storage information, and construct an abstract syntax tree based on the word sequence based on the result of passing the syntax check. Step S212: Perform constant propagation optimization on the abstract syntax tree; Step S213: Determine whether the unified SQL statement is a stored procedure. If yes, proceed to step S214; otherwise, proceed to step S217. Step S214: Perform stored procedure semantic analysis on the unified SQL statement; the stored procedure semantic analysis includes semantic analysis of various dialect SQL and control flow semantic analysis; Step S215: Based on the results of control flow semantic analysis, optimize the control flow of the unified SQL statement; Step S216: Perform cursor parallelization on the unified SQL statement that has completed control flow optimization to generate a physical execution plan; Step S217: Perform semantic analysis of various dialects on the unified SQL statement; Step S218: Optimize the logical execution plan and transform the subqueries of the unified SQL statement that has completed semantic analysis of various dialect SQL statements, and generate a physical execution plan.

7. The high-speed data acquisition and storage method for large-sample spacecraft system simulation according to claim 1, characterized in that, In step S3, during the data query process, an information pushdown mechanism is set up, specifically including: Step S31: Generate SQL commands based on the received data exchange instructions; Step S32: Compile the SQL instructions using the multi-engine unified SQL compiler to generate a database execution plan; Step S33: Optimize and identify database operation functions in the database execution plan, and perform optimal operation matching on the identified database operation functions; Step S34: Based on the optimal operation matching result, optimize the database execution plan and replace the operation function with the optimal operation function; Step S35: Return the database operation function result according to the optimal operation function.

8. The high-speed data acquisition and storage method for large-sample spacecraft system simulation according to claim 7, characterized in that, In step S3, before data exchange, multi-database data synchronization based on an automatic data synchronization mechanism is also included, specifically including: Step S301: Collect and monitor the Binlog database using Flink CDC, and push the data change operations to the Kafka message queue in real time; Step S302: Based on the data change operations in the Kafka message queue, clean, transform, adapt the format, and filter the data, and asynchronously synchronize the processed data to other databases or storage engines.

9. An electronic device, characterized in that, include: One or more processors, one or more memories, and one or more computer programs; wherein the processor is connected to the memory, and the one or more computer programs are stored in the memory, and when the electronic device is running, the processor executes the one or more computer programs stored in the memory to cause the electronic device to perform the high-speed data acquisition and storage method for large-sample spacecraft system simulation as described in any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that, Used to store computer instructions, which, when executed by a processor, implement a high-speed data acquisition and storage method for large-sample spacecraft system simulation as described in any one of claims 1 to 8.

Images