Data analysis and report generation system based on asynchronous processing
By designing a data analysis and report generation system based on asynchronous processing, the problems of time-consuming manual queries and insufficient high-concurrency processing capabilities in remote sensing satellite test data analysis were solved, and efficient and accurate data interpretation and automatic report generation were achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- CHINA ACADEMY OF SPACE TECHNOLOGY
- Filing Date
- 2025-12-15
- Publication Date
- 2026-05-01
AI Technical Summary
Existing technologies rely on manual queries in the analysis of remote sensing satellite test data, which is time-consuming and error-prone. Existing software has insufficient concurrent processing capabilities under high concurrent loads, making it difficult to meet the needs of large-scale, long-term, high-concurrency, and low-latency data access.
Design a data analysis and report generation system based on asynchronous processing, including a front-end display module, a database connection module, an asynchronous processing module, a database storage module, an EDS interface module, a data comparison module, and a document generation module. The front-end interaction is implemented using the Bootstrap and Thymeleaf frameworks, the database connection pool is managed by the Spring Boot framework, a custom thread pool using ThreadPool Executor is used for asynchronous processing, and reports are generated through Apache POI.
It improves the efficiency and accuracy of telemetry interpretation, reduces labor costs, enhances parameter query efficiency, solves the problems of insufficient concurrent processing capacity and report generation delay in the existing system, and realizes efficient data access and automatic report generation.
Smart Images
Figure CN121958342A_ABST
Abstract
Description
A data analysis and report generation system based on asynchronous processing Technical Field
[0001] This invention relates to a data analysis and report generation system based on asynchronous processing, used for statistical analysis and evaluation of ground test data at various stages of spacecraft, and automatically generating reports, belonging to the field of spacecraft testing. Background Technology
[0002] Remote sensing satellites adhere to the methodology of "everything is verified through ground testing," undergoing a complete ground testing phase before launch to verify the satellite's hardware performance, software functions, and the correctness of its technical specifications. During ground testing, a large amount of satellite telemetry data is generated. This data is characterized by its vast quantity, accuracy, and completeness, and can directly or indirectly reflect the satellite's functional performance, thus possessing significant value.
[0003] Currently, the analysis of remote sensing satellite test data relies on manual or simple statistical software. However, the manual mode or existing software has the following problems: (1) It relies on manual querying, which is not only time-consuming, but also inevitably causes data querying to be missing or incorrect due to human reasons, resulting in incomplete or incorrect data analysis; (2) The current software does not utilize mature data source interfaces, but adopts manual configuration of the correct range before querying, which is not user-friendly; (3) The existing query and report generation system only supports MongoDB database. When dealing with long-term complex processing queries and high concurrency loads, there are technical bottlenecks such as insufficient concurrent processing capacity, high query response latency, unbalanced resource utilization, and expansion of intermediate result sets, which make it difficult to meet the data access needs of large scale, long period, high concurrency, and low latency. Summary of the Invention
[0004] The technical problem solved by this invention is to overcome the shortcomings of the prior art and provide a data analysis and report generation system based on asynchronous processing, which improves the efficiency and accuracy of telemetry interpretation throughout the entire process, visualizes the query and analysis results, and saves time and manpower costs.
[0005] The technical solution of this invention is: a data analysis and report generation system based on asynchronous processing, comprising: a front-end display module, a database connection module, an asynchronous processing module, a database storage module, an EDS interface module, a data comparison module, and a document generation module; wherein: the front-end display module is used for direct interaction between the user and the system, allowing the user to directly input query information and an .xlsx file for querying; the front-end display module formats the query information and generates corresponding connection requests and batch query requests, which are sent to the database connection module; the .xlsx file input by the user is sent to the EDS interface module; the EDS interface module extracts valid information from the .xlsx file input by the front-end display module, generates a query telemetry Map set and sends it to the database connection module, and simultaneously generates a telemetry theoretical value range Map and sends it to the data comparison module; the database connection module judges the front-end display module's data based on the query telemetry Map set sent by the EDS interface module. After the connection request from the front-end display module is valid, it sends the batch query request to the asynchronous processing module. The asynchronous processing module converts the batch query request into an asynchronous query request set and sends it to the database storage and query module for querying. It also merges the normal value result set and the outlier result set from the data comparison module and sends them to the document generation module. The database storage and query module stores the telemetry data transmitted from the satellite and processes it according to the asynchronous query request set, generating a query result set which is sent to the data comparison module. The data comparison module compares the query result set with the telemetry theoretical value range Map, classifies and organizes the generated comparison results, obtains the normal value result set and the outlier result set, and sends them to the asynchronous processing module. The document generation module generates a telemetry query result table in a Word document from the results output by the asynchronous processing module, adds interpretation results to the generated results, generates a Word document report, and sends it to the front-end display module for users to download and use.
[0006] Preferably, the front-end display module uses a framework combining Bootstrap and Thymeleaf to build the page, allowing users to directly input query information including: database IP and port number, model name, query start time, and query end time.
[0007] Preferably, the database connection module judges the connection request sent by the front-end display module: if the connection request is valid, the batch query request is sent to the asynchronous processing module; if the connection request is invalid, the query failure result is returned to the front-end display module.
[0008] Preferably, the database storage and query module includes a data storage unit and a data query unit, wherein: the data storage unit stores the telemetry data transmitted by the satellite in real time; the data query unit is used to receive asynchronous query request sets, query telemetry parameters in the data storage unit, organize the queried data results into a query result set, and send it to the data comparison module.
[0009] Preferably, the valid information extracted from the .xlsx file by the EDS interface module includes: telemetry code, telemetry name, and normal value range; a telemetry Map set is generated based on the telemetry code and telemetry name; and a telemetry theoretical value range Map is generated based on the normal value range.
[0010] Preferably, the document generation module includes a document generation unit and a result return unit. The document writing unit is used to fill the results generated by the asynchronous processing module into a Word document table and store it on the server as a document resource. The table includes a normal value result table and an abnormal value result table. The normal value result table includes: telemetry code, telemetry name, telemetry range, and interpretation conclusion. The abnormal value result table includes: telemetry code, abnormal value, and the time corresponding to the abnormal value. The result return unit returns the path of the Word document to the front-end display module, providing users with a download option.
[0011] Preferably, the database connection module uses the Spring Boot framework, implements a database connection pool through DataSource and HikariCP, and configures it in application.properties.
[0012] Preferably, the database storage and query module supports storing relevant telemetry data during the ground testing phase, including the TDengine database and the MongoDB database. The telemetry parameter fields in the MongoDB database include: time, telemetry hexadecimal source code, and telemetry value.
[0013] Preferably, the asynchronous processing module uses a custom thread pool called java.util.concurrent.ThreadPool Executor; the results generated by the data comparison module are sent to the asynchronous processing module in the form of a HashMap collection.
[0014] Preferably, the document generation module is based on the Apache POI open-source library, loads templates through the XWPFDocument class, and can traverse paragraphs and cells to return .docx files.
[0015] Compared with the prior art, the present invention has the following advantages: (1) It does not rely on manual query, takes less time, and can reduce the problem of incomplete or incorrect data statistics caused by manual query.
[0016] (2) The data source can be directly obtained by utilizing the existing telemetry parameter EDS (Electronic Data Sheet) interface, which improves ease of use. (3) The parameter query efficiency is greatly improved. The asynchronous processing module designed in this invention solves the problems of insufficient concurrent processing capability, response delay and expansion of intermediate result sets in the existing query system. Attached Figure Description
[0017] Figure 1 shows the system software design and implementation architecture of the present invention. Figure 2 shows the document report generation result of the present invention. Detailed Implementation
[0018] This invention designs a data analysis and report generation system based on asynchronous processing. Using a B / S (Browser / Server) architecture, it achieves efficient statistical analysis and visualization of telemetry data from TDengine and MongoDB databases, thereby improving the accuracy and completeness of data interpretation. The specific technical solution is as follows: (I) System Architecture: Based on a B / S architecture design, it has inherent cross-platform advantages and is easy to maintain, effectively supporting simultaneous access and operation by multiple users.
[0019] (II) Front-end Implementation The front-end is implemented using the lightweight frameworks Bootstrap and Thymeleaf. Bootstrap offers rich CSS and JavaScript components, enabling rapid creation of responsive web pages. Thymeleaf, on the other hand, is a modern server-side Java template engine that supports XML-compatible syntax and generates standards-compliant HTML5 code, ensuring page readability and maintainability.
[0020] (III) The backend implementation adopts the Spring Boot framework and MVC architecture, based on database connection pooling and IO data stream read / write operations, to achieve efficient access and processing of telemetry data in TDengine and MongoDB databases. The database connection pool effectively manages and allocates database connections, significantly reducing the overhead of database creation and destruction, and improving system performance. This invention uses connection pooling technology to ensure stable and efficient system operation under high concurrency. IO data stream read / write operations enable efficient data transmission and processing, ensuring data integrity and consistency.
[0021] (iv) Asynchronous Data Analysis: This invention cleans the telemetry data in the database, removes invalid data, and divides the data into normal data and abnormal data. Abnormal values will be recorded with their corresponding time and specific values, while valid data will be recorded. Asynchronous statistical analysis is performed to accelerate data processing efficiency and effectively improve analytical capabilities.
[0022] (v) Report generation: This invention integrates the results of the data analysis in (iv) to generate a data analysis report that conforms to the specifications, which is then available for users to download and use directly.
[0023] The present invention will be further described in detail below with reference to the accompanying drawings: In order to make the objectives, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings.
[0024] This invention provides a B / S system based on asynchronous processing and automatic report generation, as shown in Figure 1. The system includes a front-end display module, a database connection module, an asynchronous processing module, a database storage and query module, an EDS interface module, a data comparison module, and a document generation module. Referring to Figure 1, the system composition diagram shows that the front-end and back-end are independent of each other, and each module is independent of the others. Data flow and interaction are achieved through an MVC architecture and a JAVA IO interface.
[0025] Specifically: The front-end display module is used for direct interaction between users and the system. This module formats the information entered by the user and generates corresponding connection requests and batch query requests, which are then sent to the database connection module. At the same time, this module also sends the input .xlsx file to the EDS interface module.
[0026] The database connection module receives connection requests from the front-end display module, processes the query telemetry Map set sent by the EDS interface module, and feeds back the results to the front-end display module or sends batch query requests to the asynchronous processing module. The EDS interface module simplifies the processing of queried telemetry parameters. It extracts valid information from the ".xlsx" format EDS file input by the front-end display module, generates the queried telemetry Map set, and sends it to the database connection module. Simultaneously, it generates a telemetry theoretical value range Map and sends it to the data comparison module. The asynchronous processing module has two functions: first, it converts batch query requests into an asynchronous query request set and sends it to the database storage and query module to query the database; second, it... The normal value result set and the outlier result set sent by the data comparison module are merged and then sent to the document generation module. The database storage and query module is used to store the telemetry data transmitted from the satellite. This module processes the asynchronous query requests sent by the asynchronous processing module and generates a query result set, which is then sent to the data comparison module. The data comparison module compares the queried telemetry data with the data input from the EDS interface, classifies the comparison results ("normal values" and "outliers"), and then sends them to the asynchronous processing module. The document generation module generates a telemetry query result table in a Word document from the results output by the asynchronous processing module, adds interpretation results to the generated results, generates a Word document report, and sends it to the front-end display module for users to "download" and use.
[0027] The front-end display module allows users to interact directly with the system. Users can enter necessary parameters in the input boxes of the front-end display module, including the queried telemetry file (.xlsx file), database IP and port number, model name, query start time, and query end time. The .xlsx file will be sent to the EDS interface module. Other user-inputted information will generate a connection request and be sent to the database connection module. The EDS interface module extracts valid information from the user-input .xlsx EDS file. It extracts the "telemetry code" and "telemetry name" to generate the queried telemetry Map set and sends it to the database connection module. It also extracts the "normal value range" to generate a telemetry theoretical value range Map and sends it to the data comparison module. The database connection module receives connection requests from the front-end display module, processes them, and generates an "error" field. The system returns two types of results: "error" and "success". If the result is "error", it directly returns "query failed" to the front end. If the result is "success", the batch query requests are sent to the asynchronous processing module. The asynchronous processing module is divided into a query request asynchronous processing unit and a data asynchronous processing unit. The query request asynchronous processing unit converts the received batch query requests into an asynchronous query request set and sends it to the database storage and query module to query the database. The data asynchronous processing unit receives the batch query results generated by the data comparison module, processes and merges them asynchronously, and sends the results to the document generation module. The database storage and query module is divided into a data storage unit and a data query unit. The data storage unit stores satellite telemetry data in real time. The data query unit receives the asynchronous query request set, queries the telemetry parameters in the data storage unit, organizes the queried data results into a query result set, and sends it to the data comparison module.
[0028] The data comparison module receives the telemetry theoretical value range Map input from the EDS interface module and the actual dataset of the query results input from the database storage and query module. This module compares the actual data with the theoretical range and generates normal data & abnormal datasets, which are then sent to the asynchronous processing module.
[0029] The document generation module includes a document generation unit and a result return unit. The document writing unit is used to form a Word document Tables from the results generated by the asynchronous processing module. The Tables include 1) a normal value result table - "Telemetry Code", "Telemetry Name", "Telemetry Range", and "Interpretation Conclusion"; 2) an outlier result table - "Telemetry Code", "Outlier", and "Time Corresponding to Outlier", and stores it on the server as a document resource. The result return unit is used to return the path of the Word document to the front-end display module and provide the user with a "Download" button to obtain it.
[0030] Example: (1) Front-end display module: Based on the input required by the data analysis system, the basic front-end display page is constructed using the Bootstrap+Thymeleaf framework. The necessary input boxes for telemetry parameter query are provided on the user side: EDS file in xlsx format, database IP and port, model code, query start time, and query end time; and the relevant information is processed to generate a connection request and sent to the database connection module.
[0031] (2) EDS interface module: The standardized EDS is used as the input file for data query (".xlsx format"), and the input .xlsx is processed to obtain the queried telemetry Map and telemetry theoretical value range Map.
[0032] (3) Database connection module: Supports multi-user, high-concurrency mode, while reducing database creation and destruction, thus reducing server load. The backend uses the Spring Boot framework, implements a database connection pool through DataSource and HikariCP, and configures application.properties as follows:
[0033] (4) Database storage and query module: It supports storing relevant telemetry data during the ground test phase, including TDengine database and MongoDB database. The telemetry parameter fields of MongoDB database are time, value-y (telemetry hexadecimal source code) and value-s (telemetry value). It also supports the query function of telemetry.
[0034] (5) Asynchronous processing module: The system uses a custom thread pool based on java.util.concurrent.ThreadPoolExecutor to avoid the resource exhaustion risk associated with using a fixed thread pool created by the Executors factory. The core parameters of the thread pool are set as follows:
[0035] This module enables the statistical analysis of specified telemetry parameters over a certain period of time, and also processes outliers and generates reports (outputting the dates of outliers) to aid in troubleshooting.
[0036] (6) Data comparison module: compares, analyzes and classifies the normal value range extracted by EDS and the data result set obtained by database query, and sends the generated result to the asynchronous processing module in the form of a Hashmap collection.
[0037] (7) Document generation module: Based on the Apache POI open source library, the template is loaded through the XWPFDocument class, paragraphs and cells are traversed, and the results that conform to the test summary and interpretation report format are output and returned as .docx files, as shown in Figure 2.
[0038] The contents not described in detail in this specification are existing technologies known to those skilled in the art.
Claims
1. A data analysis and report generation system based on asynchronous processing. Its features include: a front-end display module, a database connection module, an asynchronous processing module, a database storage module, an EDS interface module, a data comparison module, and a document generation module; wherein: the front-end display module is used for direct interaction between the user and the system, allowing the user to directly input query information and an .xlsx file for querying; the front-end display module formats the query information and generates corresponding connection requests and batch query requests, which are sent to the database connection module; the user-input .xlsx file is sent to the EDS interface module; the EDS interface module extracts valid information from the .xlsx file input by the front-end display module, generates a query telemetry Map set and sends it to the database connection module, and simultaneously generates a telemetry theoretical value range Map and sends it to the data comparison module; the database connection module determines the validity of the connection request sent by the front-end display module based on the query telemetry Map set sent by the EDS interface module. Subsequently, the batch query requests are sent to the asynchronous processing module. The asynchronous processing module converts the batch query requests into an asynchronous query request set, which is then sent to the database storage and query module for querying. It also merges the normal value result set and the outlier result set from the data comparison module and sends them to the document generation module. The database storage and query module stores the telemetry data transmitted from the satellite and processes it according to the asynchronous query request set, generating a query result set which is then sent to the data comparison module. The data comparison module compares the query result set with the telemetry theoretical value range Map, categorizes and organizes the generated comparison results, obtains the normal value result set and the outlier result set, and sends them to the asynchronous processing module. The document generation module generates a telemetry query result table in a Word document from the results output by the asynchronous processing module, adds interpretation results to the generated results, generates a Word document report, and sends it to the front-end display module for user download and use.
2. The data analysis and report generation system based on asynchronous processing according to claim 1, characterized in that: The front-end display module uses a framework combining Bootstrap and Thymeleaf to build the page, allowing users to directly input query information including: database IP and port number, model name, query start time, and query end time.
3. The data analysis and report generation system based on asynchronous processing according to claim 1, characterized in that: The database connection module judges the connection requests sent by the front-end display module: if the connection request is valid, the batch query request is sent to the asynchronous processing module; if the connection request is invalid, the query failure result is returned to the front-end display module.
4. The data analysis and report generation system based on asynchronous processing according to claim 1, characterized in that: The database storage and query module includes a data storage unit and a data query unit. The data storage unit stores telemetry data transmitted from the satellite in real time. The data query unit receives asynchronous query request sets, queries telemetry parameters in the data storage unit, organizes the queried data results into a query result set, and sends it to the data comparison module.
5. The data analysis and report generation system based on asynchronous processing according to claim 1, characterized in that: The valid information extracted from the .xlsx file by the EDS interface module includes: telemetry code, telemetry name, and normal value range; a telemetry Map set is generated based on the telemetry code and telemetry name; and a telemetry theoretical value range Map is generated based on the normal value range.
6. The data analysis and report generation system based on asynchronous processing according to claim 1, characterized in that: The document generation module includes a document generation unit and a result return unit. The document writing unit fills the results generated by the asynchronous processing module into a Word document table and stores it on the server as a document resource. The table includes a normal value result table and an abnormal value result table. The normal value result table includes the telemetry code, telemetry name, telemetry range, and interpretation conclusion. The abnormal value result table includes the telemetry code, abnormal value, and the corresponding time of the abnormal value. The result return unit returns the path of the Word document to the front-end display module, providing users with a download option.
7. The data analysis and report generation system based on asynchronous processing according to claim 1, characterized in that: The database connection module uses the Spring Boot framework, implements a database connection pool through DataSource and HikariCP, and configures it in application.properties.
8. The data analysis and report generation system based on asynchronous processing according to claim 1, characterized in that: The database storage and query module supports storing relevant telemetry data from the ground testing phase, including the TDengine database and the MongoDB database. The telemetry parameter fields in the MongoDB database include: time, telemetry hexadecimal source code, and telemetry value.
9. A data analysis and report generation system based on asynchronous processing according to claim 1, characterized in that: The asynchronous processing module uses a custom thread pool called java.util.concurrent.ThreadPool Executor; the results generated by the data comparison module are sent to the asynchronous processing module in the form of a HashMap collection.
10. A data analysis and report generation system based on asynchronous processing according to claim 1, characterized in that: The document generation module is based on the Apache POI open-source library. It loads templates through the XWPFDocument class, can traverse paragraphs and cells, and returns a .docx file.