Spacecraft system simulation basic data storage interaction system and method based on big data technology

Abstract

The application provides a spacecraft system simulation basic data storage interaction system and method based on big data technology, belongs to the field of data storage interaction, solves the problem of spacecraft system simulation basic data storage interaction difficulty, and is composed of a data storage module, a data management module and a data mining module; the data storage module respectively receives spacecraft system simulation basic data and / or target data output by the data management module through a data structure type classification interface and stores the data; the data management module obtains standardized data of the basic data by calling a multivariate data processing algorithm library in the data mining module, forms a data warehouse with an access interface; the data mining module realizes Hadoop cluster connection between data warehouses through a data access interface, carries out data mining by calling a multivariate data mining algorithm library, forms a visual display mining result and stores the result; and the application realizes storage interaction of spacecraft system simulation basic data.

Description

Technical Field

[0001] This invention relates to the field of data storage interaction, and in particular to a basic data storage interaction system and method for spacecraft system simulation based on big data technology. Background Technology

[0002] The simulation of spacecraft systems requires various basic data such as models, equipment, and environment. Without the storage, management, and analysis of the basic data for spacecraft system simulation, the simulation mission cannot be carried out smoothly. The storage, management, and analysis of basic data can enable data mining of process and result data, discover and extract valuable data, realize data-driven business, and allow business advancement to feed back into data, providing auxiliary decision support for teaching, scientific research, and other business operations.

[0003] Existing technologies utilize data lake architectures based on big data to process data. Data lake architectures are designed for storing information from multiple data sources, including the Internet of Things (IoT). Big data analytics or archiving can be performed by accessing the data lake to process or delivering subsets of data to requesting users.

[0004] However, the basic data for spacecraft system simulation has characteristics such as multiple data types, complex and difficult-to-manage business data, and poor data quality. In addition to storing the basic data, it is also necessary to realize the interaction of various types of basic data, that is, data interaction of functions such as access, storage, management and mining of basic data.

[0005] Since the storage method of data lake architecture based on big data focuses on the storage of diverse and heterogeneous raw data, it is not suitable for the storage and interaction of basic data for spacecraft system simulation. Therefore, there is an urgent need for a data storage and interaction technology based on big data technology to meet the needs of spacecraft system simulation tasks. Summary of the Invention

[0006] To address the aforementioned technical challenges, this invention provides a spacecraft system simulation basic data storage and interaction system and method based on big data technology. It addresses the unified storage management technology for multi-source data, directly forwarding stored data to a data mining and analysis platform and providing various preset mining algorithms for data mining and analysis. This invention combines the characteristics of spacecraft system simulation basic data with comprehensive big data technologies to achieve rapid storage and unified management of multi-source heterogeneous data, including structured data, semi-structured data, unstructured data, and information flow data. It enables users to customize their own data mining and analysis processes, assisting them in quickly analyzing and mining data information to discover its value.

[0007] To solve the above-mentioned technical problems, the technical solution provided by the present invention is as follows:

[0008] A basic data storage and interaction system for spacecraft system simulation based on big data technology consists of a data storage module, a data management module, and a data mining module; among which,

[0009] The data storage module is used to receive spacecraft system simulation basic data corresponding to the data structure type through the data structure type classification interface provided by the Hadoop platform, and classify and store the received spacecraft system simulation basic data according to the data structure type to form a basic data layer; and / or classify and store the target data output by the data management module according to the data structure type to form a target data layer.

[0010] The data management module is used to obtain basic data from the basic data layer in the data storage module. By calling the corresponding algorithms in the multivariate data processing algorithm library in the data mining module, the basic data is cleaned, integrated and transformed to obtain standardized data and form a data warehouse for storing standardized data. The data warehouse provides a data access interface to enable access to or storage of target data in the standardized data.

[0011] The data mining module is used to establish a Hadoop cluster connection with the data warehouse in the data management module through a data access interface, providing a computing environment for the Spark cluster. It performs data mining analysis on target data in the standardized data stored in the data warehouse by calling the corresponding algorithms in the multivariate data mining algorithm library in the data mining module, generating visualized mining analysis results, or storing the generated mining analysis results in the mining library of the data management module.

[0012] Furthermore, the data structure types in the data storage module include: structured data, unstructured data, and / or semi-structured data; among which,

[0013] Structured data is data that is logically expressed and implemented using a two-dimensional table structure, and consists of row data that strictly adheres to data format and length specifications.

[0014] Unstructured data is data with variable field lengths, and each field can be composed of repeating or non-repeating subfields.

[0015] Semi-structured data is data that lies between structured and unstructured data. In semi-structured data, structural patterns are attached to or integrated into the data itself, and the data itself describes its corresponding structural patterns.

[0016] The data structure type classification interfaces include: structured data interface, unstructured data interface, and / or semi-structured data interface;

[0017] The received spacecraft system simulation basic data is classified and stored based on the data structure type, including: using relational databases, HDFS and / or non-relational databases provided by the Hadoop platform that correspond to the data structure type to classify and store various types of data with different data structure types.

[0018] Furthermore, the data storage module receives spacecraft system simulation basic data with data structure types including structured data, unstructured data, or semi-structured data (entity attribute data and / or environmental data). Through the data structure type classification interface provided by the Hadoop platform, the received spacecraft system simulation basic data is automatically stored into the corresponding relational database, HDFS, and / or non-relational database, forming the basic data layer.

[0019] Entity attribute data includes perception-type entity data, execution-type entity data, platform framework-type data, and / or communication-type entity data;

[0020] Environmental data includes maps, nautical charts, topographic environmental data, space environmental data, and / or meteorological and astronomical data.

[0021] Furthermore, the data structure of the spacecraft system simulation basic data received by the data storage module consists of structured data of model description information and unstructured data of model file data. For the structured data of model description information, the model description information is extracted from the database and stored in the corresponding MySQL database of the Hadoop platform. For the model file data, data is collected using the model upload interface and automatically stored in the corresponding HDFS database of the Hadoop platform to provide resource access services for upper-layer simulation operations.

[0022] The model components include simulation model components, assembled entity models, environment models, evaluation index models, and / or evaluation operator models.

[0023] Furthermore, the data storage module receives spacecraft system simulation basic data consisting of semi-structured and / or unstructured file resources. These resources are stored using the HDFS distributed file system and / or the ElasticSearch full-text search engine database storage component corresponding to the Hadoop platform, providing support for users to quickly search and access them.

[0024] Document resources include planning schemes, action plans, evaluation results and / or evaluation reports.

[0025] Furthermore, the data storage module receives business process data with primarily structured data for spacecraft system simulation, which is stored using a relational database corresponding to the Hadoop platform. For business process data with a secondary unstructured data type, it uses storage components such as the Hadoop platform's distributed file system.

[0026] Business process data refers to the events and data generated by the simulation platform during its operation, including state changes of simulated entities, action instructions, control instructions, and / or decision results.

[0027] Furthermore, the data mining module establishes a Hadoop cluster connection with the data warehouse in the data management module through a data access interface, as follows:

[0028] Based on the requirements of spacecraft system simulation missions, this is achieved by using cluster large-scale data processing technology combined with various data mining techniques such as machine learning, deep learning, NLP (natural language processing), statistics, and / or data visualization.

[0029] Furthermore, the multivariate data processing algorithm library in the data mining module includes encapsulated deduplication algorithms, attribute selection algorithms, missing value handling algorithms, normalization algorithms, mapping algorithms, binning algorithms, noise addition algorithms, role setting algorithms, merging algorithms, join algorithms, sorting algorithms, data splitting algorithms, random sampling algorithms, data truncation algorithms, attribute rearrangement algorithms, type conversion algorithms, numerical binarization algorithms, discrete variable numericalization algorithms, variable binarization algorithms, factor analysis algorithms, unified attribute naming algorithms, aggregation algorithms, and / or text vectorization algorithms;

[0030] The multivariate data mining algorithm library in the data mining module includes:

[0031] Machine learning and deep learning model algorithm libraries, including encapsulated Apriori, FP-growth, linear regression, nonlinear regression, decision tree, support vector machine (SVM), Bayesian classification, KNN, logistic regression, neural network, XGB, random forest, K-Means, C-Means and / or DBSCAN open source algorithms.

[0032] A model evaluation algorithm library, including encapsulated MES, MAE, accuracy, precision, recall, confusion matrix, ROC and / or AUC model evaluation methods and metrics;

[0033] The multivariate data processing algorithm library and the multivariate data mining algorithm library each provide custom operator interfaces. User-compiled operator interfaces can be connected to the custom operator interfaces provided by the multivariate data processing algorithm library and the multivariate data mining algorithm library. User-side compilation based on custom operator interfaces via Python scripts is also supported.

[0034] Furthermore, the data mining module generates visualized mining and analysis results, which is achieved through the visualization function of multi-data structure integrated into the data mining module. This includes visualization of two-dimensional data tables, tree diagrams, bar charts, pie charts, correlation analysis diagrams, and / or neural network diagrams, to help users clearly and quickly understand data information and provide a basis for decision-making.

[0035] This invention also provides a basic data storage and interaction method for spacecraft system simulation based on big data technology, including:

[0036] The data storage module receives basic spacecraft system simulation data corresponding to the data structure type through the data structure type classification interface provided by the Hadoop platform, and classifies and stores the received basic spacecraft system simulation data according to the data structure type to form a basic data layer; and / or classifies and stores the target data output by the data management module according to the data structure type to form a target data layer.

[0037] The data management module obtains basic data from the basic data layer in the data storage module. It then cleans, integrates, and transforms the basic data by calling the corresponding algorithms in the multivariate data processing algorithm library in the data mining module to obtain standardized data and form a data warehouse that stores the standardized data. The data warehouse provides a data access interface to enable access to or storage of target data in the standardized data.

[0038] The data mining module establishes a Hadoop cluster connection with the data warehouse in the data management module through a data access interface, providing a computing environment for the Spark cluster. It performs data mining analysis on the target data in the standardized data stored in the data warehouse by calling the corresponding algorithms in the multivariate data mining algorithm library in the data mining module, generating visualized mining analysis results, or storing the generated mining analysis results in the mining library of the data management module.

[0039] Beneficial effects:

[0040] This invention provides a basic data storage and interaction system and method for spacecraft system simulation based on big data technology, which consists of a data storage module, a data management module, and a data mining module; the three modules are interconnected and inseparable.

[0041] The data storage module is used to receive spacecraft system simulation basic data corresponding to the data structure type through the data structure type classification interface provided by the Hadoop platform, and classify and store the received spacecraft system simulation basic data according to the data structure type to form a basic data layer; and / or classify and store the target data output by the data management module according to the data structure type to form a target data layer.

[0042] The data management module is used to acquire basic data from the basic data layer in the data storage module. By calling the corresponding algorithms in the multivariate data processing algorithm library in the data mining module, it cleans, integrates, and transforms the basic data to obtain standardized data and forms a data warehouse for storing the standardized data. The data warehouse provides a data access interface to access or store target data in the standardized data. Standardized data solves problems such as data duplication, missing data, and anomalies. The data warehouse provides a data access interface.

[0043] The data mining module connects to the Hadoop cluster in the data management module via a data access interface, providing a computing environment for the Spark cluster. It performs data mining analysis on target data within the standardized data stored in the data warehouse by calling corresponding algorithms from the multivariate data mining algorithm library in the data mining module. This avoids the complexity of Spark cluster technology, greatly improving the computing speed for big data and reducing the difficulty of big data analysis for users. It generates visualized mining analysis results, including two-dimensional data table visualizations, tree diagrams, bar charts, pie charts, correlation analysis diagrams, neural network diagrams, etc., to help users clearly and quickly understand data information and provide a basis for decision-making. Alternatively, the generated mining analysis results can be stored in the mining library of the data management module. This research aims to create a multi-functional integrated data mining scientific computing platform that combines data analysis, data processing, data statistics, machine learning modeling, deep learning modeling, model evaluation, and data visualization. It provides a high-quality big data mining platform to meet the needs of basic data processing services and data information mining services.

[0044] This invention addresses the challenges of diverse data types in spacecraft system simulation, including complex and difficult-to-manage business data, and poor data quality. It comprehensively utilizes big data technologies to achieve rapid storage and unified management of multi-source heterogeneous data, such as structured, semi-structured, unstructured, and information flow data. The stored and managed data is directly forwarded to data mining analysis, and various preset mining algorithms are provided for data mining analysis. Users can customize their own data mining analysis processes, such as selecting multivariate data processing algorithm libraries, multivariate data mining algorithm libraries, or visualization functions, to assist users in quickly analyzing and mining data information and discovering its value. Attached Figure Description

[0045] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0046] Figure 1 This is a schematic diagram of a spacecraft system simulation basic data storage and interaction system based on big data technology provided by the present invention.

[0047] Figure 2 This is a schematic diagram of data management and access provided by the present invention.

[0048] Figure 3 This is a schematic diagram of data mining provided by the present invention. Detailed Implementation

[0049] First, it should be noted that the embodiments of the present invention are merely preferred implementations and should not be construed as limiting the implementation of the present invention. The scope of protection of the present invention shall still be determined by the contents disclosed in the claims.

[0050] like Figure 1-3 As shown, this invention provides a basic data storage and interaction system for spacecraft system simulation based on big data technology. It consists of a data storage module, a data management module, and a data mining module, which are interconnected and inseparable.

[0051] The data storage module is used to receive spacecraft system simulation basic data corresponding to the data structure type through the data structure type classification interface provided by the Hadoop platform, and classify and store the received spacecraft system simulation basic data according to the data structure type to form a basic data layer; and / or classify and store the target data output by the data management module according to the data structure type to form a target data layer.

[0052] The basic data for spacecraft system simulation includes entity data, environmental data, model data, simulation process data, evaluation data, and basic data from external access.

[0053] The data management module is used to acquire basic data from the basic data layer of the data storage module. By calling the corresponding algorithms in the multivariate data processing algorithm library of the data mining module, it cleans, integrates, and transforms the basic data to obtain standardized data. It performs necessary normalization processing on the data to solve problems such as data duplication, missing data, and anomalies. It forms a data warehouse to store the standardized data. The data warehouse provides data access interfaces to access or store target data in the standardized data. For example, it can use the RESTful interface to uniformly manage various types of data in the data warehouse, such as simulation data, evaluation data, entity data, and environmental data.

[0054] The data mining module is used to establish a Hadoop cluster connection with the data warehouse in the data management module through a data access interface, providing a computing environment for the Spark cluster. It performs data mining analysis on target data in the standardized data stored in the data warehouse by calling the corresponding algorithms in the multivariate data mining algorithm library in the data mining module, generating visualized mining analysis results, or storing the generated mining analysis results in the mining library of the data management module.

[0055] The data structure types in the data storage module include: structured data, unstructured data, and / or semi-structured data; among them,

[0056] Structured data is data logically expressed and implemented using a two-dimensional table structure, strictly adhering to data format and length specifications for row-based data. Simply put, structured data is a database. This is easier to understand when considered in typical scenarios, such as enterprise ERP and financial systems; medical HIS databases; educational smart cards; government administrative approvals; and other core databases. These applications generally include storage solutions for high-speed storage applications, data backup requirements, data sharing needs, and data disaster recovery requirements.

[0057] Unstructured data is data with variable field lengths, and each field record can be composed of repeating or non-repeating subfields; it can not only process structured data (such as numbers, symbols and other information) but is also more suitable for processing unstructured data (full text, images, sound, video, multimedia and other information).

[0058] Semi-structured data is data that lies between structured and unstructured data. In semi-structured data, the structural pattern is attached to or integrated into the data itself, and the data itself describes its corresponding structural pattern; for example, OEM (Object Exchange Model) is a typical semi-structured data model.

[0059] The data structure type classification interfaces include: structured data interface, unstructured data interface, and / or semi-structured data interface;

[0060] The received spacecraft system simulation basic data is classified and stored based on the data structure type, including: using relational databases, HDFS and / or non-relational databases provided by the Hadoop platform that correspond to the data structure type to classify and store various types of data with different data structure types.

[0061] For example: for large amounts of unstructured data, HDFS-based distributed storage is used; process data and intermediate data generated by the system's data analysis, such as various events and data generated during the operation of the simulation platform, including state changes of simulation entities, action instructions, control instructions, and adjudication results, are stored using the open-source relational database MySQL; for massive amounts of data that need to be retrieved in real time, ElasticSearch is used for storage; and for data that needs to be statistically analyzed on a large scale, Hive or similar storage methods are chosen.

[0062] The data storage module receives spacecraft system simulation basic data with data structure types including structured data, unstructured data, or semi-structured data (entity attribute data and / or environmental data). Through the data structure type classification interface provided by the Hadoop platform, the received spacecraft system simulation basic data is automatically stored into the corresponding relational database, HDFS, and / or non-relational database, forming the basic data layer.

[0063] Entity attribute data includes perception-type entity data, execution-type entity data, platform framework-type data, and / or communication-type entity data. Within this data, entity performance parameters are primarily structured, while entity images and user manuals are mainly unstructured. The data storage module provides collection interfaces for various basic data types, such as perception-type data storage interfaces and entity image interfaces. Users can collect data through these interfaces or manually upload data on the corresponding data pages. Data collected through these interfaces is automatically stored in MySQL, Hive, or HDFS based on its data type.

[0064] Environmental data includes maps, nautical charts, topographic environmental data, space environmental data, and / or meteorological and astronomical data. Data types include structured, semi-structured, and unstructured types. The majority of environmental data is unstructured image-based data. The data storage module provides acquisition interfaces for maps, nautical charts, topographic environmental data, space environmental data, and meteorological and astronomical data. Users can upload environmental data of various types according to their respective data types, and store them separately, thereby providing high-performance resource access services.

[0065] The data storage module receives basic spacecraft system simulation data, which consists of structured data (model description information) and unstructured data (model file data). The structured data (model description information) is extracted from the database and stored in the corresponding MySQL database on the Hadoop platform. The model file data is collected via a model upload interface and automatically stored in the corresponding HDFS database on the Hadoop platform, providing resource access support for upper-layer simulation services.

[0066] The model components include simulation model components, assembled entity models, environment models, evaluation index models, and / or evaluation operator models.

[0067] The data storage module receives spacecraft system simulation basic data consisting of semi-structured and / or unstructured file resources. These resources are stored using the HDFS distributed file system and / or the ElasticSearch full-text search engine database storage component corresponding to the Hadoop platform, providing support for users to quickly search and access the data.

[0068] Document resources include planning schemes, action plans, evaluation results and / or evaluation reports.

[0069] The data storage module receives spacecraft system simulation basic data, primarily structured business process data, which is stored using a relational database on the Hadoop platform. For business process data with a secondary unstructured data structure, storage components such as the Hadoop platform's distributed file system are used.

[0070] Business process data refers to the events and data generated by the simulation platform during its operation, including state changes of simulated entities, action instructions, control instructions, and / or decision results.

[0071] The data mining module establishes a Hadoop cluster connection with the data warehouse in the data management module through a data access interface.

[0072] Based on the requirements of spacecraft system simulation missions, this project utilizes large-scale cluster data processing technology combined with various data mining techniques, including machine learning, deep learning, natural language processing (NLP), statistics, and / or data visualization. The data mining module addresses the massive amounts of data accumulated across various fields and the need for big data analysis and processing. It provides a distributed storage and Spark cluster computing environment by connecting to a Hadoop cluster. Furthermore, it connects with Spark's specific data operations through built-in operators (such as multivariate data processing algorithm libraries and multivariate data mining algorithm libraries), avoiding the complexity of Spark cluster technology, greatly improving the computing speed for big data, and reducing the difficulty of big data analysis and processing for users.

[0073] Utilizing large-scale cluster data processing technologies, including big data storage and big data computing technologies, and combining various technologies such as machine learning, deep learning, NLP (natural language processing), statistics, and data visualization in data mining, we encapsulate various algorithms to research and develop a multi-functional integrated data mining scientific computing platform that combines data analysis, data processing, data statistics, machine learning modeling, deep learning modeling, model evaluation, and data visualization. This platform provides a high-quality big data mining platform to meet the needs of basic data processing services and data information mining services.

[0074] The data mining module is connected to the data warehouse of the data management module at its core. It can directly select various types of data in the data warehouse for mining and analysis. All the provided operators have been modularized and can be directly connected and used on the page by dragging and dropping. The results of the analysis and mining can be directly saved to the mining library of the data management module or directly visualized.

[0075] The multivariate data processing algorithm library in the data mining module includes encapsulated deduplication algorithms, attribute selection algorithms, missing value handling algorithms, normalization algorithms, mapping algorithms, binning algorithms, noise addition algorithms, role setting algorithms, merging algorithms, join algorithms, sorting algorithms, data splitting algorithms, random sampling algorithms, data truncation algorithms, attribute rearrangement algorithms, type conversion algorithms, numerical binarization algorithms, discrete variable numericalization algorithms, variable binarization algorithms, factor analysis algorithms, unified attribute naming algorithms, aggregation algorithms, and / or text vectorization algorithms;

[0076] The multivariate data mining algorithm library in the data mining module includes:

[0077] Machine learning and deep learning model algorithm libraries, including encapsulated Apriori, FP-growth, linear regression, nonlinear regression, decision tree, support vector machine (SVM), Bayesian classification, KNN, logistic regression, neural network, XGB, random forest, K-Means, C-Means and / or DBSCAN open source algorithms.

[0078] This library of model evaluation algorithms includes encapsulated MES, MAE, accuracy, precision, recall, confusion matrix, ROC, and / or AUC model evaluation methods and metrics. It aims to meet users' basic algorithmic applications and model evaluation needs in big data mining and modeling.

[0079] For example, the acquired stored data can be mined using random forests, regression algorithms, and decision trees to generate visualized mining analysis results.

[0080] The multivariate data processing algorithm library and the multivariate data mining algorithm library each provide custom operator interfaces. User-compiled operator interfaces can be connected to the custom operator interfaces provided by the multivariate data processing algorithm library and the multivariate data mining algorithm library. User-side compilation based on custom operator interfaces via Python scripts is also supported.

[0081] To address specific user algorithm requirements, the data mining module provides a custom operator interface, allowing users to compile custom operators using Python scripts. These user-compiled operators can be integrated with the module's built-in operators and seamlessly connected to other platform functions such as automated anomaly detection and data visualization, thus expanding the platform's data mining algorithm capabilities.

[0082] The data mining module generates visualized mining and analysis results, which is achieved through the visualization function of multi-data structure integrated into the data mining module. This includes visualization of two-dimensional data tables, tree diagrams, bar charts, pie charts, correlation analysis diagrams and / or neural network diagrams, which are used to help users clearly and quickly understand data information and provide a basis for decision-making.

[0083] This invention also provides a basic data storage and interaction method for spacecraft system simulation based on big data technology, including:

[0084] The data storage module receives basic spacecraft system simulation data corresponding to the data structure type through the data structure type classification interface provided by the Hadoop platform, and classifies and stores the received basic spacecraft system simulation data according to the data structure type to form a basic data layer; and / or classifies and stores the target data output by the data management module according to the data structure type to form a target data layer.

[0085] The data management module obtains basic data from the basic data layer in the data storage module. It then cleans, integrates, and transforms the basic data by calling the corresponding algorithms in the multivariate data processing algorithm library in the data mining module to obtain standardized data and form a data warehouse that stores the standardized data. The data warehouse provides a data access interface to enable access to or storage of target data in the standardized data.

[0086] The data mining module establishes a Hadoop cluster connection with the data warehouse in the data management module through a data access interface, providing a computing environment for the Spark cluster. It performs data mining analysis on the target data in the standardized data stored in the data warehouse by calling the corresponding algorithms in the multivariate data mining algorithm library in the data mining module, generating visualized mining analysis results, or storing the generated mining analysis results in the mining library of the data management module.

[0087] Beneficial effects:

[0088] This invention provides a basic data storage and interaction system and method for spacecraft system simulation based on big data technology, which consists of a data storage module, a data management module, and a data mining module; the three modules are interconnected and inseparable.

[0089] The data storage module is used to receive spacecraft system simulation basic data corresponding to the data structure type through the data structure type classification interface provided by the Hadoop platform, and classify and store the received spacecraft system simulation basic data according to the data structure type to form a basic data layer; and / or classify and store the target data output by the data management module according to the data structure type to form a target data layer.

[0090] The data management module is used to acquire basic data from the basic data layer in the data storage module. By calling the corresponding algorithms in the multivariate data processing algorithm library in the data mining module, it cleans, integrates, and transforms the basic data to obtain standardized data and forms a data warehouse for storing the standardized data. The data warehouse provides a data access interface to access or store target data in the standardized data. Standardized data solves problems such as data duplication, missing data, and anomalies. The data warehouse provides a data access interface.

[0091] The data mining module connects to the Hadoop cluster in the data management module via a data access interface, providing a computing environment for the Spark cluster. It performs data mining analysis on target data within the standardized data stored in the data warehouse by calling corresponding algorithms from the multivariate data mining algorithm library in the data mining module. This avoids the complexity of Spark cluster technology, greatly improving the computing speed for big data and reducing the difficulty of big data analysis for users. It generates visualized mining analysis results, including two-dimensional data table visualizations, tree diagrams, bar charts, pie charts, correlation analysis diagrams, neural network diagrams, etc., to help users clearly and quickly understand data information and provide a basis for decision-making. Alternatively, the generated mining analysis results can be stored in the mining library of the data management module. This research aims to create a multi-functional integrated data mining scientific computing platform that combines data analysis, data processing, data statistics, machine learning modeling, deep learning modeling, model evaluation, and data visualization. It provides a high-quality big data mining platform to meet the needs of basic data processing services and data information mining services.

[0092] This invention addresses the challenges of diverse data types in spacecraft system simulation, including complex and difficult-to-manage business data, and poor data quality. It comprehensively utilizes big data technologies to achieve rapid storage and unified management of multi-source heterogeneous data, such as structured, semi-structured, unstructured, and information flow data. The stored and managed data is directly forwarded to data mining analysis, and various preset mining algorithms are provided for data mining analysis. Users can customize their own data mining analysis processes, such as selecting multivariate data processing algorithm libraries, multivariate data mining algorithm libraries, or visualization functions, to assist users in quickly analyzing and mining data information and discovering its value.

[0093] Finally, it should be pointed out that any use or implementation of the technical solution of this invention by any entity or individual constitutes an infringement of this invention, and no entity or individual may implement this patent independently without the permission of the applicant. Any implementation by any entity or individual inspired by or with simple modifications to this invention should also be considered within the scope of protection of this patent.

Claims

1. A spacecraft system simulation basic data storage and interaction system based on big data technology, characterized in that, It consists of a data storage module, a data management module, and a data mining module; among which... The data storage module is used to receive spacecraft system simulation basic data corresponding to the data structure type through the data structure type classification interface provided by the Hadoop platform, and to classify and store the received spacecraft system simulation basic data according to the data structure type to form a basic data layer. Then, based on the data structure type, the target data output by the data management module is classified and stored to form a target data layer; The data management module is used to obtain basic data from the basic data layer in the data storage module. By calling the corresponding algorithms in the multivariate data processing algorithm library in the data mining module, the basic data is cleaned, integrated and transformed to obtain standardized data, and a data warehouse is formed to store the standardized data. The data warehouse provides a data access interface. The data mining module is used to establish a Hadoop cluster connection with the data warehouse in the data management module through a data access interface, providing a computing environment for the Spark cluster. It performs data mining analysis on target data in the standardized data stored in the data warehouse by calling the corresponding algorithms in the multivariate data mining algorithm library in the data mining module, generating visualized mining analysis results, or storing the generated mining analysis results in the mining library of the data management module.

2. The system as described in claim 1, characterized in that, The data structure types in the data storage module include: structured data, unstructured data, and / or semi-structured data; among them, Structured data is data that is logically expressed and implemented using a two-dimensional table structure, and consists of rows of data that strictly adhere to data format and length specifications. Unstructured data is data with variable field lengths, and each field can be composed of repeating or non-repeating subfields. Semi-structured data is data that lies between structured and unstructured data. In semi-structured data, structural patterns are attached to or integrated into the data itself, and the data itself describes its corresponding structural patterns. The data structure type classification interfaces include: structured data interface, unstructured data interface, and / or semi-structured data interface; The received spacecraft system simulation basic data is classified and stored based on the data structure type, including: using relational databases, HDFS and / or non-relational databases provided by the Hadoop platform that correspond to the data structure type to classify and store various types of data with different data structure types.

3. The system as described in claim 1, characterized in that, The data storage module receives spacecraft system simulation basic data with data structure types including structured data, unstructured data, or semi-structured data (entity attribute data and / or environmental data). Through the data structure type classification interface provided by the Hadoop platform, the received spacecraft system simulation basic data is automatically stored into the corresponding relational database, HDFS, and / or non-relational database, forming the basic data layer. Entity attribute data includes perception-type entity data, execution-type entity data, platform framework-type data, and / or communication-type entity data; Environmental data includes maps, nautical charts, topographic environmental data, space environmental data, and / or meteorological and astronomical data.

4. The system as described in claim 1, characterized in that, The data storage module receives basic spacecraft system simulation data, which consists of structured data (model description information) and unstructured data (model file data). The structured data (model description information) is extracted from the database and stored in the corresponding MySQL database on the Hadoop platform. The model file data is collected via a model upload interface and automatically stored in the corresponding HDFS database on the Hadoop platform, providing resource access support for upper-layer simulation services. The model components include simulation model components, assembled entity models, environment models, evaluation index models, and / or evaluation operator models.

5. The system as described in claim 1, characterized in that, The data storage module receives spacecraft system simulation basic data consisting of semi-structured and / or unstructured file resources. These resources are stored using the HDFS distributed file system and / or the ElasticSearch full-text search engine database storage component corresponding to the Hadoop platform, providing support for users to quickly search and access the data. Document resources include planning schemes, action plans, evaluation results and / or evaluation reports.

6. The system as described in claim 1, characterized in that, The data storage module receives spacecraft system simulation basic data, primarily structured business process data, which is stored using a relational database on the Hadoop platform. For business process data with a secondary unstructured data structure, storage components such as the Hadoop platform's distributed file system are used. Business process data refers to the events and data generated by the simulation platform during its operation, including state changes of simulated entities, action instructions, control instructions, and / or decision results.

7. The system as described in claim 1, characterized in that, The data mining module establishes a Hadoop cluster connection with the data warehouse in the data management module through a data access interface. Based on the requirements of spacecraft system simulation missions, this is achieved by using cluster large-scale data processing technology combined with various data mining techniques such as machine learning, deep learning, NLP (natural language processing), statistics, and / or data visualization.

8. The system as described in claim 1, characterized in that, The multivariate data processing algorithm library in the data mining module includes encapsulated deduplication algorithms, attribute selection algorithms, missing value handling algorithms, normalization algorithms, mapping algorithms, binning algorithms, noise addition algorithms, role setting algorithms, merging algorithms, join algorithms, sorting algorithms, data splitting algorithms, random sampling algorithms, data truncation algorithms, attribute rearrangement algorithms, type conversion algorithms, numerical binarization algorithms, discrete variable numericalization algorithms, variable binarization algorithms, factor analysis algorithms, unified attribute naming algorithms, aggregation algorithms, and / or text vectorization algorithms; The multivariate data mining algorithm library in the data mining module includes: Machine learning and deep learning model algorithm libraries, including encapsulated Apriori, FP-growth, linear regression, nonlinear regression, decision tree, support vector machine (SVM), Bayesian classification, KNN, logistic regression, neural network, XGB, random forest, K-Means, C-Means and / or DBSCAN open source algorithms. A model evaluation algorithm library, including encapsulated MES, MAE, accuracy, precision, recall, confusion matrix, ROC and / or AUC model evaluation methods and metrics; The multivariate data processing algorithm library and the multivariate data mining algorithm library each provide custom operator interfaces. User-compiled operator interfaces can be connected to the custom operator interfaces provided by the multivariate data processing algorithm library and the multivariate data mining algorithm library. User-side compilation based on custom operator interfaces via Python scripts is also supported.

9. The system as described in claim 1, characterized in that, The data mining module generates visualized mining and analysis results, which is achieved through the visualization function of multi-data structure integrated into the data mining module. This includes visualization of two-dimensional data tables, tree diagrams, bar charts, pie charts, correlation analysis diagrams and / or neural network diagrams, which are used to help users clearly and quickly understand data information and provide a basis for decision-making.

10. A basic data storage and interaction method for spacecraft system simulation based on big data technology, characterized in that, include: The data storage module receives spacecraft system simulation basic data corresponding to the data structure type through the data structure type classification interface provided by the Hadoop platform, and classifies and stores the received spacecraft system simulation basic data according to the data structure type to form a basic data layer; And / or classify and store the target data output by the data management module based on the data structure type to form a target data layer; The data management module obtains basic data from the basic data layer in the data storage module. It then cleans, integrates, and transforms the basic data by calling the corresponding algorithms in the multivariate data processing algorithm library in the data mining module to obtain standardized data and form a data warehouse that stores the standardized data. The data warehouse provides a data access interface to enable access to or storage of target data in the standardized data. The data mining module establishes a Hadoop cluster connection with the data warehouse in the data management module through a data access interface, providing a computing environment for the Spark cluster. It performs data mining analysis on the target data in the standardized data stored in the data warehouse by calling the corresponding algorithms in the multivariate data mining algorithm library in the data mining module, generating visualized mining analysis results, or storing the generated mining analysis results in the mining library of the data management module.

Images