A method and system for mining and analyzing big data in aerospace engineering

Abstract

The present invention provides a method and system for mining and analyzing big data of aerospace engineering, which can provide full-process data mining and analysis services in a convenient and efficient construction and application manner, and provide support for mining and analysis of big data of aerospace engineering scientific experiments. In this method, in response to an input mining and analysis service request, aerospace engineering data is preprocessed, and the preprocessed aerospace engineering data is obtained and displayed, and a first page is displayed. The first page includes at least one first algorithm model, and the first algorithm model is used to mine and analyze the data. Based on the input selection operation of at least one target first algorithm model in at least one first algorithm model, a second algorithm model corresponding to each target first algorithm model is obtained. Based on the input orchestration operation for each second algorithm model, an orchestration process is determined. Based on the determined orchestration process, a mining and analysis operation on aerospace engineering data is performed to obtain corresponding mining and analysis results.

Description

Technical Field

[0001] The present invention relates to the field of aerospace engineering scientific experiment data processing and mining technology, and in particular to an aerospace engineering big data mining and analysis method and system. Background Art

[0002] In the field of manned space science and technology, the scientific experiment mirror platform (referred to as mirror platform) can be used to support ground research work and space-ground comparison experiments of space science experiment projects, and provide ground research support conditions for experimental research projects.

[0003] The mirror platform will generate a large amount of scientific experimental data during the operation of the space station mission. In order to meet the analysis needs of various scientific experimental data such as life ecology, biotechnology, ultra-cold atomic physics, high-temperature materials science, combustion science, and high microgravity science in the scientific experiment cabinets of the ground mirror platform, it is necessary to build a multi-source scientific experimental data mining and analysis system, so as to provide a basic platform support for the mining and analysis of multi-source scientific experimental data of the mirror platform. Summary of the invention

[0004] In order to solve the above problems, the present invention provides a method and system for mining and analyzing aerospace engineering big data. The technical solution of the present invention to solve the above technical problems is as follows:

[0005] On the one hand, the present invention provides a system for mining and analyzing aerospace engineering big data, including a big data computing basic engine unit, an algorithm model management unit, and a data mining and analysis task management unit. The big data computing basic engine unit is used to respond to an input mining and analysis service request, pre-process the aerospace engineering data, obtain and display the pre-processed aerospace engineering data, and display a first page. The mining and analysis service request is used to request to perform a mining and analysis operation on the aerospace engineering data, and the first page includes at least one first algorithm model; the first algorithm model is used to perform mining and analysis on the data. The algorithm model management unit is used to obtain a second algorithm model corresponding to each target first algorithm model based on an input selection operation of at least one target first algorithm model in at least one first algorithm model. The data mining and analysis task management unit is used to determine an orchestration process based on an input orchestration operation for each second algorithm model, and perform a mining and analysis operation on the aerospace engineering data based on the orchestration process to obtain a mining and analysis result of the aerospace engineering data. The orchestration operation includes determining the execution order of each second algorithm model.

[0006] Furthermore, the big data computing basic engine unit includes an offline data batch distributed processing and analysis module, which is used to analyze the aerospace engineering data by using a distributed batch processing method to obtain the aerospace engineering data after preprocessing.

[0007] Furthermore, the big data computing basic engine unit also includes a machine learning computing engine module and a deep learning computing engine module; the machine learning computing engine module is used to train the first initial model based on the training data to obtain the second initial model. The training data is associated with aerospace engineering data; the first initial model is a model created by the user and uploaded to the system for mining and analyzing data. The deep learning computing engine module is used to train the first initial model based on the training data to obtain a third initial model. The third initial model is different from the second initial model, and the first algorithm model is determined based on the second initial model or the third initial model.

[0008] Furthermore, the algorithm model management unit includes an algorithm management module and a custom algorithm model module. The algorithm management module is used to classify and display at least one first algorithm model in response to an input classification operation on at least one first algorithm model. The custom algorithm model module is used to display a second page for editing at least one target first algorithm model based on an input selection operation on at least one target first algorithm model in at least one first algorithm model. The custom algorithm model module is also used to obtain a second algorithm model corresponding to each target first algorithm model based on an input optimization operation on at least one target first algorithm model. Wherein, if the target first algorithm model is subjected to an optimization operation, the target first algorithm model after the optimization operation is performed is determined as the second algorithm model corresponding to the target first algorithm model. If the target first algorithm model is not subjected to an optimization operation, the target first algorithm model is determined as the second algorithm model corresponding to the target first algorithm model.

[0009] Furthermore, the algorithm model management unit also includes an algorithm model library interface encapsulation module. The algorithm model library interface encapsulation module is used to encapsulate the calling interface corresponding to each second algorithm model. Among them, the calling interface corresponding to the second algorithm model is used to support a third-party device to call the corresponding second algorithm model. The algorithm model library interface encapsulation module is also used to encapsulate the calling interface corresponding to the orchestration process. The calling interface corresponding to the orchestration process is used to support a third-party device to call the orchestration process.

[0010] Furthermore, the data mining and analysis task management unit includes a data mining and analysis process arrangement module. The data mining and analysis process arrangement module is used to display a third page, and the third page is used to arrange the execution order of each second algorithm model. The data mining and analysis process arrangement module is also used to receive the arrangement operation for each second algorithm model input on the third page, and determine the arrangement process.

[0011] Furthermore, the data mining and analysis task management unit includes an algorithm service management module. The algorithm service management module is used to call the orchestration process, and perform mining and analysis operations on aerospace engineering data based on the orchestration process to obtain mining and analysis results on aerospace engineering data.

[0012] Furthermore, the aerospace engineering big data mining and analysis system provided by the present invention also includes a resource management and task scheduling monitoring unit. The resource management and task scheduling monitoring unit includes a resource management module, a multi-task scheduling module and a task monitoring module. Among them, the resource management module is used to allocate corresponding resources to the at least one orchestration process based on preset rules when executing at least one orchestration process. The multi-task scheduling module is used to determine the execution priority of each of the at least two orchestration processes when executing at least two orchestration processes. The task monitoring module is used to monitor the execution information of the target orchestration process, and the target orchestration process includes the unexecuted orchestration process and the executed orchestration process.

[0013] On the other hand, the present invention provides a method for mining and analyzing aerospace engineering big data, in which, in response to an input mining and analysis service request, aerospace engineering data is preprocessed, the preprocessed aerospace engineering data is obtained and displayed, and a first page is displayed. The mining and analysis service request is used to request a mining and analysis operation to be performed on aerospace engineering data. The first page includes at least one first algorithm model, and the first algorithm model is used to perform mining and analysis on the data. Based on the input selection operation of at least one target first algorithm model in the at least one first algorithm model, a second algorithm model corresponding to each target first algorithm model is obtained. Based on the input orchestration operation for each second algorithm model, an orchestration process is determined. The orchestration operation includes determining the execution order of each second algorithm model. The mining and analysis operation of the aerospace engineering data is performed based on the orchestration process to obtain the mining and analysis results of the aerospace engineering data.

[0014] Further, when executing at least one orchestration process, corresponding resources are allocated to the at least one orchestration process based on a preset rule. When executing at least two orchestration processes, the execution priority of each of the at least two orchestration processes is determined. The execution information of the target orchestration process is monitored, and the target orchestration process includes an unexecuted orchestration process and / or an executed orchestration process.

[0015] Compared with the prior art, the present invention has the following beneficial effects:

[0016] 1. Support users to create and edit algorithm models online. Users can determine the orchestration process for each algorithm model by simply dragging and dropping, providing users with a real-time, interactive programming environment, allowing users to directly interact with the system, test and adjust the code, simplify user operations, and improve orchestration efficiency.

[0017] 2. After the orchestration process is determined, the system automatically triggers data processing. After the background service receives the process trigger request, it automatically generates the process execution script and submits it to the data processing engine. At the same time, the system's built-in high-performance distributed algorithm can cope with large-scale and highly complex data application scenarios by integrating computing and optimizing algorithm performance.

[0018] 3. Support monitoring when performing mining and analysis operations on aerospace engineering data, and visualize the monitoring results to show users the entire process of mining and analysis operations on aerospace engineering data in a more intuitive way.

[0019] 4. Supports the management, visualization and storage of data mining and analysis results, improving the intelligence level and security of data mining and analysis management.

[0020] 5. Support efficient mining and analysis of massive scientific experimental data on the mirror platform, support functions including distributed computing analysis system, algorithm model management, multi-source heterogeneous data access management, task management and resource management, comprehensively integrate and optimize big data mining and analysis related technologies, and provide full-process data mining and analysis services such as creation, operation and management of mining and analysis tasks in a convenient and efficient way of construction and application, thereby providing technical method support for the mining and analysis of multi-source scientific experimental data on the mirror platform. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1A A schematic diagram of the structure of an aerospace engineering big data mining and analysis system provided by an embodiment of the present invention;

[0022] Figure 1B A schematic diagram of a process for determining a first algorithm model provided by an embodiment of the present invention;

[0023] Figure 2 A schematic diagram of a process of creating a new online programming project provided by an embodiment of the present invention;

[0024] Figure 3 A schematic diagram of a process for editing basic information of an algorithm model provided by an embodiment of the present invention;

[0025] Figure 4A A schematic diagram of a flow chart of a custom algorithm model provided by an embodiment of the present invention;

[0026] Figure 4BA schematic diagram of the relationship between a first algorithm model, a target first algorithm model and a second algorithm model provided in an embodiment of the present invention;

[0027] Figure 5A A schematic diagram of a scenario of an orchestration process provided by an embodiment of the present invention;

[0028] Figure 5B A schematic diagram of a layout page provided by an embodiment of the present invention;

[0029] Figure 5C A schematic diagram of a process for arranging a new process for a data mining analysis process provided by an embodiment of the present invention;

[0030] Figure 6 A logic flow chart of distributed resource management provided by an embodiment of the present invention;

[0031] Fig. 7A A logic flow chart of multi-type task scheduling provided by an embodiment of the present invention;

[0032] Figure 7B A schematic diagram of a monitoring interface provided by an embodiment of the present invention;

[0033] Figure 8 A schematic diagram of a flow chart of a method for mining and analyzing aerospace engineering big data provided by an embodiment of the present invention;

[0034] Fig. 9 A flowchart of the steps of data mining and analysis task management provided by an embodiment of the present invention;

[0035] Fig.10 A schematic diagram of an algorithm creation process provided by an embodiment of the present invention;

[0036] Fig.11 A schematic diagram of an algorithm editing process provided by an embodiment of the present invention;

[0037] Fig.12 A schematic diagram of an algorithm management process provided by an embodiment of the present invention;

[0038] Fig.13 A schematic diagram of an algorithm publishing process provided by an embodiment of the present invention;

[0039] Fig.14 A schematic diagram of a process arrangement process provided by an embodiment of the present invention;

[0040] Fig.15 A schematic diagram of a process submission process provided by an embodiment of the present invention;

[0041] Fig.16 A schematic diagram of an algorithm calculation flow provided by an embodiment of the present invention;

[0042] Fig.17 A schematic diagram of a mining and analysis process provided by an embodiment of the present invention;

[0043] Fig.18 A schematic diagram of a result export process provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0044] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application. Among them, in the description of the present application, unless otherwise specified, " / " indicates that the objects associated before and after are in an "or" relationship, for example, A / B can represent A or B; "and / or" in the present application is only a description of the association relationship of the associated objects, indicating that there can be three relationships, for example, A and / or B, which can represent: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. And, in the description of the present application, unless otherwise specified, "multiple" refers to two or more than two. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, c can be single or multiple. In addition, in order to facilitate the clear description of the technical solutions of the embodiments of the present application, in the embodiments of the present application, the words "first", "second" and the like are used to distinguish the same items or similar items with substantially the same functions and effects. Those skilled in the art will understand that the words "first", "second" and the like do not limit the quantity and execution order, and the words "first", "second" and the like do not necessarily limit the differences. At the same time, in the embodiments of the present application, the words "exemplary" or "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or design solutions. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a concrete manner for ease of understanding.

[0045] The present invention provides a method and system for mining and analyzing big data of aerospace engineering. Among them, the system provided by the present invention is a big data computing and analysis support system for massive mirror platform scientific experiment data, which is established on the basis of distributed storage and distributed computing. The system provided by the present invention can support efficient mining and analysis of massive mirror platform scientific experiment data, support functions such as distributed computing analysis system, algorithm model management, multi-source heterogeneous data access management, task management and resource management, comprehensively integrate and optimize big data mining and analysis related technologies, and provide full-process data mining and analysis services such as creation, operation and management of analysis tasks in a convenient and efficient construction and application manner, so as to provide support for mirror platform data mining and analysis.

[0046] See also Figure 1A , a space engineering big data mining and analysis system provided by the present invention, including a big data computing basic engine unit, an algorithm model management unit, a data mining and analysis task management unit, and a resource management and task scheduling monitoring unit. Among them, the output end of the big data computing basic engine unit is communicatively connected to the input end of the algorithm model management unit. The output end of the algorithm model management unit is communicatively connected to the input end of the data mining and analysis task management unit. The output end of the data mining and analysis task management unit is communicatively connected to the input end of the resource management and task scheduling monitoring unit.

[0047] In some embodiments, the big data computing basic engine unit is used to pre-process the aerospace engineering data (for example, perform a cleaning operation on the aerospace engineering data) in response to the input mining and analysis service request, obtain and display the pre-processed aerospace engineering data, and display the first page. The mining and analysis service request is used to request the execution of a mining and analysis operation on the aerospace engineering data. The first page includes at least one first algorithm model, and the first algorithm model is used to perform mining and analysis on the data. The algorithm model management unit is used to obtain the second algorithm model corresponding to each target first algorithm model based on the input selection operation of at least one target first algorithm model in at least one first algorithm model. The data mining and analysis task management unit is used to determine the orchestration process based on the input orchestration operation for each of the second algorithm models, and to perform the mining and analysis operation on the aerospace engineering data based on the orchestration process to obtain the mining and analysis results of the aerospace engineering data. The orchestration operation includes determining the execution order of each second algorithm model.

[0048] Combine the following Figure 1A , and continue to introduce the big data computing basic engine unit, algorithm model management unit, data mining and analysis task management unit, and resource management and task scheduling monitoring unit in detail.

[0049] In some embodiments, see Figure 1AThe big data computing basic engine unit includes an offline data batch distributed processing and analysis module. The offline data batch distributed processing and analysis module can be used to pre-process aerospace engineering data (for example, perform a cleaning operation on aerospace engineering data) to obtain pre-processed aerospace engineering data.

[0050] In some embodiments, the offline data batch distributed processing and analysis module builds a Hadoop big data ecosystem. The offline data batch distributed processing and analysis module can load data offline from a database or file, and use batch distributed preprocessing for offline data, which can improve the preprocessing efficiency of offline data.

[0051] In some embodiments, when the big data computing basic engine unit receives an input mining and analysis service request, the offline data batch distributed processing and analysis module can use the Spark computing engine and the Hive data warehouse tool to offline aerospace engineering data, and can use distributed batch processing to pre-process the aerospace engineering data to obtain pre-processed aerospace engineering data.

[0052] In some embodiments, see Figure 1A ,The big data computing basic engine unit also includes a machine learning computing engine module and a deep learning computing engine module.

[0053] Among them, the machine learning computing engine module has built a machine learning framework to meet the machine learning environment, including Scikit-learn and Spark ML lib, which supports distributed machine learning, elastic scalability, high learning efficiency, and supports the entire process of machine learning, including model building, model verification, evaluation, and deployment. In order to meet the deep learning environment, the deep learning computing engine module has built a deep learning framework service, supporting distributed training, and supporting training modes such as CPU, GPU, multi-GPU, and CPU+GPU hybrid. The underlying layer uses deep learning frameworks such as PyTorch and Keras, supporting the entire process of deep learning, including deep learning model building, model verification and evaluation, and model deployment, and supports one-click (script) version upgrades and dependency package installation. The machine learning computing engine module is used to execute various machine algorithms to achieve model training and prediction. The deep learning computing engine module is used to process complex deep learning models, which contain a large number of parameters and complex network structures.

[0054] In some embodiments, see Figure 1B, the machine learning computing engine module can train a first initial model based on the training data to obtain a second initial model. The training data is associated with aerospace engineering data. The first initial model can be a model for mining and analyzing data that is pre-created by a user and uploaded to the system provided by the present invention. The deep learning computing engine module can train the first initial model based on the training data to obtain a third initial model. The third initial model is different from the second initial model. The first algorithm model is determined based on the second initial model or the third initial model.

[0055] The offline data batch distributed processing and analysis module, the machine learning computing engine module, and the deep learning computing engine module jointly build a computing environment that supports users to process various big data analysis and computing tasks.

[0056] In some embodiments, see Figure 1A ,The algorithm model management unit includes an algorithm management module and a custom algorithm model module.

[0057] The algorithm management module provides multiple functions, including algorithm model management, algorithm model deployment, algorithm model monitoring, and algorithm model iteration. The algorithm management module is responsible for effectively organizing and managing various algorithm models, ensuring their accuracy and availability, and working with the custom algorithm model module to build a comprehensive algorithm library. The custom algorithm model module focuses on the creation and management of user-customized models, and provides personalized model development and adjustment functions.

[0058] In some embodiments, the custom algorithm model supports online interactive programming functions. Operator algorithms can be debugged online, and online interactive SQL, Python and other online programming modules are integrated. The online interactive programming function provides query, modification, deletion, and addition operations for online algorithms, and interactive data processing programs can be written online on the interactive programming page. Figure 2 A schematic diagram of a process of creating a new online programming project provided by an embodiment of the present invention, Figure 3 A schematic diagram of a process for editing basic information of an algorithm model provided by an embodiment of the present invention. Figure 2 and Figure 3 As can be seen in the content, the online interactive programming function of the custom algorithm model can provide users with a real-time, interactive programming environment, enabling them to directly interact with the system, test and adjust the code.

[0059] In some embodiments, the algorithm management module configures the model library management page to establish model classification, provides a multi-level category management method, establishes a standard system according to different model classifications, shields the differences between algorithm models of different systems, and standardizes the model by unifying the description format of the algorithm model to constrain the structure and implementation of the algorithm model. The algorithm management module can allow the same algorithm model to be classified into multiple categories, and by analyzing the correlation and complementarity between models, the algorithm model can be subdivided by category and multiple models can be integrated.

[0060] In some embodiments, the custom algorithm model module is based on data mining, machine learning, and deep learning computing operating environments, see Figure 4A The custom algorithm model module can upload operator models customized based on Python, Java, Scala and other languages ​​through the algorithm management page to meet the needs of scalable algorithm models. At the same time, it supports the export of analysis results and report generation based on file download functions and report generation tools.

[0061] In some embodiments, the algorithm management module may classify and display at least one first algorithm model in response to an input classification operation on at least one first algorithm model. The custom algorithm model module may display a second page for editing at least one target first algorithm model based on an input selection operation on at least one target first algorithm model in at least one first algorithm model. The custom algorithm model module may obtain a second algorithm model corresponding to each target first algorithm model based on an input optimization operation on at least one target first algorithm model.

[0062] In some embodiments, if the target first algorithm model is optimized, the target first algorithm model after the optimization operation is determined as the second algorithm model corresponding to the target first algorithm model. If the target first algorithm model is not optimized, the target first algorithm model is determined as the second algorithm model corresponding to the target first algorithm model.

[0063] That is, see Figure 4B ,The target first algorithm model is the algorithm model selected by the user from at least one first algorithm model. The target first algorithm model after the optimization operation and the target first algorithm model without optimization are both the second algorithm model, that is, the second algorithm model is the algorithm model waiting for the orchestration operation.

[0064] In some embodiments, see Figure 1A ,The algorithm model management unit also includes an algorithm model library interface encapsulation module.

[0065] Among them, the algorithm model library interface encapsulation module provides algorithm model interface management functions, supports the creation, editing, and deletion of algorithm models through APIs. After the algorithm model configuration is completed, the algorithm model training page is viewed through the algorithm model management page to configure key information and then start model training.

[0066] Furthermore, the algorithm model library interface encapsulation module allows for secondary development of the tables of the offline data batch distributed processing and analysis module, and publishes them as online API services and shared data sets. API services support management, flow control, upgrade / downgrade, and security control, while shared data sets support exchange and sharing of process control, data application, approval, and data asset search. The algorithm model library interface encapsulation module can effectively manage the interfaces of algorithm models, ensure their seamless integration and collaboration in the system, enable different types of algorithm models to cooperate with each other through standardized interfaces, and provide the system with flexible algorithm model application capabilities.

[0067] In some embodiments, the algorithm model library interface encapsulation module can encapsulate the calling interface corresponding to each second algorithm model so that the third-party device can call the corresponding second algorithm model based on the corresponding calling interface.

[0068] In some embodiments, see Figure 1A ,The data mining and analysis task management unit includes the ,algorithm service management module and the data mining and analysis ,process orchestration module.

[0069] Among them, the algorithm service management module can be used to register, call and monitor various algorithm services. The data mining analysis process orchestration module customizes the analysis workflow through the process orchestration definition page, sets the data calculation process through drag-and-drop methods, and connects various data processing links through wiring.

[0070] For example, see Figure 5A In (a), the system provided by the present invention can support a device with a display function to display a process arrangement page 500. The process arrangement page 500 includes a control area 510 and an arrangement area 520. Among them, the control area 510 includes controls corresponding to at least one algorithm model to be arranged (for example, algorithm model 1, algorithm model 2, algorithm model 3, etc.). The user can move the controls corresponding to any algorithm model in the control area 510 to the arrangement area 520 by dragging and dropping, so as to arrange the corresponding controls.

[0071] For example, users can Figure 5A The controls corresponding to the algorithm model 1, algorithm model 2, and algorithm model 3 shown in (a) are moved to the layout area 520 respectively, and then, the following may be displayed: Figure 5APage 530 shown in (b) of FIG. The layout area 531 of page 530 displays the controls corresponding to algorithm model 1, algorithm model 2, and algorithm model 3. The user can arrange the execution order of algorithm model 1, algorithm model 2, and algorithm model 3 by connecting the controls corresponding to algorithm model 1, algorithm model 2, and algorithm model 3.

[0072] For example, referring to (b) in 5A, the user can connect the controls corresponding to algorithm model 1 with the controls corresponding to algorithm model 3, starting from the position of the controls corresponding to algorithm model 1 and ending at the position of the controls corresponding to algorithm model 3. Afterwards, the user can connect the controls corresponding to algorithm model 3 with the controls corresponding to algorithm model 2, starting from the position of the controls corresponding to algorithm model 3 and ending at the position of the controls corresponding to algorithm model 2. In this case, the user's arranged process is algorithm model 1-algorithm model 3-algorithm model 2. That is, during execution, algorithm model 1 is executed first, then algorithm model 3, and finally algorithm model 2.

[0073] In some embodiments, see Figure 5A In (a), the control area may also include controls corresponding to process templates (eg, process template 1), and the user may select a control corresponding to any process template (eg, process template 1) to indicate that the corresponding process template is displayed.

[0074] See also Figure 5B , for users Figure 5A After selecting the control corresponding to the process template 1 displayed in (a), the process template 1 displayed on the device supported by this system. Process template 1 includes nodes 541, node 542, node 543, etc. The user can move the controls corresponding to any algorithm model (for example, algorithm model 1) in the control area 550 to any node (for example, node 543) by dragging and dropping. Among them, if there is no control corresponding to the algorithm model that the user has historically moved to the node at node 543, the control corresponding to algorithm model 1 will be directly displayed at node 543. If there is a control corresponding to the algorithm model that the user has historically moved to the node at node 543, the control corresponding to the algorithm model displayed at node 543 will be removed from node 3, and then the control corresponding to algorithm model 1 will be displayed at node 543.

[0075] It can be seen that the business process orchestration of data mining and analysis supports graphical and drag-and-drop modeling, and can achieve in-depth data analysis and model building without coding. It supports users to independently model data mining task processes according to business needs, implement algorithm calls in a drag-and-drop manner, and support the use of wires to connect operators into data mining and analysis workflows. It supports users to select specified process templates and directly move the algorithm model to the corresponding node of the process template to quickly form a mining and analysis workflow, which reduces the difficulty of users in arranging models and is beneficial to user experience. Figure 5C A flow chart of a data mining analysis process arrangement and a new flow provided in an embodiment of the present invention.

[0076] In some embodiments, the data mining and analysis process arrangement module is used to display a third page. The third page is used to arrange the execution order of each second algorithm model. After receiving the arrangement operation for each second algorithm model input on the third page, the data mining and analysis process arrangement module can determine the arrangement process.

[0077] In some embodiments, after determining the orchestration process, the algorithm model library interface encapsulation module may further encapsulate a calling interface corresponding to the orchestration process so that a third-party device may call the orchestration process based on the calling interface.

[0078] In some embodiments, the algorithm service management module may call a determined orchestration process, and perform mining and analysis operations on aerospace engineering data based on the orchestration process to obtain mining and analysis results on the aerospace engineering data.

[0079] In some embodiments, see Figure 1A ,The resource management and task scheduling and monitoring unit includes a resource management module, a multi-task scheduling module and a task monitoring module.

[0080] Among them, the resource management module specifies the mining process through the data mining process definition page. After the data mining process is started, the data mining analysis task is submitted to the distributed computing cluster. The resource management module is responsible for allocating cluster resources for the mining analysis task and scheduling the job and model operators to different computing nodes. The computing cluster supports fault-tolerant processing. When a computing node fails, the resource management engine automatically moves the computing task to other nodes for calculation. Figure 6 A logic flow chart of distributed resource management provided by an embodiment of the present invention.

[0081] In some embodiments, the resource management module may allocate corresponding resources to the at least one orchestration process based on preset rules when executing the at least one orchestration process.

[0082] Among them, the resource management and task scheduling monitoring unit provides user authority and data authority management, enhances system operation and maintenance capabilities through automatic inspection and status monitoring functions, supports resource scheduling, meets the needs of different users for the division of different resource queues, and realizes resource isolation. In addition, the unit provides a message center that enables users to view information such as system resources and task status.

[0083] The resource management module is responsible for effectively managing resources in a distributed environment to ensure their rational use and efficient allocation. This module covers hardware resources (such as computing nodes, storage devices) and software resources (such as dependencies), and its goal is to optimize resource utilization and overall performance.

[0084] The multi-task scheduling module focuses on the reasonable allocation of tasks to available resources, and works closely with the resource management module to achieve intelligent scheduling and distribution of tasks based on resource availability and task requirements. The multi-task scheduling module and the resource management module work together to ensure that system resources are fully utilized and improve overall computing efficiency.

[0085] The task monitoring module is responsible for monitoring and recording the execution of tasks, including running status, progress, and consumed resources. It works in conjunction with the multi-task scheduling module to monitor the execution of tasks in real time, respond to potential problems in a timely manner, and optimize task scheduling strategies. Resource management provides basic resource support, multi-task scheduling reasonably allocates tasks based on it, and task monitoring monitors task execution in real time to provide timely feedback for scheduling and resource management. The three modules together construct the resource management and task scheduling monitoring unit.

[0086] In some embodiments, the multi-task scheduling module configures the task scheduling strategy through the process configuration page, and supports various types of task scheduling strategies such as first-in-first-out, priority sorting, and polling. The task scheduling method is configured through the task process definition, and supports scheduled scheduling, dependent scheduling, manual scheduling, manual recovery, and other scheduling of tasks. The multi-task scheduling module provides a task management interface, and performs task management operations through the task management page, supporting multi-task management, including task deletion, resubmission, etc. During the operation of the task process, the operation log and system log are recorded in real time and classified. The operation log and system usage log can be queried in real time through the task operation management page, and the operation status of the data analysis and orchestration process can be timely grasped. Fig. 7A A logic flow chart of multi-type task scheduling provided by an embodiment of the present invention.

[0087] In some embodiments, the multi-task scheduling module may determine the execution priority of each of the at least two orchestration processes when executing at least two orchestration processes.

[0088] In some embodiments, the task monitoring module supports visual monitoring of resource usage and task operation scheduling, including monitoring of the system's own operating status indicators, resource usage, job operation status, job operation logs, execution flow, exception handling, etc., real-time monitoring and charting of monitoring indicators, support for configuration of thresholds, alarm strategies, and alarm methods for key indicators, and alarms for dangerous situations of the system's key indicators.

[0089] In some embodiments, the task monitoring module may monitor the execution information of the target orchestration process, wherein the target orchestration process includes an unexecuted orchestration process and an executed orchestration process.

[0090] The uniqueness of the system provided by the present invention lies in its flexibility in multi-source data access, supporting multiple data sources such as JDBC, Hive, HBase, HDFS, etc., and applying multiple databases to solve the demand for data storage and computing in the overall project stage and subsequent application implementation stage, including relational database Mysql, real-time database Hyperbase, distributed search and analysis engine Scope, and distributed transaction database KunDB. Each database is responsible for storing different types of data, from structured data to text data, and has the ability to achieve dynamic expansion and continuous expansion of computing based on current data storage.

[0091] Furthermore, the core components of the system provided by the present invention adopt a highly available microservice architecture based on containerized operation and scheduling. The front-end entrance adopts the Nginx multi-active solution to provide stable entrance services. The back-end instance uses the distributed Key-val ue storage Etcd developed based on the Raft protocol for registration and discovery, and adopts multi-active deployment to quickly implement master-slave node switching to ensure the stable operation of the system. In terms of data storage, it relies on the distributed cache layer Redis to manage status information, and the file storage service relies on the multi-copy distributed file system HDFS to ensure high availability. The computing engine adopts the mature distributed computing engine Spark on Yarn, which supports task retries.

[0092] Furthermore, the custom algorithm model module in the present invention provides two modes: visual modeling and online interactive programming. The system automatically manages the algorithm version according to the algorithm editing history to solve the problem of chaotic code management, make the version control clearer and more orderly, and have the ability to trace back to any version, while being able to trace the development history of the algorithm. The function of publishing the algorithm allows the user to set the edited algorithm to the published state, thereby improving the reusability and sharing of the algorithm. Through the front-end process arrangement page, users can combine multiple algorithms in an orderly and efficient manner to achieve the orderliness of data processing and analysis. After submitting the completed process, the system automatically triggers data processing. After the background service receives the process trigger request, it automatically generates the process execution script and submits it to the data processing engine. At the same time, the system's built-in high-performance distributed algorithm can cope with large-scale and highly complex data application scenarios by integrating computing and optimizing algorithm performance.

[0093] Furthermore, the algorithm service management module in the present invention has key functions to ensure the effectiveness of model services. First, through service testing, the availability of services can be quickly verified, and the interface status of online services can be tested in real time. With feature parameters in JSON format as input, the system simulates the operating status of the real environment API and returns the service reasoning results immediately. Secondly, the service audit function allows users to view all monitoring indicator contents of a single service, and display them independently by version, providing a comprehensive operating status view. In terms of model evaluation, the system compares the actual result data fed back by the business offline with the predicted data generated when the model is running, obtains multi-dimensional performance evaluation index values, and automatically generates a model evaluation report, providing an important reference for model iteration. Finally, through a variety of interpretable algorithms, the influence weight of each feature input parameter on the model prediction result is graphically displayed, which can quickly locate and optimize important feature factors to improve model accuracy.

[0094] Furthermore, the task monitoring module ensures the stability of the model service through a variety of monitoring functions and early warning rules. First, the task monitoring module can monitor the running status of the model service in real time, focusing on key indicators such as cluster resource utilization, throughput, usage, response time and access records, so as to effectively control potential risks in the model production environment. Secondly, the task monitoring module supports the monitoring of the distribution differences between the predicted data and the training data of the model service. Through the statistics of the maximum value, mean, standard deviation and distribution interval comparison of the feature dimension, data deviation indicators such as PSI and JS divergence are used to reduce the degradation of model prediction performance due to changes in the input data distribution. Finally, the pre-task monitoring module allows administrators to customize the configuration of early warning rules, covering data deviation indicators such as PSI, JS divergence, and service resource indicators such as Figure 7B The host memory usage, host CPU usage, etc. in the monitoring interface diagram shown.

[0095] Furthermore, in the process of publishing algorithm model services, first of all, service parameters can be configured through zero-code one-click deployment, and service APIs can be quickly generated, which simplifies the model publishing process. Secondly, it supports custom configuration of elastic scaling strategies, and uses dynamic scaling to achieve load balancing, ensuring that the system can flexibly adjust resource allocation when dealing with changing workloads. The system also has multi-version traffic distribution strategies, including grayscale release and A / B testing capabilities, which can flexibly control traffic distribution for different model versions, evaluate performance and gradually promote new versions. It supports on-demand allocation of resource usage of each node in the inference graph, including CPU, GPU, and memory, and provides refined resource management to meet the hardware resource requirements of different models. Finally, after the model is released, it supports both online and offline modes. The offline mode especially supports batch prediction, which obtains batch data for model prediction by directly connecting to the database, and writes the results back to the corresponding library table to achieve closed-loop data processing and model application.

[0096] In some embodiments, the present invention also provides a method for mining and analyzing aerospace engineering big data, see Figure 8 , the method comprises the following steps:

[0097] S801: In response to an input mining and analysis service request, preprocess the aerospace engineering data, obtain and display the preprocessed aerospace engineering data, and display the first page.

[0098] The mining and analysis service request is used to request to perform a mining and analysis operation on aerospace engineering data. The first page includes at least one first algorithm model, and the first algorithm model is used to perform mining and analysis on the data.

[0099] S802: Based on the input selection operation of at least one target first algorithm model in at least one first algorithm model, a second algorithm model corresponding to each target first algorithm model is obtained.

[0100] S803: Determine an orchestration process based on the input orchestration operation for each second algorithm model.

[0101] The arrangement operation includes determining the execution order of each second algorithm model.

[0102] S804: Execute a mining and analysis operation on the aerospace engineering data based on the orchestration process to obtain a mining and analysis result on the aerospace engineering data.

[0103] In some embodiments, when at least one orchestration process is executed, corresponding resources may be allocated to the at least one orchestration process based on a preset rule. When at least two orchestration processes are executed, the priority of execution of each of the at least two orchestration processes may be determined. Also, execution information of a target orchestration process may be monitored. The target orchestration process includes an unexecuted orchestration process and / or an executed orchestration process.

[0104] Combine the following Fig. 9 The specific embodiment shown provides a detailed description of an aerospace engineering big data mining and analysis method provided by the present invention.

[0105] S91: Create (or upload) a multi-source scientific experiment data mining algorithm model through the front-end page. In order to achieve algorithm creation, Fig.10 A schematic diagram of an algorithm creation process provided by the present invention includes the steps of inputting an algorithm name, verifying the algorithm name, storing the algorithm in a database, and creating an algorithm path.

[0106] S92: After creating the algorithm, the algorithm model is edited online in the online interactive programming component to adjust and optimize the various parameters of the algorithm. Fig.11 A schematic diagram of an algorithm editing process provided by the present invention includes the steps of querying an algorithm, clicking on the algorithm to be edited, opening the algorithm editor, editing the algorithm, and automatically saving.

[0107] S93: After editing is completed, basic operations such as adding, deleting, modifying, and checking the algorithm model can be performed through the algorithm model library management page. At the same time, the system automatically manages the algorithm version: the algorithm version is automatically numbered according to the algorithm editing history, and the version is automatically changed after each editing. In order to achieve algorithm version management, Fig.12 A schematic diagram of an algorithm management process provided by the present invention includes the steps of querying the algorithm, editing the algorithm, closing the editing and automatically saving the algorithm, querying the current algorithm version, modifying the algorithm version, and prompting success information.

[0108] S94: Publish the algorithm, set the edited algorithm to the published state so that it can be used in subsequent data processing and analysis. Fig.13 A schematic diagram of an algorithm publishing process provided by the present invention includes the steps of querying an algorithm, publishing an algorithm, determining whether it is a repeated publication, modifying the algorithm status, entering the algorithm publishing library, and prompting the publishing result.

[0109] S95: Add operator templates through the front-end process arrangement page, and set the parameters and functions of the operators according to actual needs. After the operator template is set, multiple algorithms can be combined in a certain order and logic by dragging and dropping to form a data mining and analysis process. In order to achieve process arrangement, Fig.14 A process arrangement flow diagram provided by the present invention includes creating a new algorithm template, creating a new process, and editing a process.

[0110] S96: After submitting the completed process, the system automatically triggers the data processing process; after receiving the process trigger request, the background service will generate a corresponding execution script based on the data processing process script submitted by the user, and submit the script to the data processing engine for data processing; the data processing engine will verify the submitted script, and start the data processing script after the verification passes. Then, the operator execution command is sent to the client, and the client performs the corresponding data processing operation. In order to realize process submission, Fig.15 A schematic diagram of a process submission process provided by the present invention includes querying the process, clicking to submit, and returning the submission result. When the user clicks on the process library, a list of process names is displayed. The user can quickly search for a process by process name through the process library quick search function. After clicking on the process, the process arrangement content is displayed. Clicking the start button prompts that the start is successful and returns the process batch number.

[0111] S97: After receiving the calculation command, the client submits the calculation task according to the user's needs and configures it to process and analyze the data. Fig.16 A schematic diagram of an algorithm calculation process provided by the present invention includes receiving an execution operator, parsing an execution command, determining resources, submitting a job, and returning an execution result.

[0112] S98: Use visualization tools to mine and analyze the processed data, and use intuitive charts and reports to display the results of data mining and analysis. Fig.17 A schematic diagram of a mining and analysis process provided by the present invention.

[0113] S99: The system outputs the results of data mining and analysis in the form of charts, reports, etc., so that decision makers and business personnel can quickly understand and apply these results, thereby making more informed decisions and formulating corresponding business strategies. At the same time, users can also export these results locally for further analysis and use. In order to achieve the result export, Fig.18 A result export process diagram provided by the present invention includes selecting to view jobs on a visualization page, clicking to view running results, and clicking to download.

[0114] Compared with the prior art, the system and method provided by the present invention realizes the functions of massive data collection, data quality management, data integration management, distributed computing analysis, data analysis task control and resource management, algorithm model library management, etc., integrates typical and general model algorithms related to big data mining and analysis, and provides technical and capability support for various data modeling and predictive analysis algorithms. In the implementation process, a systematic process is formed by constructing a data mining system. First, data information is collected and the data quality is identified, followed by preliminary screening. On the collected data, data preprocessing is performed and views are established for model training, and corresponding evaluations are performed through evaluation models, and visual display is performed. The resource management and task scheduling monitoring module monitors potential risk issues, and the evaluation module compares the results of the model output with the actual results to further evaluate the model. The subsequent data mining steps are determined according to the evaluation results and adjusted accordingly to shorten the overall mining analysis cycle. In addition, the analysis results of data mining are managed, visualized and stored, which improves the intelligence level and security of mining data management.

[0115] In some schemes, multiple embodiments of the present application can be combined, and the combined scheme can be implemented. Optionally, some operations in the process of each method embodiment are optionally combined, and / or the order of some operations is optionally changed. In addition, the execution order between the steps of each process is only exemplary and does not constitute a restriction on the execution order between the steps. There can also be other execution orders between the steps. It is not intended to indicate that the execution order is the only order in which these operations can be performed. A person of ordinary skill in the art will think of a variety of ways to reorder the operations described herein. In addition, it should be noted that the process details involved in a certain embodiment of this article are also applicable to other embodiments in a similar manner, or different embodiments can be used in combination.

[0116] In addition, some steps in the method embodiment may be equivalently replaced by other possible steps. Alternatively, some steps in the method embodiment may be optional and may be deleted in certain usage scenarios. Alternatively, other possible steps may be added to the method embodiment. Furthermore, the method embodiments may be implemented separately or in combination.

[0117] Through the description of the above implementation methods, technical personnel in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the system can be divided into different functional modules to complete all or part of the functions described above.

[0118] In the several embodiments provided in the present application, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are only schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of systems or units, which can be electrical, mechanical or other forms.

[0119] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0120] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium, including several instructions to enable a device (which can be a single-chip microcomputer, chip, etc.) or a processor (processor) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program code.

[0121] The above contents are only specific implementation methods of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions within the technical scope disclosed in the present application shall be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims

1. A space engineering big data mining and analysis system, characterized in that: include: A big data computing basic engine unit is used to pre-process aerospace engineering data in response to an input mining and analysis service request, obtain and display the pre-processed aerospace engineering data, and display a first page; the mining and analysis service request is used to request to perform a mining and analysis operation on the aerospace engineering data; the first page includes at least one first algorithm model; the first algorithm model is used to perform mining and analysis on the data; An algorithm model management unit, configured to obtain a second algorithm model corresponding to each of the target first algorithm models based on an input selection operation of at least one target first algorithm model in the at least one first algorithm model; A data mining and analysis task management unit: used for determining an arrangement process based on an arrangement operation inputted for each of the second algorithm models, and performing a mining and analysis operation on the aerospace engineering data based on the arrangement process to obtain a mining and analysis result on the aerospace engineering data; the arrangement operation includes determining an execution order of each of the second algorithm models; Wherein, the big data computing basic engine unit also includes a machine learning computing engine module and a deep learning computing engine module; The machine learning computing engine module is used to train a first initial model based on training data to obtain a second initial model; the training data is associated with the aerospace engineering data; the first initial model is a model created by a user and uploaded to the aerospace engineering big data mining and analysis system for mining and analyzing data; The deep learning computing engine module is used to train the first initial model based on the training data to obtain a third initial model; the third initial model is different from the second initial model; the first algorithm model is determined based on the second initial model or the third initial model; The algorithm model management unit includes an algorithm management module and a custom algorithm model module; the algorithm model management unit is used to select at least one target first algorithm model in the at least one first algorithm model based on the input, and obtain the second algorithm model corresponding to each target first algorithm model: The algorithm management module is used for classifying and displaying the at least one first algorithm model in response to the input classification operation of the at least one first algorithm model; The custom algorithm model module is used to display a second page for editing the at least one target first algorithm model based on an input selection operation of at least one target first algorithm model in the at least one first algorithm model; The custom algorithm model module is also used to obtain the second algorithm model corresponding to each target first algorithm model based on an optimization operation of at least one of the target first algorithm models input; wherein, if the target first algorithm model is subjected to an optimization operation, the target first algorithm model after the optimization operation is determined as the second algorithm model corresponding to the target first algorithm model; if the target first algorithm model is not subjected to an optimization operation, the target first algorithm model is determined as the second algorithm model corresponding to the target first algorithm model.

2. The system according to claim 1, characterized in that The big data computing basic engine unit includes an offline data batch distributed processing and analysis module, which is used to respond to the input mining and analysis service request, pre-process the aerospace engineering data, obtain and display the pre-processed aerospace engineering data, and the offline data batch distributed processing and analysis module is used to analyze the aerospace engineering data in a distributed batch processing manner to obtain the pre-processed aerospace engineering data.

3. The system according to claim 2, characterized in that The algorithm model management unit also includes an algorithm model library interface encapsulation module, and after the algorithm model management unit is used to obtain the second algorithm model corresponding to each of the target first algorithm models, the algorithm model library interface encapsulation module is used to encapsulate the calling interface corresponding to each of the second algorithm models; the calling interface corresponding to the second algorithm model is used to support a third-party device to call the corresponding second algorithm model; Furthermore, after the data mining and analysis task management unit is used to determine the orchestration process, the algorithm model library interface encapsulation module is also used to encapsulate the calling interface corresponding to the orchestration process; the calling interface corresponding to the orchestration process is used to support third-party devices to call the orchestration process.

4. The system according to claim 3, characterized in that The data mining and analysis task management unit includes a data mining and analysis process arrangement module; The data mining and analysis task management unit is used to determine, based on the input arrangement operation for each of the second algorithm models, that in the arrangement process, the data mining and analysis process arrangement module is used to display a third page; the third page is used to arrange the execution order of each of the second algorithm models; The data mining analysis process orchestration module is further used to receive the orchestration operation input on the third page for each of the second algorithm models, and determine the orchestration process.

5. The system according to claim 4, characterized in that The data mining and analysis task management unit includes an algorithm service management module; the data mining and analysis task management unit is used to perform mining and analysis operations on the aerospace engineering data based on the orchestration process to obtain mining and analysis results of the aerospace engineering data; the algorithm service management module is used to call the orchestration process, and, based on the orchestration process, perform mining and analysis operations on the aerospace engineering data to obtain mining and analysis results of the aerospace engineering data.

6. The system according to claim 5, characterized in that It also includes a resource management and task scheduling monitoring unit; the resource management and task scheduling monitoring unit includes a resource management module, a multi-task scheduling module and a task monitoring module; The resource management module is used to allocate corresponding resources to the at least one orchestration process based on preset rules when executing the at least one orchestration process; The multi-task scheduling module is used to determine the execution priority of each of the at least two orchestration processes when executing at least two orchestration processes; The task monitoring module is used to monitor the execution information of the target orchestration process, and the target orchestration process includes an unexecuted orchestration process and an executed orchestration process.

7. A method for mining and analyzing aerospace engineering big data, characterized in that: include: In response to an input mining and analysis service request, preprocessing the aerospace engineering data, obtaining and displaying the preprocessed aerospace engineering data, and displaying a first page; the mining and analysis service request is used to request to perform a mining and analysis operation on the aerospace engineering data; the first page includes at least one first algorithm model; the first algorithm model is used to perform mining and analysis on the data; Based on the input selection operation of at least one target first algorithm model in the at least one first algorithm model, obtaining a second algorithm model corresponding to each of the target first algorithm models; Determine an orchestration process based on an input orchestration operation for each of the second algorithm models; the orchestration operation includes determining an execution order of each of the second algorithm models; Performing a mining and analysis operation on the aerospace engineering data based on the orchestration process to obtain a mining and analysis result on the aerospace engineering data; Wherein, the method further comprises: Training a first initial model based on training data to obtain a second initial model; the training data is associated with the aerospace engineering data; the first initial model is a model created and uploaded by a user for mining and analyzing data; The first initial model is trained based on the training data to obtain a third initial model; the third initial model is different from the second initial model; the first algorithm model is determined based on the second initial model or the third initial model; The operation of selecting at least one target first algorithm model in the at least one first algorithm model based on the input, to obtain a second algorithm model corresponding to each of the target first algorithm models, includes: In response to an input classification operation on the at least one first algorithm model, classifying and displaying the at least one first algorithm model; Based on an input selection operation of at least one target first algorithm model among the at least one first algorithm model, displaying a second page for editing the at least one target first algorithm model; Based on an optimization operation input to at least one of the target first algorithm models, a second algorithm model corresponding to each of the target first algorithm models is obtained; wherein, if the target first algorithm model is subjected to an optimization operation, the target first algorithm model after the optimization operation is determined as the second algorithm model corresponding to the target first algorithm model; if the target first algorithm model is not subjected to an optimization operation, the target first algorithm model is determined as the second algorithm model corresponding to the target first algorithm model.

8. The method according to claim 7, characterized in that The mining and analysis operation of the aerospace engineering data based on the orchestration process also includes: In the case of executing at least one orchestration process, allocating corresponding resources to the at least one orchestration process based on preset rules; In the case of executing at least two orchestration processes, determining a priority for executing each of the at least two orchestration processes; Monitoring execution information of a target orchestration process, where the target orchestration process includes an unexecuted orchestration process and / or an executed orchestration process.

Images