Customized processing method and system based on multi-source real-time spaceflight measurement and control data

Abstract

The invention discloses a customized processing method and system based on multi-source real-time spaceflight measurement and control data, and the method comprises the steps: carrying out the unified storage and classification of multi-type data source data, distributing data source types which can be provided, distributing a customer system occupation seat, and enabling a customer to subscribe according to the distributed data source types. And configuring a data source type required by a client, calling a corresponding data processing configuration scheme according to the data source type required by the client, and automatically executing a data processing task of the data source of the corresponding type. The invention provides an extensible multi-source data configuration function, data of various types of data sources can be subjected to configuration connection, multi-type configuration of the data of the data sources is realized, the required types of the data sources in a spaceflight measurement and control service system can be basically covered, and one set of system can meet the data application requirements of a multi-client system; and utilization of development resources is reduced, and application and popularization of spaceflight measurement and control data are facilitated.

Description

Technical Field

[0001] The present invention relates to the technical field of collecting and processing aerospace measurement and control data, and in particular to a customized processing method and system based on multi-source real-time aerospace measurement and control data. Background Art

[0002] With the development of the aerospace industry, there are more and more requirements for data reception and processing in the field of aerospace measurement. The types and formats of data involved are becoming more and more complex. The requirements for receiving and customized processing of multi-source real-time data are becoming higher and higher.

[0003] The aerospace measurement and control data source receiving method is different, and there are differences in data message and data encoding format, such as target attitude data, target position data, space environment data, safety control information, etc. At present, common application systems receive data through fixed data sources and fixed encoding and decoding formats, and then develop data processing systems separately to perform data cleaning, data governance and other operations. In addition, data storage and subsequent playback control require separate system customization and development. In the subsequent data association processing, separate function development is also required, which consumes a lot of resources and affects the application of aerospace measurement and control data.

[0004] The patent document with application number 202110289044.0 discloses a method and device for real-time interaction of aerospace measurement and control data. The topological architecture of the scheme published by it has good scalability, and the functions of the central server are distributed to each node. There is no master-slave distinction between each network node, which overcomes the problems of single point failure and local network congestion. Based on the P2P architecture status message transmission, it does not pass through any intermediate nodes and has high real-time performance. However, its scheme also has the problem that multiple types of data sources cannot be centrally processed, which affects the application of aerospace measurement and control data. Summary of the invention

[0005] In view of the above-mentioned problems, the purpose of the present invention is to provide a customized processing method and system based on multi-source real-time aerospace measurement and control data, which can perform customized processing on data from multiple types of data sources and realize expanded processing of new types of data.

[0006] The embodiments of the present invention provide a customized processing method and system based on multi-source real-time aerospace tracking and control data.

[0007] Aspect 1: A customized processing method based on multi-source real-time aerospace measurement and control data, comprising:

[0008] S1. Unified storage and classification of data from multiple data sources;

[0009] S2. Distribute the available data source data types and allocate customer system occupied seats;

[0010] S3. The customer subscribes according to the distributed data source data type and configures the data source data type required by the customer;

[0011] S4. Based on the data type of the data source required by the customer, call the corresponding data processing configuration plan to automatically execute the data processing task of the corresponding type of data source.

[0012] Optionally: multi-type data source data is classified into streaming data and batch data, wherein streaming data includes message middleware type data and socket type data; batch data includes file type data and database type data.

[0013] Optionally: middleware type data includes rabbitmq data and kafka data; socket type data includes unicast data, multicast data and websocket data; file type data includes local storage file data and network storage file data; database type data includes DAMO database data, MySQL database data and PostgreSQL database data.

[0014] Optionally: Data processing configuration scheme generation includes, based on the external data import technology of the Flink framework, designing a streaming data import interface and a batch data import interface, respectively importing streaming data and batch data, respectively parsing, assembling and configuring information of different types of streaming data and batch data, and obtaining different types of data processing configuration schemes.

[0015] Aspect 2: A customized processing system based on multi-source real-time aerospace measurement and control data, including:

[0016] Data source management module, adding, deleting and editing data source data, and managing the multi-type data source data formed;

[0017] The task flow management module manages and executes data processing tasks for multiple types of data sources;

[0018] The subscription distribution module is used to distribute data source data types, allocate customer system occupied seats, configure the data source data types subscribed by customers, and automatically execute the corresponding data processing configuration plan according to the data source data types subscribed by customers to complete the data processing tasks of multiple types of data sources;

[0019] The user rights management module allocates and manages system user rights.

[0020] Optionally: the system also includes a processing plug-in management module, which adds, deletes or edits corresponding algorithm plug-ins according to changes in data types of data sources, and generates corresponding data processing configuration plans.

[0021] Optionally: the task flow management module includes a task management module and a task execution module, wherein:

[0022] The task management module manages the topological link diagram and task information storage of data processing tasks of multiple types of data sources;

[0023] The task execution module executes the corresponding data processing configuration plan according to the data type of the imported data source, completes the data processing tasks of multiple types of data sources, saves the processed data output, and provides it to the client system.

[0024] A third aspect: An electronic device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of the method provided in the first aspect are implemented.

[0025] A fourth aspect: A non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method provided in the first aspect.

[0026] Beneficial effects of the present invention:

[0027] 1. The present invention provides an expandable multi-source data configuration function, which can configure and connect various types of data sources to achieve multi-type configuration of data source data, and can basically cover the required types of data sources in aerospace measurement and control business systems. One system can meet the data application needs of multiple customer systems, reduce the utilization of development resources, and is conducive to the application and promotion of aerospace measurement and control data.

[0028] 2. The system of the present invention transforms the real-time data stream processing framework Flink to generate a set of plug-in data processing flows, which can develop data processing modules in a plug-in manner and freely combine data processing flows to realize customized processing functions for different types of data. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 It is a flow chart of a customized processing method based on multi-source real-time aerospace measurement and control data of the present invention;

[0030] Figure 2 It is a structural schematic diagram of a customized processing system based on multi-source real-time aerospace measurement and control data of the present invention;

[0031] Figure 3 A schematic diagram of the process of managing classified access to data sources of the present invention;

[0032] Figure 4 This is a schematic diagram of the principle of the Flink framework external data connection technology of the present invention;

[0033] Figure 5A schematic diagram of data configuration of multiple data sources by the processing system of the present invention;

[0034] Figure 6 A schematic diagram of data configuration of multiple data sources by the processing system of the present invention;

[0035] Figure 7 This is a schematic diagram of operator plug-in management for the processing system of the present invention;

[0036] Figure 8 The processing system job flow job configuration of the present invention;

[0037] Fig. 9 The present invention processes the system job flow job configuration;

[0038] Fig.10 It is a schematic diagram of the structure of the electronic device of the present invention. DETAILED DESCRIPTION

[0039] The embodiments of the present invention are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar symbols throughout represent the same or similar elements or elements with the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be understood as limiting the present invention.

[0040] In aerospace measurement and control systems, real-time reception of multi-type source data is often done through fixed data source configurations, and the imported data is processed using fixed parsing methods. This makes it impossible to expand data import and expansion parsing of newly added data types, which is very inconvenient for system expansion deployment.

[0041] In view of the above problems, the present invention provides a customized processing method and system based on multi-source real-time aerospace measurement and control data. Figure 2 A schematic diagram of the structure of a customized processing system based on multi-source real-time aerospace tracking and control data provided by an embodiment of the present invention.

[0042] The system includes a task flow management module, a data management module, a subscription distribution module, a processing plug-in management module and a user authority management module.

[0043] The data source management module is used to add, delete and edit data source data, manage the multi-type data source data formed, and realize the unified storage and classification management of multi-type data source data.

[0044] The task flow management module is used to manage and execute data processing tasks for multiple types of data sources.

[0045] The task flow management module includes a task management module and a task execution module, wherein: the task management module manages the topological link diagram and task information storage of multi-type data source data processing tasks; the task execution module executes the corresponding data processing configuration plan based on the introduced data source data type, completes multi-type data source data processing tasks, saves the processed data output, and provides it to the client system.

[0046] The subscription distribution module is used to distribute data source data types, allocate customer system occupied seats, configure the data source data types subscribed by customers, and automatically execute the corresponding data processing configuration plan based on the data source data types subscribed by customers to complete multi-type data source data processing tasks.

[0047] The user rights management module is used to allocate and manage system user rights.

[0048] The processing plug-in management module is used to add, delete or edit corresponding algorithm plug-ins according to changes in data source data types.

[0049] Using the above system, the embodiment of the present invention also discloses a customized processing method based on multi-source real-time aerospace measurement and control data. Figure 1 A flow chart of a customized processing invention based on multi-source real-time aerospace measurement and control data provided by an embodiment of the present invention, the method includes:

[0050] S1. Unified storage and classification of data from multiple types of data sources.

[0051] S2. Distribute the available data source data types and allocate customer system occupied seats;

[0052] S3. The customer subscribes according to the distributed data source data type and configures the data source data type required by the customer;

[0053] S4. Based on the data type of the data source required by the customer, call the corresponding data processing configuration plan to automatically execute the data processing task of the corresponding type of data source.

[0054] By adopting the above system and method and utilizing the data source management module, unified storage and classified management of various types of data source data can be achieved.

[0055] Among them, multi-type data source data is classified into streaming data and batch data. Streaming data includes message middleware type data and socket type data, etc.; batch data includes file type data and database type data, etc.

[0056] Middleware type data includes rabbitmq data and kafka data; socket type data includes unicast data, multicast data and websocket data; file type data includes local storage file data and network storage file data; database type data includes DAMO database data, MySQL database data, PostgreSQL database data, etc.

[0057] For different types of data sources, corresponding data processing configuration plans are formed, such as Figure 3 As shown:

[0058] Based on the external data connection technology of the Flink framework, the design of streaming data connection interface and batch data connection interface is implemented. Different interfaces accept different types of data source data to connect the data to the task execution module. The task execution module parses, assembles and configures different types of streaming data and batch data respectively to obtain different types of data processing configuration schemes. Each data processing configuration scheme corresponds to a type of data source data. For example, when the data source data is unicast socket type streaming data, the corresponding unicast data processing configuration scheme processes the unicast data stream in real time to obtain a data stream that meets the customer system requirements.

[0059] like Figure 2 As shown, after the system is started, the system distributes the data source data types that can be processed to the client system through the subscription distribution module. The client system subscribes to the required data source data types according to the distribution announcement, allocates the interface seat for the client system to access, and further configures the data source data types required by the client.

[0060] Based on the customer's system subscription, the system will automatically call the corresponding type of data processing configuration plan to automatically configure and validate the data source data.

[0061] Furthermore, in order to achieve the scalability of multi-source data types, the plug-in upload method can be adopted. The algorithm plug-in for streaming data or batch data is uploaded to the system, and the system automatically loads and takes effect, dynamically realizing the expansion of data source data types.

[0062] When customizing algorithm plug-ins and processing data from data sources, you can obtain the data processing configuration solution workflow based on the external data connection technology of the Flink framework.

[0063] Flink framework external data access technology is a method of accessing, processing and distributing a set of data based on job flow. Figure 4 As shown, it mainly includes three parts: Source, Operator and Sink.

[0064] Source provides data input function, Operator belongs to the operator function and provides data processing capability, Sink belongs to data output and can push processed data externally. Flink job flow jobs also use Taskmanager function for management and scheduling.

[0065] First, we use the job configuration function of the Flink framework to design and implement a functional module for automatic job flow generation. This functional module can combine the data import function of multiple types of source data to jointly realize the generation of Source nodes.

[0066] The data source data processing algorithm is loaded in the generated job flow. Specifically, a set of data processing algorithm class management modules can be established to manage the design-based streaming data algorithm and batch data algorithm processing classes. Your own data source data algorithm processing class can be developed through inheritance to implement new types of data source data processing algorithms.

[0067] The system uses the processing plug-in management module to set up the upload, parsing and loading functions of the algorithm jar package. Users can package the data algorithm processing class they edited into a jar package, upload it to the system as an algorithm plug-in of the data processing configuration plan, and it will take effect dynamically.

[0068] During the system processing, the dataflow component can also be used in the data processing workflow to realize the visual drag-and-drop editing function of the workflow. Users can freely edit the operators in the workflow and configure complex data source data processing processes. The configured data processing configuration solution workflow is saved in the system's background database in a tree structure.

[0069] The system reads the tree configuration information of the data processing configuration scheme job flow, uses the job startup configuration of the Flink framework taskmanager module, traverses the tree nodes of the job flow, and loads and starts the data processing configuration scheme.

[0070] In this way, a customized multi-type data source data processing flow is started and completed. After startup, the data processing configuration scheme job flow can run the real-time streaming data and batch data according to the pre-configured data processing configuration scheme processing algorithm. The final data stream is saved and pushed to the customer system.

[0071] like Figure 5 and Figure 6 As shown in the figure, it is a schematic diagram of the system configuration for multiple data sources. Figure 7 As shown in the figure, it is a schematic diagram of system processing operator plug-in management. Figure 8 and Fig. 9 The following is a schematic diagram of the job flow job configuration.

[0072] The present invention also provides an electronic device, Fig.10 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention, such as Fig.10 As shown, the electronic device may include: a processor, a communications interface, a memory, and a communications bus, wherein the processor, the communications interface, and the memory communicate with each other via the communications bus. The processor may call the logic instructions in the memory, for example, to execute the following method:

[0073] S1. Unified storage and classification of data from multiple data sources;

[0074] S2. Distribute the available data source data types and allocate customer system occupied seats;

[0075] S3. The customer subscribes according to the distributed data source data type and configures the data source data type required by the customer;

[0076] S4. Based on the data type of the data source required by the customer, call the corresponding data processing configuration plan to automatically execute the data processing task of the corresponding type of data source.

[0077] In addition, the logic instructions in the above-mentioned memory can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on such an understanding, the technical solution of the present invention can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0078] An embodiment of the present invention further provides a non-transitory computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the method provided in each of the above embodiments is implemented, for example, including:

[0079] S1. Unified storage and classification of data from multiple data sources;

[0080] S2. Distribute the available data source data types and allocate customer system occupied seats;

[0081] S3. The customer subscribes according to the distributed data source data type and configures the data source data type required by the customer;

[0082] S4. Based on the data type of the data source required by the customer, call the corresponding data processing configuration plan to automatically execute the data processing task of the corresponding type of data source.

[0083] The system embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, i.e., they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Those of ordinary skill in the art may understand and implement it without creative effort.

[0084] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods of each embodiment or some parts of the embodiment.

[0085] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A customized processing method based on multi-source real-time aerospace measurement and control data, characterized in that: include: S1. Unified storage and classification of data from multiple data sources; S2. Distribute the available data source data types and allocate customer system occupied seats; S3. The customer subscribes according to the distributed data source data type and configures the data source data type required by the customer; S4. Based on the data type of the data source required by the customer, call the corresponding data processing configuration plan to automatically execute the data processing task of the corresponding type of data source.

2. The customized processing method based on multi-source real-time aerospace measurement and control data according to claim 1 is characterized in that: Multi-type data source data is classified into streaming data and batch data, where: Streaming data includes message middleware type data and socket type data; Batch data includes file type data and database type data.

3. The customized processing method based on multi-source real-time aerospace measurement and control data according to claim 2 is characterized in that: Middleware type data includes rabbitmq data and kafka data; Socket type data includes unicast data, multicast data and websocket data; File type data includes local storage file data and network storage file data; Database type data includes DAMO database data, MySQL database data, and PostgreSQL database data.

4. The customized processing method based on multi-source real-time aerospace measurement and control data according to claim 1 is characterized in that: Data processing configuration scheme generation includes: Based on the external data import technology of the Flink framework, we design streaming data import interfaces and batch data import interfaces to import streaming data and batch data respectively. We parse, assemble and configure different types of streaming data and batch data respectively to obtain different types of data processing configuration solutions.

5. A customized processing system based on multi-source real-time aerospace measurement and control data applied to any one of the methods of claims 1 to 4, characterized in that: The system includes: Data source management module, adding, deleting and editing data source data, and managing the multi-type data source data formed; The task flow management module manages and executes data processing tasks for multiple types of data sources; The subscription distribution module distributes data source data types, allocates customer system occupied seats, configures the data source data types subscribed by customers, and automatically executes the corresponding data processing configuration plan based on the data source data types subscribed by customers to complete multi-type data source data processing tasks; The user rights management module allocates and manages system user rights.

6. The customized processing system based on multi-source real-time aerospace measurement and control data according to claim 5 is characterized in that: The system also includes: The processing plug-in management module adds, deletes or edits the corresponding algorithm plug-ins according to the changes in the data type of the data source, and generates the corresponding data processing configuration plan.

7. The customized processing system based on multi-source real-time aerospace measurement and control data according to claim 5 is characterized in that: The task flow management module includes a task management module and a task execution module, wherein: The task management module manages the topological link diagram and task information storage of data processing tasks of multiple types of data sources; The task execution module executes the corresponding data processing configuration plan according to the data type of the imported data source, completes the data processing tasks of multiple types of data sources, saves the processed data output, and provides it to the client system.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of a customized processing method based on multi-source real-time aerospace measurement and control data as claimed in any one of claims 1 to 4 are implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of a customized processing method based on multi-source real-time aerospace measurement and control data as claimed in any one of claims 1 to 4 are implemented.

Images