A general-purpose massive airborne data management method based on metadata separation

Abstract

The present application relates to the technical field of mass data management, and particularly relates to a general mass airborne data management method based on metadata separation, which separates and organically stores and manages metadata and data files, logically realizes independent and autonomous management characteristics of data and metadata, can avoid direct contact with data sets, and guarantees the security of sensitive data; uniformly and standardizes data management of respective independent system sensing devices, avoids data islands; combines a time sequence database to manage metadata information, decouples direct management of data files, takes the metadata model as a benchmark to achieve efficient hierarchical organization and management of data files, improves the service efficiency of business-driven airborne data management, and further explores the potential value of deep analysis and application development of airborne data.

Description

Technical Field

[0001] The present invention relates to the technical field of massive data management, and in particular to a universal massive airborne data management method based on metadata separation. Background Art

[0002] Data management technologies guided by the Internet of Things and big data are leading a new round of industrial transformation. In particular, the aviation industry has an inherent advantage in data. It organizes and manages the massive data sets generated by airborne equipment, deeply explores the massive characteristics of complex data itself, and improves the safety, reliability and high intelligence of airborne sensor equipment.

[0003] Currently, data from aviation onboard equipment is scattered across different locations and sensor systems, leading to difficulties in centralized data acquisition, limited targeted service delivery, and inefficient data utilization. Furthermore, most information resources are unable to process and analyze massive amounts of data. Given the constraints of highly complex and volatile massive onboard data storage costs and data standardization, research into the application and implementation of massive onboard data management technologies is particularly important.

[0004] Furthermore, the independence and fragmented nature of databases within airborne equipment information systems, coupled with decentralized data management, has led to numerous data silos. Research and development of efficient management, fusion, and processing technologies for distributed data storage are currently a core research trend in massive airborne data management. Summary of the Invention

[0005] In view of this, the present invention provides a universal massive airborne data management method based on metadata separation, which is used to solve the problems of massive airborne data dispersion and lack of efficient and unified data organization and management.

[0006] In order to achieve the above technical objectives, the specific technical solutions adopted by the present invention are:

[0007] A general method for managing massive airborne data based on metadata separation includes the following steps:

[0008] S1. Perform standardized processing on massive amounts of airborne data according to a pre-set rule base;

[0009] S2. Build a metadata model corresponding to business information and organize metadata information for the standardized real-time data stream;

[0010] S3. Obtain data stream and metadata information based on the organized metadata information, and construct corresponding index directories and data files in a standard format;

[0011] S4. Store the data file in the corresponding index directory disk and write the metadata information into the time series database in real time.

[0012] Furthermore, the specific processing rules for standardization are as follows:

[0013] Verification method, filtering conditions, protocol standards, cleaning method and constraint scope.

[0014] Furthermore, the standardization processing parses the data in the form of semantic understanding and field meaning of airborne data; the verification method ensures the accuracy and consistency of the data by performing static verification and repeatability verification on the airborne data; the filtering conditions filter and screen the data according to certain conditions; the protocol standard standardizes the data according to the data protocol specification in the airborne field; the cleaning method mainly processes dirty data, illegal data and invalid data in the airborne sensor data; the constraint range limits the data selection in the area by a critical value.

[0015] Furthermore, the S2 specifically includes:

[0016] Based on the time series database, initialize the metadata table structure corresponding to the business information;

[0017] According to the real-time update of the data stream, the corresponding metadata information is extracted.

[0018] Furthermore, the metadata information includes: business scenario, business type, sensor device number, data type, data file encoding, data structure description, data file attributes, data file length, data start time, data end time, file storage index address, data file name and file flag information.

[0019] Furthermore, the S3 specifically includes:

[0020] Obtain data streams and corresponding metadata information in real time and build corresponding index directories;

[0021] Cache data streams to memory in real time and write them to index data files in a standard format periodically or quantitatively.

[0022] Furthermore, the structure of the index directory is: a hierarchical structure corresponding to a dynamic directory rule is pre-set.

[0023] Furthermore, the specific format of the data file in the standard format includes: business type, data start time, data end time and auto-increment sequence.

[0024] Furthermore, S4 specifically includes:

[0025] Real-time caching of data streams, periodically or quantitatively writing data files in standard formats to disk;

[0026] Extract metadata information for real-time caching and perform persistent write database operations according to the write-ahead log mechanism.

[0027] Furthermore, the persistent write database operation according to the write-ahead log mechanism is as follows:

[0028] The metadata information is pre-written to store the generated index file in the corresponding index directory, and at the same time, the corresponding metadata information is written to the pre-write log; when the data file is written successfully, the metadata information of the pre-write log content is triggered to be persisted to the time series database, and the pre-write log is updated; if the pre-write log fails to be written, the corresponding pre-write log is retained and written regularly until it succeeds.

[0029] The beneficial effects of the present invention are:

[0030] The method of this application organizes, stores and manages metadata and data files separately, logically realizing the independent and autonomous management characteristics of data and metadata, avoiding direct contact with data sets and ensuring the security of sensitive data; performing unified and standardized data management for each independent system sensor device to avoid data silos; combining time series databases to manage metadata information, decoupling direct management of data files, and achieving efficient hierarchical organization and management of data files based on metadata models, thereby improving the efficiency of business-driven airborne data management services and exploring the potential value of in-depth analysis and application development of airborne data. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0032] Figure 1 Schematic diagram of the flow of a general-purpose method for managing massive airborne data based on metadata separation according to the present invention;

[0033] Figure 2 This is a structural diagram of the general-purpose massive airborne data management based on metadata separation of the present invention. DETAILED DESCRIPTION

[0034] The embodiments of the present disclosure are described in detail below with reference to the accompanying drawings.

[0035] The following describes the embodiments of the present disclosure through specific examples, and those skilled in the art can easily understand other advantages and effects of the present disclosure from the contents disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, rather than all of the embodiments. The present disclosure can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present disclosure. It should be noted that, in the absence of conflict, the following embodiments and features in the embodiments can be combined with each other. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present disclosure.

[0036] It should be noted that various aspects of the embodiments within the scope of the appended claims are described below. It should be apparent that the aspects described herein can be embodied in a wide variety of forms, and any specific structure and / or function described herein is merely illustrative. Based on this disclosure, it should be understood by those skilled in the art that an aspect described herein can be implemented independently of any other aspect, and two or more of these aspects can be combined in various ways. For example, any number of aspects described herein can be used to implement an apparatus and / or practice a method. In addition, other structures and / or functionalities other than one or more of the aspects described herein can be used to implement this apparatus and / or practice this method.

[0037] It should also be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present disclosure. The illustrations only show components related to the present disclosure and are not drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component can be changed at will, and the component layout type may also be more complicated.

[0038] Additionally, in the following description, specific details are provided to provide a thorough understanding of the examples. However, one skilled in the art will appreciate that the aspects described can be practiced without these specific details.

[0039] In one embodiment of the present invention, a general method for managing massive airborne data based on metadata separation is proposed, comprising the following steps:

[0040] S1. Perform standardized processing on massive amounts of airborne data according to a pre-set rule base;

[0041] S2. Build a metadata model corresponding to business information and organize metadata information for the standardized real-time data stream;

[0042] S3. Obtain data stream and metadata information based on the organized metadata information, and construct corresponding index directories and data files in a standard format;

[0043] S4. Store the data file in the corresponding index directory disk and write the metadata information into the time series database in real time.

[0044] In this embodiment, the processing rules of the standardization process are specifically as follows:

[0045] Verification method, filtering conditions, protocol standards, cleaning method and constraint scope.

[0046] In this embodiment, the standardization processing parses the data in the form of semantic understanding and field meaning of the airborne data; the verification method ensures the accuracy and consistency of the data by performing static verification and repeatability verification on the airborne data; the filtering conditions filter and screen the data according to certain conditions; the protocol standard standardizes the data according to the data protocol specification in the airborne field; the cleaning method mainly processes dirty data, illegal data and invalid data in the airborne sensor data; the constraint range limits the data selection in the area by a critical value.

[0047] In this embodiment, S2 specifically includes:

[0048] Based on the time series database, initialize the metadata table structure corresponding to the business information;

[0049] According to the real-time update of the data stream, the corresponding metadata information is extracted.

[0050] In this embodiment, the metadata information includes: business scenario, business type, sensor device number, data type, data file encoding, data structure description, data file attributes, data file length, data start time, data end time, file storage index address, data file name and file flag information.

[0051] In this embodiment, S3 specifically includes:

[0052] Obtain data streams and corresponding metadata information in real time and build corresponding index directories;

[0053] Cache data streams to memory in real time and write them to index data files in a standard format periodically or quantitatively.

[0054] In this embodiment, the structure of the index directory is: a hierarchical structure corresponding to a pre-set dynamic directory rule.

[0055] In this embodiment, the specific format of the data file in the standard format includes: service type, data start time, data end time and auto-increment sequence.

[0056] In this embodiment, S4 specifically includes:

[0057] Real-time caching of data streams, periodically or quantitatively writing data files in standard formats to disk;

[0058] Extract metadata information for real-time caching and perform persistent write database operations according to the write-ahead log mechanism.

[0059] In this embodiment, the persistent write operation to the database according to the write-ahead log mechanism is specifically as follows:

[0060] The metadata information is pre-written to store the generated index file in the corresponding index directory, and at the same time, the corresponding metadata information is written to the pre-write log; when the data file is written successfully, the metadata information of the pre-write log content is triggered to be persisted to the time series database, and the pre-write log is updated; if the pre-write log fails to be written, the corresponding pre-write log is retained and written regularly until it succeeds.

[0061] Further explanation

[0062] As attached Figure 1 As shown, this embodiment provides a general method for managing massive airborne data based on metadata separation, which specifically includes the following steps:

[0063] Step S1: Perform unified and standardized processing on massive amounts of airborne data according to a pre-set rule base;

[0064] Specifically, this embodiment pre-processes the data through a preset rule base, integrates the fields and semantic analysis, CRC verification method, non-compliant data filtering and critical value constraint method designed by the rule base to perform data pre-processing of each sensor-end airborne device in the early stage, and outputs the airborne data in a unified and standardized manner according to unified data standard rules.

[0065] Step S2: Build a metadata model corresponding to the business information and organize metadata information for the real-time data stream;

[0066] Specifically, we design a specific metadata table structure based on the time series database and initialize the metadata model, as shown in Table 1. We then extract the corresponding metadata information by describing the standard output data stream.

[0067] Table 1 Metadata table structure (METADATA)

[0068]

[0069] Furthermore, the airborne data is hierarchically structured through the rule base, uniformly standardized according to different forms, and cached and updated to the memory in real time;

[0070] Step S3: Obtain data stream and metadata information, and construct corresponding index directory and data files in standard format;

[0071] Specifically, data streams and corresponding metadata information are acquired in real time, and corresponding index directories are constructed. Based on specific business types, dates, and device types, a hierarchical directory structure is set using dynamic directory rules.

[0072] Specifically, the data stream needs to be cached in real time to memory, and index data files in a standard format need to be written regularly or quantitatively. The specific standard format files need to be constrained by specific rules, including: business type, data start time, data end time, and auto-increment sequence;

[0073] Step S4: Store the data file in the corresponding index directory disk and write the metadata information into the time series database in real time;

[0074] Specifically, the data stream and corresponding metadata information are decomposed through the rule base, the data stream is batch cached, and persisted to the pre-set index directory disk according to a quantitative or periodic basis;

[0075] Specifically, as attached Figure 2 As shown, metadata information is extracted for real-time caching, and concurrent database write operations are performed according to the write-ahead log mechanism. The metadata information is persisted in the time series database using SQL format.

[0076] Specifically, metadata needs to be pre-written into the log. First, the distributed data packet structure is disassembled according to the rules, and the extracted data stream information is written into a data file in a specified standard format by building an index directory. At the same time, the corresponding metadata information extracted is pre-written into the log, and a complete SQL write statement is organized to trigger the time series database table write operation. If the write is successful, the pre-write log metadata information can be cleared. If the write fails, the pre-write log metadata information is retained and waits for the next cycle to trigger the write operation until the write is successful. It can be cleared to ensure the integrity and consistency of the metadata information and the corresponding index file information.

[0077] The above description is merely a specific embodiment of the present disclosure, but the scope of protection of the present disclosure is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this disclosure should be included in the scope of protection of the present disclosure. Therefore, the scope of protection of the present disclosure should be based on the scope of protection of the claims.

Claims

1. A general-purpose method for managing massive airborne data based on metadata separation, characterized in that: The following steps are involved: S1. Perform standardized processing on massive amounts of airborne data according to a pre-set rule base; S2. Build a metadata model corresponding to business information and organize metadata information for the standardized real-time data stream; S3. Acquire data streams and metadata information based on the organized metadata information, and construct corresponding index directories and data files in a standard format, including: acquiring data streams and corresponding metadata information in real time, and constructing corresponding index directories; caching data streams in real time into memory, and periodically or quantitatively writing index data files in a standard format; S4. Storing data files in the corresponding index directory disk and writing metadata information to the time series database in real time. This includes: real-time caching of data streams, periodic or quantitative writing of standard format data files to disk; extracting metadata information for real-time caching, and performing persistent write operations to the database using the write-ahead log mechanism. The specific operations for persistent database writes according to the write-ahead log mechanism are as follows: The metadata information is pre-written to store the generated index file in the corresponding index directory, and at the same time, the corresponding metadata information is written to the pre-write log; when the data file is written successfully, the metadata information of the pre-write log content is triggered to be persisted to the time series database, and the pre-write log is updated; if the pre-write log fails to be written, the corresponding pre-write log is retained and written regularly until it succeeds.

2. The general-purpose method for managing massive airborne data based on metadata separation according to claim 1, characterized in that: The specific processing rules for standardized processing are: Verification method, filtering conditions, protocol standards, cleaning method and constraint scope.

3. The general-purpose method for managing massive airborne data based on metadata separation according to claim 2, characterized in that: The standardization processing parses the data in the form of semantic understanding and field meaning of airborne data; the verification method ensures the accuracy and consistency of the data by performing static verification and repeatability verification on the airborne data; the filtering conditions filter and screen the data according to certain conditions; the protocol standard standardizes the data according to the data protocol specification in the airborne field; the cleaning method mainly processes dirty data, illegal data and invalid data in the airborne sensor data; the constraint range limits the data selection in the area through the critical value.

4. The general-purpose method for managing massive airborne data based on metadata separation according to claim 3, characterized in that: The S2 specifically includes: Based on the time series database, initialize the metadata table structure corresponding to the business information; According to the real-time update of the data stream, the corresponding metadata information is extracted.

5. The general-purpose method for managing massive airborne data based on metadata separation according to claim 4, characterized in that: The metadata information includes: business scenario, business type, sensor device number, data type, data file encoding, data structure description, data file attributes, data file length, data start time, data end time, file storage index address, data file name and file flag information.

6. The general-purpose method for managing massive airborne data based on metadata separation according to claim 1, characterized in that: The structure of the index directory is: a corresponding hierarchical structure is pre-set according to dynamic directory rules.

7. The general-purpose method for managing massive airborne data based on metadata separation according to claim 1, characterized in that: The specific format of the data file in the standard format includes: business type, data start time, data end time and auto-increment sequence.

Images