Distributed warehousing method, system and equipment for civil aircraft test flight data and storage medium

Abstract

The invention relates to a distributed warehousing method, system and device for civil aircraft test flight data and a storage medium. The method comprises the following steps of S1, uploading an original file; s2, standardization and segmentation are carried out; performing standardization processing and segmentation processing on the original file to obtain a plurality of fragment files; s3, conversion and collection are carried out; the cluster of the Hadoop platform generates a plurality of sub-tasks according to the names of the plurality of fragment files and the position information of the storage path, and dynamically distributes each sub-task to each distributed computing node of the cluster for processing, and all generated TsFile time sequence files are collected in a file directory of the Hadoop platform; and S4, loading and warehousing. Through distributed processing, high-availability design, concurrent expansion and write-in capability optimization, the import efficiency, reliability and expandability of the civil aircraft test flight data large file are remarkably improved, and therefore the user experience is improved.

Description

Technical Field

[0001] The present invention relates to the field of aircraft data processing, and in particular to a distributed storage method, system, computer equipment and storage medium for civil aircraft flight test data. Background Art

[0002] The test data of civil aircraft flight test process is usually presented as time series data, and its sampling frequency varies widely. For example, the sampling frequency of a slowly changing temperature sensor may be only 1Hz, while the sampling frequency of some power quality parameters may be as high as 2MHz. In the performance stability test project, the common data sampling frequency is between 8Hz and 512Hz. If it is assumed that each data field occupies 8 bytes, according to the normal sampling frequency of 64Hz, 4 hours of test flight time and 400 working condition data fields, the capacity of a single raw data file generated by each test flight is about 2.75GB (64×4×3600×400×8 bytes). Similarly, for the performance stability professional vibration and jitter test subject at a sampling frequency of 512Hz, the capacity of a single raw data file for each test flight can reach 22GB (512×4×3600×400×8 bytes).

[0003] At present, the file upload and storage solutions mainly rely on a monolithic architecture, which has an inherent single point failure risk and cannot achieve high availability. The reliability of file import tasks is insufficient and is only suitable for application scenarios with low complexity and small scale. Although the performance of the monolithic architecture can be improved through multi-threaded optimization, it is still limited by the number of cores of a single physical machine. For example, this multi-threaded technology stipulates that the number of sub-threads used must be less than the number of cores of the physical machine, which limits the expansion of the number of concurrency and leads to limited import performance in large-scale scenarios.

[0004] In the actual operation of storing offline flight test data files, for large files larger than 2GB, the existing single-machine centralized import method will lead to performance degradation. Users need to wait for a long time when querying, about 210 seconds, which affects the efficiency of flight test data processing and analysis. During the process of uploading files to the library, the write capacity, concurrent write number and transmission bandwidth of the database node are all limited, resulting in a poor user experience. Summary of the invention

[0005] In view of the above-mentioned defects of the prior art, a first aspect of the present invention provides a distributed storage method for civil aircraft flight test data, which includes the following steps:

[0006] Step S1: Upload the original file; upload the original file containing the civil aircraft flight test data to the server;

[0007] Step S2: Standardization and segmentation: Standardization and segmentation are performed on the original file to obtain multiple segment files;

[0008] Step S3: conversion and collection; the Hadoop platform cluster generates multiple subtasks corresponding to each of the shard files according to the names of the multiple shard files and the location information of the storage paths, and dynamically assigns each of the subtasks to each distributed computing node of the cluster for processing, each of the subtasks reads, maps, and converts the data in each of the shard files, and generates TsFile time series files respectively, and all the generated TsFile time series files are collected in the file directory of the Hadoop platform;

[0009] Step S4: loading into the database; loading each of the TsFile time series files into the time series database IoTDB.

[0010] In the above-mentioned distributed storage method for civil aircraft flight test data, optionally, step S2 includes the following steps:

[0011] Step S2.1: Determine whether the original file contains a table header and a first column time; if so, execute step S2.2; if not, no further processing is performed and the execution ends;

[0012] Step S2.2: performing the standardization process on the original file to unify the data format and obtain a standard file;

[0013] Step S2.3: performing the segmentation process on the standard file according to the configuration parameters to obtain a plurality of segment files;

[0014] Step S2.4: Block caching, storing the segmented files in Hadoop platform blocks;

[0015] Step S2.5: Submit the name of the shard file and the location information of the storage path to the task interface of the cluster, so that the cluster can schedule the distributed computing nodes and execute each subtask in the subsequent step S3;

[0016] Step S2.6: The timing program monitors the status information of the shard file processing the subtask at the task interface end of the cluster and records it into the task list of the relational database MySQL.

[0017] In the distributed warehousing method for civil aircraft flight test data as described above, optionally, the standardization processing in step S2.2 includes the following steps:

[0018] Remove the header description line of the original file;

[0019] Converting the special characters in the header of the original file to form a file with a unified header format;

[0020] The timestamp field of the time series data in the original file is standardized to form a file with a consistent timestamp format.

[0021] In the distributed storage method for civil aircraft flight test data as described above, optionally, the segmentation process in step S2.3 includes the following steps:

[0022] Identify line delimiters and field delimiters of the standard file;

[0023] The field value is disassembled according to the line separator and the field separator and the standard file is divided into a plurality of the shard files; the configuration parameters are determined according to the size of the Hadoop platform block and the data volume limit that can be stored in each TsFile time series file; the configuration parameters include the size of the shard file, the sharding method, the target IoTDB unique identifier and the naming rule; the naming rule stipulates that the name of the shard file includes the original file name, the sharding sequence number and the target IoTDB unique identifier;

[0024] Each of the slice files is given a name according to the naming rule.

[0025] In the above-mentioned distributed storage method for civil aircraft flight test data, optionally, step S3 includes the following steps:

[0026] Step S3.1: registering metadata for the segmented file; the metadata includes the structure, attributes and source information of the time series data in the segmented file;

[0027] Step S3.2: dynamically assigning the subtasks to different distributed computing nodes according to the current load of each distributed computing node to read, map and convert the records of the shard file line by line, wherein each record corresponds to a data row in the shard file; each subtask includes a Map task and a Reduce task, the Map task is used to read, map and convert the shard file, and the Reduce task is used to collect the TsFile time series files generated by the Map task into the file directory of the Hadoop platform;

[0028] Each of the distributed computing nodes calls the built-in Map function of the Hadoop platform to read the records of the shard file line by line, map and convert them into the values ​​of the TsFile time series file, write the processed records into the time series record TSRecord and write the time series record TSRecord into Tablet, and judge whether the Tablet capacity exceeds the limit; if the upper limit of the Tablet capacity has been reached, the Tablet is refreshed to generate the TsFile time series file; otherwise, the next line of records is read and mapped and converted;

[0029] Whether the Map task of the shard file is completed is determined by whether the next line of records to be read is empty; if the next line of records to be read is empty, the subtask is completed, and the TsFile time series file with a complete time series record TSRecord is generated, and step S3.3 is executed; otherwise, the next line of records is read and mapped and converted until the next line of records to be read is empty;

[0030] Step S3.3: calling the Reduce function built into the Hadoop platform to collect the TsFile time series files generated by all the shard files with the same file name into the same file directory.

[0031] In the above-mentioned distributed storage method for civil aircraft flight test data, optionally, the Map task and the Reduce task of the same shard file occupy the same memory and CPU resources of the distributed computing node; in the process of dynamically allocating the subtasks on the Hadoop platform, the following steps are included:

[0032] Load calculation: Calculate the memory and CPU resources required for the Map task or the Reduce task corresponding to each of the shard files;

[0033] Resource usage statistics: calculating resource usage according to the number of subtasks being processed and to be run by each distributed computing node;

[0034] Distributed computing node load sorting; sorting the resource usage, the load of the distributed computing node is sorted in proportion to the resource usage, and the Hadoop platform preferentially assigns the newly generated subtask to the distributed computing node with smaller resource usage than other distributed computing nodes for processing.

[0035] In the distributed warehousing method for civil aircraft flight test data as described above, optionally, in step S4, a loading service for the TsFile time sequence file is constructed through the Springboot framework, the target IoTDB unique identifier in the configuration parameters of each TsFile time sequence file is identified and automatically loaded into the target IoTDB; step S4 includes the following steps:

[0036] Step S4.1: Obtain the file directory of the Hadoop platform to determine the directory addresses of all the TsFile time series files that need to be loaded into the target IoTDB;

[0037] Step S4.2: Parse the path of the TsFile time sequence file in the file directory to obtain the target IoTDB unique identifier;

[0038] Step S4.3: Obtain the configuration items of the target IoTDB according to the unique identifier of the target IoTDB, ensure that the TsFile time series file is distributed to the node of the target IoTDB, and load the TsFile time series file into the time series database IoTDB;

[0039] Step S4.4: Determine whether all the TsFile timing files have been loaded; if not, continue loading until all the TsFile timing files are stored; if completed, execute step S4.5;

[0040] Step S4.5: Refresh the data in the time series database IoTDB to ensure that the time series database IoTDB reflects the latest data status in a timely manner.

[0041] In order to achieve the above-mentioned object, the second aspect of the present invention provides a distributed storage system for civil aircraft flight test data, which includes:

[0042] A cloud database, wherein the cloud database includes a relational database MySQL and a time series database IoTDB;

[0043] A distributed cluster includes multiple servers, each of which works together to share the load; each of the servers includes:

[0044] An import engine module, wherein the import engine module is used to perform standardization processing and segmentation processing on the original file containing the civil aircraft flight test data to obtain multiple segment files;

[0045] A Hadoop platform module, wherein the Hadoop platform module converts the plurality of shard files into a plurality of TsFile time-series files and collects the files into a directory;

[0046] A loading engine module is used to load each of the TsFile time series files into the time series database IoTDB.

[0047] In order to achieve the above-mentioned purpose, the third aspect of the present invention provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor runs the program, the steps of the distributed warehousing method for civil aircraft flight test data as described in any one of the first aspects above are implemented.

[0048] In order to achieve the above-mentioned purpose, the fourth aspect of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions or computer programs, and when the computer-executable instructions or computer programs are processed and executed, a distributed warehousing method for civil aircraft flight test data as described in any one of the first aspects above is implemented.

[0049] The distributed storage method for civil aircraft flight test data provided by the present invention significantly improves the efficiency of importing flight test data files into a database by processing the original files in pieces and using distributed computing nodes for parallel processing. Relying on the scalability of the Hadoop platform, the method can effectively process data in parallel, and each data stage is independent of each other, thereby solving the single point failure problem existing in the traditional centralized file storage service. In addition, the method can dynamically allocate file import tasks according to the load conditions of the computing nodes, ensuring efficient use of resources.

[0050] By optimizing the data import process and improving processing efficiency, the present invention significantly reduces the waiting time of users when querying, thereby improving the user experience. By adopting a distributed processing method, the complex file conversion work is handed over to the Hadoop platform for distributed task execution, further improving the writing ability of the professional time series database IoTDB node.

[0051] In summary, the present invention significantly improves the import efficiency, reliability and scalability of large files of civil aircraft flight test data through distributed processing, high availability design, concurrent expansion and optimized writing capabilities, thereby improving the user experience.

[0052] The concept, specific structure and technical effects of the present invention will be further described below in conjunction with the accompanying drawings to fully understand the purpose, characteristics and effects of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 It is a flow chart of an embodiment of a distributed storage method for civil aircraft flight test data of the present invention;

[0054] Figure 2 yes Figure 1Detailed flow chart of step S2;

[0055] Figure 3 yes Figure 1 Detailed flow chart of step S3;

[0056] Figure 4 yes Figure 1 Detailed flow chart of step S4;

[0057] Figure 5 It is a structural schematic diagram of an embodiment of a distributed storage system for civil aircraft flight test data of the present invention. DETAILED DESCRIPTION

[0058] In order to make the technical means, creative features, objectives and effects of the invention easier to understand, the invention is further described below with reference to specific diagrams. However, the invention is not limited to the following implementation cases.

[0059] It should be noted that the structures, proportions, sizes, etc. illustrated in the drawings in this specification are only used to match the contents disclosed in the specification so as to facilitate understanding and reading by persons familiar with this technology. They are not used to limit the conditions under which the present invention can be implemented, and therefore have no substantive technical significance. Any structural modification, change in proportion or adjustment of size, without affecting the effects and purposes that can be achieved by the present invention, should still fall within the scope of the technical contents disclosed by the present invention.

[0060] Terms such as “comprise” and “include” indicate that in addition to the components directly and explicitly stated in the specification and claims, the technical solution of the present invention does not exclude the situation where it has other components that are not directly or explicitly stated.

[0061] The Hadoop platform is an open source distributed computing platform, which is used in the present invention to process and store large-scale data sets, namely civil aircraft test flight data. The Hadoop platform has two core components built in: the Hadoop Distributed File System (HDFS) and the MapReduce computing framework.

[0062] like Figures 1 to 5 As shown, the method may specifically include the following steps:

[0063] Step S1: Upload the original file from the local client to the server.

[0064] In step S1, the original file includes a large amount of time series data during the civil aircraft test flight.

[0065] Step S2: Standardization and segmentation.

[0066] In step S2, the original file is subjected to standardization and segmentation processing by the import engine, and the large file, i.e., the original file, is segmented into a plurality of small file segments to obtain a plurality of segmented files (i.e., small file segments). In an optional embodiment, the large file and the small file are defined according to the volume of the file, and the large file generally refers to a file with a file size between 2.75 GB and 22 GB, while the small file refers to a file with a relatively small file size relative to the original file, which is a large data capacity file. In this embodiment, the large file refers to the original file with a file size of more than 2 GB, and the small file refers to the segmented file after the original file is segmented.

[0067] like Figure 2 and Figure 5 As shown, in an optional embodiment, step S2 may specifically include the following steps:

[0068] Step S2.1: Determine whether the original file contains the table header and the first column time; if so, it means that the original file can be standardized, and then execute step S2.2; if not, no standardization is required, and no subsequent processing is performed.

[0069] It should be noted here that the original file containing the header and the first column of time can be more easily parsed and converted. The header information can help identify the meaning of each field, so as to correctly map the time series data to the time series database IoTDB, and the first column of time information can help determine the time series characteristics of the time series data, which is very important for the storage and query of the time series database IoTDB. Some original files may not contain the header and the first column of time information due to specific application scenarios or different sensor device data generation methods. These files may directly contain data records without additional necessary metadata information. The present invention does not perform subsequent processing on these data.

[0070] Step S2.2: Standardize the original files to obtain standard files. This step standardizes different types of original flight test data files into the same data format, removes irrelevant information, simplifies the subsequent data processing process, improves the overall data processing efficiency, and provides convenience for subsequent distributed processing and warehousing operations.

[0071] In this embodiment, the standardization process may specifically include the following steps:

[0072] Remove the header description line of the original file; the header description line includes descriptive information about the original file content and has no practical significance for data import.

[0073] Convert special characters in the header of the original file; ensure the header format is unified to facilitate subsequent data processing.

[0074] Standardize the timestamp field of the time series data in the original file; ensure that the timestamp format is consistent, so that users can perform time series analysis in the IoTDB time series database later.

[0075] Step S2.3: Slice the standard file obtained after standardization according to the configuration parameters to obtain multiple fragment files.

[0076] In this embodiment, the segmentation process may specifically include the following steps:

[0077] Identify the line separators and field separators of standard files; determine the separators between each line of data and each field in the file to reasonably parse the data and split the standard file.

[0078] The standard file is split into multiple fragment files according to the configuration parameters.

[0079] The configuration parameters are determined based on the size of the Hadoop platform block and the data volume limit that each TsFile time series file can store. Specifically, the configuration parameters may include the size of the shard file, the segmentation method, the target IoTDB unique identifier, and the naming rules. The naming rules specify that the name of the shard file must include the original file name, the segmentation sequence number, and the target IoTDB unique identifier. The name of each shard file is given according to the naming rules.

[0080] It should also be noted that the Hadoop platform block refers to the basic unit for storing files in the above-mentioned Hadoop distributed file system. In the distributed file system, standard files are divided into multiple blocks for storage, and the size of each block defaults to 128MB. In an optional embodiment, the size of each block can be configured according to the specific needs of actual work. The size of the block determines the storage method of the shard file in the distributed file system, wherein each block can be stored on different DataNode data nodes in the Hadoop platform to achieve distributed storage of data. Therefore, in order to make full use of the distributed processing capabilities of the Hadoop platform when slicing files, the standard file is divided into shards of a size close to the Hadoop platform block to ensure that each shard file can be independently assigned to different distributed computing nodes for subsequent processing.

[0081] The destination IoTDB unique identifier is a specific identifier used to uniquely identify the destination IoTDB. In a distributed system, there may be multiple time series database IoTDB instances, each of which is responsible for storing different data. This identifier is unique in the distributed system and is used to ensure that the data can be correctly imported into the specified time series database IoTDB to avoid data import errors or confusion. In addition, the destination IoTDB unique identifier can also be used to query the configuration service and obtain the configuration items of the destination IoTDB, thereby achieving the correct loading and storage of time series data.

[0082] Step S2.4: Block caching, caching each shard file into a Hadoop platform block so that it can be subsequently submitted to the Hadoop platform cluster for data processing.

[0083] Step S2.5: Submit the name of the shard file and the location information of the storage path to the task interface end (Client end) of the cluster for subsequent node scheduling and execution of each subtask in the cluster. Even if a computing node fails, other nodes can still continue processing, reducing the risk of single point failure of the overall distributed system.

[0084] Step S2.6: The timer program monitors the status information of the subtasks processed by the shard file at the task interface end of the cluster and records it in the task list of the relational database MySQL. This information can help engineers monitor and manage the execution of tasks processed by each distributed computing node, and promptly discover and handle problems. At the same time, the relational database MySQL supports complex query and analysis functions, which makes it easy for engineers to query and analyze the task processing status of each distributed node, count the success rate of tasks or analyze the reasons for task failure, etc., to provide data support for optimizing and improving the system.

[0085] Step S3: conversion and aggregation.

[0086] In step S3, the cluster of the Hadoop platform will generate multiple subtasks, namely, storage tasks, corresponding to multiple shard files. Each subtask corresponds to mapping and converting the data of a shard file. These subtasks will be dynamically assigned to multiple different distributed computing nodes in the cluster to convert each shard file and generate TsFile time series files corresponding to each shard file. TsFile time series files are readable and writable files of the professional time series database IoTDB. All generated TsFile time series files are collected in the distributed file directory of the Hadoop platform to facilitate subsequent reading and loading into the time series database IoTDB by the loading engine.

[0087] The Hadoop platform cluster consists of multiple server nodes, namely distributed computing nodes, which can access each other. Each node can run the components of the Hadoop platform (such as NameNode, DataNode, MapReduce), work together to complete the distributed processing and storage of time series data, share the load, and improve data processing capabilities and reliability.

[0088] like Figure 3 and Figure 5 As shown, in an optional embodiment, step S3 may specifically include the following steps:

[0089] Step S3.1: Configure the environment variables of the subtasks to ensure that each subtask can access the necessary resources and perform its functions correctly, and register the metadata of the shard files.

[0090] Metadata registration is achieved by recording the metadata information of each shard file in the Hadoop platform, which helps to manage and track the status of the shard file in the cluster of the Hadoop platform to store and process data. Metadata is data about data, which is used to describe, explain, locate and manage data resources. In this embodiment, metadata includes the structure, attributes and source information of the time series data in the shard file, provides information such as file name, file size, storage location, creation time, modification time, etc., and plays a role in locating data because it records the specific location of data storage, such as file path, database table name, data block ID, etc. The metadata information of each data block is recorded in the naming node NameNode, and the data block is stored on the data node DataNode, and the data block information stored is reported to the naming node regularly. In addition, metadata can provide a query path. When a user queries a file by file name or path, the naming node returns the metadata information of the file. The user can locate the specific storage location of the data and read the data from the data node DataNode based on the metadata information.

[0091] Step S3.2: Assign subtasks to different distributed computing nodes according to the attributes of the metadata to perform line-by-line record reading, mapping, and conversion processing on the shard files.

[0092] In a shard file (time series data file), a record refers to a complete data point or a group of related data points. Each record contains one or more fields, which together describe a specific data event or state, and each field in the record represents a specific data attribute, such as timestamp, temperature, humidity, etc. When reading the file content line by line, each line read is a record, and the read records are disassembled according to the field separator, and the disassembled field values ​​are mapped to the corresponding data types and converted into TsFile time series file output.

[0093] In an optional embodiment, the process of reading, mapping and converting the shard file is completed in the Hadoop platform, each record corresponds to a data row in the shard file, each subtask includes a Map task and a Reduce task, and each distributed computing node may specifically include the following steps when processing the subtask:

[0094] Start the Map task to read, map and convert the shard file. Each distributed computing node calls the built-in Map function of the Hadoop platform to read and map the records of the shard file line by line, and convert them into the values ​​of the TsFile time series file. The records of this processing process are written into the time series record TSRecord, and the time series record TSRecord is written into the Tablet to determine whether the capacity of the Tablet exceeds the limit; if the capacity limit of the Tablet has been reached, the Tablet is flushed to directly generate the TsFile time series file; otherwise, the next row of records is read and mapped and converted. After each reading, mapping and conversion of records, each distributed computing node determines whether the subtask of the shard file has been processed by whether the next row of records to be read is empty; if the next row of records to be read is empty, the subtask is processed, and the complete TsFile time series file of the time series record TSRecord is generated and the next step is executed; otherwise, the next row of records is read to complete the mapping and conversion until the next row of records to be read is empty, which means that the Map task of the shard file has been processed.

[0095] Step S3.3: Call the Reduce function built into the Hadoop platform to collect all TsFile time series files with the same file name into the specified file directory.

[0096] The process of reading, mapping, and converting shard files is a key step in the distributed processing framework to process time series data files. It reads the file content line by line, identifies line and field delimiters, disassembles and converts field values, starts the Map function to process records, determines the Tablet capacity, writes to the Tablet, flushes the Tablet, generates TsFile time series files, and starts the Reduce function to process all TsFile time series files containing TSRecord generated by the Map function. The TsFile time series files generated by all shard files with the same file name are collected in the specified directory. This process ensures that the data can be correctly parsed and processed.

[0097] In addition, the Hadoop platform of the present invention can dynamically allocate subtasks to each distributed computing node according to the current load of each distributed computing node. Each subtask needs to be allocated with Map and Reduce tasks, and each Map / Reduce task occupies the same memory and CPU.

[0098] Optionally, here are the specific steps:

[0099] Load calculation: Calculate the memory and CPU resources required for the Map task or Reduce task corresponding to each shard file;

[0100] Resource usage statistics: Calculate resource usage based on the number of subtasks being processed and to be run on each distributed computing node;

[0101] Node load sorting: Sort resource usage. Node load is proportional to resource usage. New subtasks are preferentially assigned to nodes with small loads, that is, nodes with low resource usage. If all nodes have the same load, any node can be selected for assignment.

[0102] Step S4: Loading into the library. Call the loading engine to route the TsFile time series file into the time series database IoTDB. The routing here refers to the end-to-end path selection. Since the distributed computing nodes of the distributed platform Hadoop and the nodes of the target IoTDB can provide different loading paths, the TsFile time series file of the present invention can realize route loading.

[0103] like Figure 4 and Figure 5 As shown, in an optional embodiment, the loading service of the TsFile timing file is constructed through the Springboot framework, the configuration parameters of each TsFile timing file are identified and automatically loaded into the target IoTDB, and step S4 may specifically include the following steps:

[0104] Step S4.1: Obtain the file directory of the Hadoop platform to determine the directory address of all TsFile time-series files that need to be loaded into the target IoTDB. Specifically, obtain a list of all files in the specified directory through the API interface of the distributed file system of the Hadoop platform, and these file lists contain all TsFile time-series files that need to be loaded into the target IoTDB.

[0105] Step S4.2: Parse the path of the TsFile time series file in the file directory to obtain the target IoTDB unique identifier. Specifically, parse the obtained TsFile time series file path and extract the target IoTDB unique identifier contained therein. This unique identifier is usually reflected in the file path or file name and is used to determine which specific time series database instance the file should be loaded into.

[0106] Step S4.3: Load the TsFile time series file into the time series database IoTDB according to the unique identifier of the target IoTDB. Specifically, according to the unique identifier of the target IoTDB obtained in step S4.2, query the configuration service to obtain the specific configuration information of the target IoTDB, such as the IP address and port number of the database node, and then use the network transmission protocol (such as FTP, SFTP, HTTP, etc.) to distribute the TsFile time series file from the cluster of the Hadoop platform to the node of the target IoTDB. Use the file loading interface provided by the time series database IoTDB on the node of the target IoTDB to load the TsFile time series file into the library.

[0107] Step S4.4: Determine whether all TsFile sequential files have been loaded; if not, continue loading until all TsFile sequential files are stored; if completed, execute step S4.5. Specifically, during the loading process, the loading status of all TsFile sequential files is continuously monitored, and the API interface or log record provided by the database can be used to check whether the file is loaded successfully.

[0108] Step S4.5: Refresh the data in the time series database IoTDB. Specifically, after the TsFile time series file is loaded, the metadata in the time series database IoTDB needs to be refreshed to ensure that the database can reflect the latest data status in a timely manner. The following are the steps to refresh the metadata in the time series database IoTDB:

[0109] Execute refresh operation: Use the refresh operation API provided by IoTDB to refresh the database. The refresh operation will write the data in the memory to the disk to ensure the persistence of the data.

[0110] Update metadata: Update the metadata information of the database, including the newly added data file information, data table structure, etc. This step ensures that the database can correctly identify and manage the newly loaded data.

[0111] Check refresh status: Check whether the refresh operation is completed successfully. You can confirm the status of the refresh operation through the API or log records provided by the database.

[0112] Exception handling: If the refresh operation fails, exception handling is required. The error log can be recorded and the administrator can be notified for manual intervention.

[0113] like Figure 5 As shown, in order to achieve the above purpose, the present invention also provides a distributed storage system for civil aircraft flight test data. The system can specifically include a distributed cluster and a cloud database.

[0114] The distributed cluster includes multiple servers, each server works together as a node of distributed computing to share the load. Each server includes an import engine module, a Hadoop platform module and a loading engine module.

[0115] The cloud database includes the relational database MySQL and the time series database IoTDB.

[0116] In the distributed storage system for civil aircraft flight test data, the connection relationship between the cloud database and the engine module is as follows:

[0117] The import engine module builds services through the Springboot framework and connects to the client task interface of the distributed cluster, namely the Hadoop cluster. The import engine submits the shard file information after the preprocessing and segmentation processing described in the previous embodiment to the Hadoop cluster, which receives the information and distributes it to the distributed computing nodes for processing.

[0118] The Hadoop cluster starts the Map and Reduce tasks and calls its internal conversion engine to perform file conversion processing. The conversion engine reads, maps, and converts the shard files, generates TSRecords and writes them into Tablets, and finally generates TsFile time series files, which are saved in the distributed file system of the Hadoop platform.

[0119] The loading engine obtains these file paths from the distributed file system of the Hadoop platform, parses out the unique identifier of the target IoTDB, queries the configuration service according to the identifier, obtains the configuration items of the target IoTDB, and distributes the TsFile time series file to the target IoTDB node.

[0120] The loading engine builds services through the Springboot framework, connects to the target IoTDB node, distributes the TsFile time series file to the target IoTDB node, and the IoTDB node receives the file and loads it into the library, refreshing the metadata.

[0121] Through the above connection relationship, the cloud database and the distributed cluster engine work together. The specific process is as described in the above embodiment, which will not be repeated here. The efficient distributed storage of flight test data is realized, ensuring the reliability and scalability of the system. According to actual measurements, the existing technology takes about 210s (3-4 minutes) to process and store data files above 2GB. The method and system provided by the present invention divide the files into multiple tasks for parallel processing and import into the database in only 50 seconds.

[0122] In order to achieve the above-mentioned objectives, the present invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor runs the program, the steps of the distributed warehousing method for civil aircraft flight test data as described in any of the aforementioned embodiments can be implemented.

[0123] The processor and the memory can be set separately or integrated together, for example, integrated on a system-on-chip (SOC) of a terminal device. It should be understood that the processor of the embodiment of the present application may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method embodiment can be completed by an integrated logic circuit of hardware in the processor or an instruction in the form of software. The above processor can be a general-purpose processor, a digital signal processor (Digital Signal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field programmable gate array (Field Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in conjunction with the embodiment of the present application can be directly embodied as a hardware decoding processor to be executed, or a combination of hardware and software modules in the decoding processor can be executed. The software module can be located in a storage medium mature in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.

[0124] In order to achieve the above-mentioned purpose, the present invention also provides a computer-readable storage medium, which stores computer-executable instructions or computer programs. When the computer-executable instructions or computer programs are processed and executed, the distributed warehousing method of civil aircraft flight test data as described above is implemented.

[0125] The computer-readable storage medium is, for example, a memory. The memory may be a volatile memory or a nonvolatile memory, or the memory may include both a volatile memory and a nonvolatile memory. Among them, the nonvolatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct rambus RAM (DR RAM).

[0126] If the integrated units in the above embodiments are implemented in the form of software functional units and sold or used as independent products, they can be stored in the above computer-readable storage medium. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling one or more computer devices (which can be personal computers, servers or network devices, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention.

[0127] The preferred specific embodiments of the present invention are described in detail above. It should be understood that ordinary technicians in the field can make many modifications and changes based on the concept of the present invention without creative work. Therefore, all technical solutions that can be obtained by technicians in the technical field based on the concept of the present invention through logical analysis, reasoning or limited experiments on the basis of the prior art should be within the scope of protection determined by the claims.

Claims

1. A distributed storage method for civil aircraft flight test data, characterized in that: The steps include: Step S1: Upload the original file; upload the original file containing the civil aircraft flight test data to the server; Step S2: Standardization and segmentation: Standardization and segmentation are performed on the original file to obtain multiple segment files; Step S3: conversion and collection; the Hadoop platform cluster generates multiple subtasks corresponding to each of the shard files according to the names of the multiple shard files and the location information of the storage paths, and dynamically assigns each of the subtasks to each distributed computing node of the cluster for processing, each of the subtasks reads, maps, and converts the data in each of the shard files, and generates TsFile time series files respectively, and all the generated TsFile time series files are collected in the file directory of the Hadoop platform; Step S4: loading into the database; loading each of the TsFile time series files into the time series database IoTDB.

2. The distributed storage method for civil aircraft flight test data according to claim 1 is characterized in that: The step S2 comprises the following steps: Step S2.1: Determine whether the original file contains a table header and a first column time; if so, execute step S2.2; if not, no further processing is performed and the execution ends; Step S2.2: performing the standardization process on the original file to unify the data format and obtain a standard file; Step S2.3: performing the segmentation process on the standard file according to the configuration parameters to obtain a plurality of segment files; Step S2.4: Block caching, storing the segmented files in Hadoop platform blocks; Step S2.5: Submit the name of the shard file and the location information of the storage path to the task interface of the cluster, so that the cluster can schedule the distributed computing nodes and execute each subtask in the subsequent step S3; Step S2.6: The timing program monitors the status information of the shard file processing the subtask at the task interface end of the cluster and records it into the task list of the relational database MySQL.

3. The distributed storage method for civil aircraft flight test data according to claim 2 is characterized in that: The standardization process in step S2.2 comprises the following steps: Remove the header description line of the original file; Converting the special characters in the header of the original file to form a file with a unified header format; The timestamp field of the time series data in the original file is standardized to form a file with a consistent timestamp format.

4. The distributed storage method for civil aircraft flight test data according to claim 2 is characterized in that: The segmentation process in step S2.3 includes the following steps: Identify line delimiters and field delimiters of the standard file; The field value is disassembled according to the line separator and the field separator and the standard file is divided into a plurality of the shard files; the configuration parameters are determined according to the size of the Hadoop platform block and the data volume limit that can be stored in each TsFile time series file; the configuration parameters include the size of the shard file, the sharding method, the target IoTDB unique identifier and the naming rule; the naming rule stipulates that the name of the shard file includes the original file name, the sharding sequence number and the target IoTDB unique identifier; Each of the slice files is given a name according to the naming rule.

5. The distributed storage method for civil aircraft flight test data according to claim 1 is characterized in that: The step S3 comprises the following steps: Step S3.1: registering metadata for the segmented file; the metadata includes the structure, attributes and source information of the time series data in the segmented file; Step S3.2: dynamically assigning the subtasks to different distributed computing nodes according to the current load of each distributed computing node to read, map and convert the records of the shard file line by line, wherein each record corresponds to a data row in the shard file; each subtask includes a Map task and a Reduce task, the Map task is used to read, map and convert the shard file, and the Reduce task is used to collect the TsFile time series files generated by the Map task into the file directory of the Hadoop platform; Each of the distributed computing nodes calls the built-in Map function of the Hadoop platform to read the records of the shard file line by line, map and convert them into the values ​​of the TsFile time series file, write the processed records into the time series record TSRecord and write the time series record TSRecord into Tablet, and judge whether the Tablet capacity exceeds the limit; if the upper limit of the Tablet capacity has been reached, the Tablet is refreshed to generate the TsFile time series file; otherwise, the next line of records is read and mapped and converted; Whether the Map task of the shard file is completed is determined by whether the next line of records to be read is empty; if the next line of records to be read is empty, the subtask is completed, and the TsFile time series file with a complete time series record TSRecord is generated, and step S3.3 is executed; otherwise, the next line of records is read and mapped and converted until the next line of records to be read is empty; Step S3.3: calling the Reduce function built into the Hadoop platform to collect the TsFile time series files generated by all the shard files with the same file name into the same file directory.

6. The distributed storage method for civil aircraft flight test data according to claim 5 is characterized in that: The Map task and the Reduce task of the same shard file occupy the same memory and CPU resources of the distributed computing node; the process of dynamically allocating the subtasks on the Hadoop platform includes the following steps: Load calculation: Calculate the memory and CPU resources required for the Map task or the Reduce task corresponding to each of the shard files; Resource usage statistics; Calculate resource usage according to the number of subtasks being processed and to be executed by each distributed computing node; Distributed computing node load sorting; sorting the resource usage, the load of the distributed computing node is sorted in proportion to the resource usage, and the Hadoop platform preferentially assigns the newly generated subtask to the distributed computing node with smaller resource usage than other distributed computing nodes for processing.

7. The distributed storage method for civil aircraft flight test data according to claim 1 is characterized in that: In the step S4, the loading service of the TsFile timing file is constructed through the Springboot framework, the unique identifier of the target IoTDB in the configuration parameters of each TsFile timing file is identified and automatically loaded into the target IoTDB; the step S4 includes the following steps: Step S4.1: Obtain the file directory of the Hadoop platform to determine the directory addresses of all the TsFile time series files that need to be loaded into the target IoTDB; Step S4.2: Parse the path of the TsFile time sequence file in the file directory to obtain the target IoTDB unique identifier; Step S4.3: Obtain the configuration items of the target IoTDB according to the unique identifier of the target IoTDB, ensure that the TsFile time series file is distributed to the node of the target IoTDB, and load the TsFile time series file into the time series database IoTDB; Step S4.4: Determine whether all the TsFile timing files have been loaded; if not, continue loading until all the TsFile timing files are stored; if completed, execute step S4.5; Step S4.5: Refresh the data in the time series database IoTDB to ensure that the time series database IoTDB reflects the latest data status in a timely manner.

8. A distributed storage system for civil aircraft flight test data, characterized in that: include: A cloud database, wherein the cloud database includes a relational database MySQL and a time series database IoTDB; A distributed cluster includes multiple servers, each of which works together to share the load; each of the servers includes: An import engine module, wherein the import engine module is used to perform standardization processing and segmentation processing on the original file containing the civil aircraft flight test data to obtain multiple segment files; A Hadoop platform module, wherein the Hadoop platform module converts the plurality of shard files into a plurality of TsFile time-series files and collects the files into a directory; A loading engine module is used to load each of the TsFile time series files into the time series database IoTDB.

9. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor runs the program, the steps of the distributed warehousing method for civil aircraft flight test data as described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions or computer programs, and when the computer-executable instructions or computer programs are processed and executed, a distributed warehousing method for civil aircraft flight test data according to any one of claims 1 to 7 is implemented.

Images