A method and device for processing multi-fleet reliability data based on the ATASPEC2000 standard

Abstract

This application provides a method for processing multi-fleet reliability data based on the ATASPEC2000 specification. The method includes acquiring different types of raw reliability data files from multiple fleets; parsing the raw reliability data files to obtain a raw reliability data table; configuring a data cleaning template for the raw reliability data table using a CSDD data dictionary and performing data cleaning processing to obtain a reliability data cache table; and configuring a data transformation template for the reliability data cache table using a CSDD data dictionary and performing data transformation processing to obtain a reliability data specification table. The data cleaning template includes a first mapping between fields in the raw table and fields in the cache table, as well as data cleaning rules associated with the cache table fields. The data transformation template includes a second mapping between fields in the cache table and fields in the specification table, as well as data transformation rules associated with the specification table fields.

Description

Technical Field

[0001] This application relates to the field of data processing technology for reliability data, specifically to a method for processing multi-fleet reliability data based on the ATASPEC2000 specification and a reliability data processing apparatus for performing the aforementioned processing method. Background Technology

[0002] Reliability data is crucial for aircraft operational reliability management, and its quality directly impacts the accuracy of reliability analysis results. However, reliability data from different airlines and systems is complex and variable, often requiring further processing of collected multi-fleet reliability data to ensure its quality, bringing it within the reliability data range defined by the ATASPEC 2000 standard and meeting regulatory requirements.

[0003] Existing data processing technologies based on ETL (Extract-Transform-Load) can directly load business data from the source system into the target dataset for data transformation. However, multi-fleet reliability data involves at least eight types of reliability data, including flight hour cycles, unplanned replacements, fault records, and irregular flights, requiring processing a large variety and volume of reliability data. Moreover, the quality of collected multi-fleet reliability data is often inconsistent, making it difficult to meet the ATASPEC2000 specification requirements even after ETL processing, and may even lead to a large amount of inefficient and repetitive manual development work. Summary of the Invention

[0004] Therefore, in order to overcome the problem that existing data processing methods are unable to obtain reliability data that meets the requirements of the ATASPEC2000 specification after processing multi-fleet reliability data, this application proposes a multi-fleet reliability data processing method based on the ATASPEC2000 specification.

[0005] This application solves the above-mentioned technical problems through the following technical solution:

[0006] Specifically, according to one aspect of this application, a method for processing multi-fleet reliability data based on the ATASPEC2000 specification is provided, which includes the following steps:

[0007] Obtain raw reliability data files of different types from multiple fleets and parse the raw reliability data files to obtain a raw reliability data table with a two-dimensional table structure;

[0008] Using a CSDD data dictionary built according to the ATASPEC2000 specification, a data cleaning template is configured for each raw reliability data table, and data cleaning processing is performed accordingly to obtain the corresponding reliability data cache table; and

[0009] The CSDD data dictionary is used to configure a data transformation template for each reliability data cache table, and data transformation processing is performed accordingly to obtain reliability data and reliability data specification tables that conform to the ATASPEC2000 specification.

[0010] The data cleaning template includes a first mapping between the original table fields of the reliability data original table and the cache table fields of the reliability data cache table, as well as the data cleaning rules associated with the cache table fields. The data transformation template includes a second mapping between the cache table fields and the specification table fields of the reliability data specification table, as well as the data transformation rules associated with the specification table fields.

[0011] The multi-fleet reliability data processing method based on the ATASPEC2000 standard provided in this application offers a simpler, more reliable, efficient, and convenient processing approach for data integration scenarios involving multi-source heterogeneous reliability data. It can clean and transform raw reliability data files from different data providers and in different formats, resulting in high-quality, compliant reliability data. Furthermore, it can consolidate the processed data into a standardized table, thereby completing the data integration work for multiple sources and saving significant labor costs.

[0012] According to one embodiment of this application, the step of obtaining a raw reliability data table includes reading a raw reliability data file to generate a raw reliability data table containing raw table fields and their corresponding data, including data source, title field, file name and upload time, and displaying the raw reliability data table using a graphical user interface.

[0013] According to one embodiment of this application, the processing method further includes, based on business needs, writing data cleaning rules or data transformation rules and corresponding rule low-level codes for the CSDD data dictionary through a graphical user interface, and displaying the completed CSDD data dictionary. The embedded development of all operations and rule low-level codes is completed through the graphical user interface, and the CSDD data dictionary is updated and maintained, making the development and maintenance process simplified and transparent.

[0014] According to one embodiment of this application, the processing method further includes automatically matching the original reliability data table with data cleaning templates and data transformation templates stored in the database using a text semantic similarity algorithm; and determining the original reliability data file as a new type when the matching fails.

[0015] According to one embodiment of this application, when the original reliability data file is determined to be of a new type, the step of configuring the data cleaning template includes selecting an original table field from the original reliability data table displayed in the graphical user interface, selecting the element mapped by the selected original table field according to the first mapping from the CSDD data dictionary through the graphical user interface as a cache table field, and obtaining the data cleaning rules associated with the cache table field from the CSDD data dictionary.

[0016] According to one embodiment of this application, when the original reliability data file is determined to be of a new type, the step of configuring the data conversion template includes selecting a cache table field to be processed for data conversion in a reliability data cache table displayed in a graphical user interface, selecting the element mapped by the selected cache table field according to the second mapping from the CSDD data dictionary as a canonical table field, and obtaining the data conversion rule associated with the canonical table field from the CSDD data dictionary.

[0017] According to one embodiment of this application, the method further includes storing the CSDD data dictionary, as well as the created and configured data cleaning template and data transformation template, in a database. When faced with a new type of raw reliability data file, the data cleaning template and data transformation template only need to be configured once. After the template configuration is completed, subsequent uploads of the same type of raw reliability data file will automatically match the data cleaning template and data transformation template, thus automatically performing data cleaning and data transformation processing.

[0018] According to one embodiment of this application, the step of automatically matching data cleaning templates and data transformation templates includes reading data source data from the original reliability data table, and using a text semantic similarity algorithm to search for and obtain data cleaning templates and data transformation templates that include the data source data.

[0019] According to one embodiment of this application, the step of automatically matching the data cleaning template and the data transformation template further includes:

[0020] For each data cleaning template obtained from the search, read the existing original table fields mapped to the data cleaning template, and then compare the read existing original table fields with the original table fields in the reliability data original table. The data cleaning template with the most consistent comparison results is taken as the most similar data cleaning template.

[0021] The original reliability data table is cleaned using the most similar data cleaning template to obtain a reliability data cache table; and

[0022] For each data transformation template obtained from the search, read the existing cache table fields mapped to the data transformation template, and compare the read existing cache table fields with the cache table fields in the reliability data cache table in turn. The data transformation template with the most consistent comparison results is taken as the most similar data transformation template.

[0023] According to one embodiment of this application, the processing method further includes:

[0024] Read all known original table fields mapped by the most similar data cleaning template, and automatically match the read known original table fields with the original table fields sequentially through text comparison; and

[0025] When automatic field matching fails, the corresponding known original table field will be displayed.

[0026] According to one embodiment of this application, the processing method further includes sending the array containing the reliability data to a table to be corrected when the reliability data cleaning or data conversion fails, and highlighting the reliability data in the array that failed to be cleaned or converted in a different color in the table to be corrected.

[0027] According to one embodiment of this application, the processing method further includes correcting the reliability data rows in the table to be corrected through a graphical user interface, so as to re-perform data cleaning or data transformation processing as needed.

[0028] According to one embodiment of this application, the processing method further includes encapsulating the rule low code associated with the data cleaning rule and the rule low code associated with the data transformation rule into a batch processing template of the rule engine, so that the batch processing template can be called to perform batch data cleaning and data transformation processing when data cleaning processing is performed using the data cleaning template and when data transformation processing is performed using the data transformation template.

[0029] According to one embodiment of this application, the batch data cleaning and data transformation processing steps include using the input class interface of the batch processing template to obtain the original table of reliability data and the cache table field types mapped by the fields of the original table, and using the output class interface of the batch processing template to output the reliability data after data cleaning and data transformation processing and the status information of data cleaning and data transformation processing.

[0030] According to one embodiment of this application, the data cleaning process includes:

[0031] Reconstruct the original reliability data table according to the preset reliability data cache table structure to obtain the first object dataset.

[0032] Based on the first mapping in the data cleaning template, obtain the cache table fields corresponding to the original table fields in the object dataset, and replace the corresponding original table fields in the first object dataset with the cache table fields; and

[0033] The reliability data in the first object dataset corresponding to the cache table fields is cleaned according to the data cleaning rules associated with the cache table fields, thereby obtaining the reliability data cache table.

[0034] According to one embodiment of this application, the data cleaning rules include removing duplicate data rows, padding missing data with zeros, verifying the validity of data, and correcting data with numerical errors.

[0035] According to one embodiment of this application, the data cleaning step further includes:

[0036] For each cache table field in the first object dataset, when there are no associated data cleaning rules, the cache table field and its corresponding reliability data are stored in an array, and the data in the array is directly returned to the first object dataset; and

[0037] When there are associated data cleaning rules, the cache table fields and their corresponding reliability data are stored in an array as key-value pairs. Then, the data is cleaned according to the associated data cleaning rules, and the cleaned data in the array is directly returned to the first object dataset.

[0038] According to one embodiment of this application, the data cleaning step further includes, after performing data cleaning processing on all original reliability data tables, for each first object dataset, traversing the arrays in the first object dataset, encapsulating all traversed arrays into first CRUD objects according to a certain size, and then loading all first CRUD objects into a reliability data cache table in batches.

[0039] According to one embodiment of this application, the data conversion processing steps include:

[0040] Reconstruct the reliability data cache table according to the preset reliability data specification table structure to obtain the second object dataset;

[0041] Based on the second mapping in the data transformation template, obtain the canonical table fields corresponding to the cache table fields in the second object dataset, and replace the corresponding cache table fields in the second object dataset; and

[0042] According to the data transformation rules associated with the fields in the specification table, the reliability data corresponding to the fields in the specification table in the second object dataset is transformed, and a reliability data specification table is obtained.

[0043] According to one embodiment of this application, the processing method further includes designing a reliability data specification table structure that conforms to the ATASPEC2000 specification based on the reliability data acquisition standard, wherein the fields in the reliability data specification table structure have consistent naming and constraint rules.

[0044] According to one embodiment of this application, the data conversion step further includes:

[0045] When the second mapping contains a many-to-one mapping relationship and the canonical table fields do not have associated data transformation rules, multiple canonical table fields are concatenated with a separator, and the multiple canonical table fields obtained therefrom and their corresponding reliability data are stored in an array. Then, the data in the array is directly returned to the second object dataset.

[0046] When the second mapping contains a many-to-one mapping relationship and the canonical table field has associated data transformation rules, the canonical table field and its corresponding reliability data are stored in an array as multiple key-value pairs. Then, the data is transformed according to the data transformation rules corresponding to the canonical table field, and the transformation result data in the array is directly returned to the second object dataset.

[0047] When the second mapping contains a one-to-many mapping relationship and the canonical table fields do not have associated data transformation rules, each canonical table field and its corresponding reliability data are stored in an array, and the data in the array is directly returned to the second object dataset; and

[0048] When the second mapping contains a one-to-many mapping relationship and the canonical table fields have associated data transformation rules, each canonical table field and its corresponding reliability data are stored in an array as key-value pairs. Then, the data is transformed according to the data transformation rules corresponding to each canonical table field, and the transformation result data in the array is directly returned to the second object dataset.

[0049] According to one embodiment of this application, the data transformation step further includes, after performing data transformation processing on all reliability data cache tables, for each second object dataset, traversing the arrays in the second object dataset, encapsulating all traversed arrays into second CRUD objects according to a certain size, and then loading all second CRUD objects in batches into a reliability data specification table.

[0050] According to another aspect of this application, a reliability data processing apparatus for performing the aforementioned processing method is provided, the reliability data processing apparatus comprising:

[0051] The data acquisition module is used to acquire the uploaded raw reliability data file and parse the raw reliability data file to obtain the corresponding raw reliability data table.

[0052] The graphical configuration module allows users to configure data cleaning and transformation templates for the raw reliability data table, maintain the CSDD data dictionary built according to the ATASPEC2000 specification, and display the raw reliability data table and the reliability data cache table.

[0053] The data processing module is used to clean the original reliability data table using a data cleaning template to obtain a corresponding reliability data cache table, and then uses a data transformation template to transform the reliability data cache table to obtain a corresponding reliability data specification table; and

[0054] The database storage module is used to receive and store data cleaning templates, data transformation templates, and CSDD data dictionaries from the graphical configuration module, receive and store reliability data cache tables and reliability data specification tables from the data processing module, and provide existing data cleaning templates and data transformation templates to the data acquisition module.

[0055] According to one embodiment of this application, the data acquisition module includes:

[0056] The file upload unit is used to allow the upload of raw reliability data files and to parse the raw reliability data files to obtain the corresponding raw reliability data table; and

[0057] The template auto-matching unit is used to retrieve existing data cleaning templates and data transformation templates from the database storage module, and automatically match the most similar existing data cleaning templates and data transformation templates for the original data table for reliability.

[0058] According to one embodiment of this application, the graphical configuration module includes:

[0059] The data cleaning template configuration unit is used to configure data cleaning templates, allowing the establishment of the first mapping of the original table fields in the original table of reliable data.

[0060] The data transformation template configuration unit is used to configure data transformation templates, allowing the establishment of a second mapping for cache table fields in the reliability data cache table; and

[0061] The data dictionary maintenance unit allows users to create, edit, and delete field standards in the CSDD data dictionary as needed. The maintainable attributes of field standards include business term names, definitions, examples, data types, default data cleaning rules, and data transformation rules.

[0062] According to one embodiment of this application, the data processing module includes a data batch processing unit for calling a batch processing template of the rule engine. The batch processing template encapsulates low-level rule codes for data cleaning rules and data transformation rules, enabling batch and rapid data cleaning and data transformation processing of the reliability data to be processed using the batch processing template.

[0063] According to one embodiment of this application, the data processing module further includes a data object loading unit, which is used to encapsulate the reliability data after data cleaning and / or data transformation into CRUD objects, and to load the encapsulated CRUD object data in batches into a reliability data cache table or a reliability data specification table.

[0064] Based on common knowledge in the field, the above-mentioned preferred conditions can be combined arbitrarily to obtain various preferred embodiments of this application.

[0065] The beneficial technical effects and advantages that can be achieved by the processing method and reliability data processing apparatus according to the above embodiments of this application are as follows:

[0066] This application addresses data integration scenarios for reliability data tables, creating a simpler, more reliable, efficient, and convenient processing method. It cleanses and transforms reliability data files from different data providers and in different formats, resulting in high-quality, compliant reliability data. Furthermore, it consolidates the processed data into a single table, thus completing data integration from multiple sources and significantly reducing labor costs.

[0067] When faced with a new type of raw reliability data file, only a one-time configuration of the data cleaning template and data transformation template is required. After template configuration, subsequent uploads of the same type of raw reliability data file will automatically generate the corresponding data cleaning and transformation templates, and then automatically complete the data cleaning and data standardization processes. Furthermore, the use of a rule engine for batch data processing effectively improves the efficiency of data acquisition, transmission, and processing, significantly reduces data errors, and efficiently processes reliability data to meet specification requirements.

[0068] The graphical user interface enables the creation and configuration of data cleaning and transformation templates for new types of raw reliability data files, as well as the embedded development of rules in low-code mode. It also enables the updating and maintenance of the CSDD data dictionary, making the development and maintenance process simple and transparent. Attached Figure Description

[0069] Figure 1 This is a flowchart of a method for processing multi-fleet reliability data based on the ATASPEC2000 specification according to a preferred embodiment of this application.

[0070] Figure 2 This is a schematic diagram of an exemplary reliability data source table, reliability data cache table, and reliability data specification table according to a preferred embodiment of this application.

[0071] Figure 3 This is a schematic diagram of a reliability data processing apparatus according to a preferred embodiment of the present application. Detailed Implementation

[0072] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the accompanying drawings showing multiple embodiments according to this application. It should be understood that all other embodiments obtained by those skilled in the art based on the embodiments described in this application without creative effort will fall within the scope of protection of this application.

[0073] like Figure 1 As shown, the method for processing multi-fleet reliability data based on the ATASPEC2000 specification includes the following steps:

[0074] S1. Obtain raw reliability data files of different types from multiple fleets, and parse the raw reliability data files to obtain a raw reliability data table with a two-dimensional table structure.

[0075] S2. Utilize the CSDD data dictionary constructed according to the ATASPEC2000 specification to configure a data cleaning template for each raw reliability data table and perform data cleaning processing accordingly, thereby obtaining the corresponding reliability data cache table; and

[0076] S3. Configure a data transformation template for each reliability data cache table using the CSDD data dictionary, and perform data transformation processing on each reliability data cache table to obtain reliability data and reliability data specification tables that conform to the ATASPEC2000 specification.

[0077] The data cleaning template includes a first mapping between the original table fields of the reliability data original table and the cache table fields of the reliability data cache table, as well as the data cleaning rules associated with the cache table fields. The data transformation template includes a second mapping between the cache table fields and the specification table fields of the reliability data specification table, as well as the data transformation rules associated with the specification table fields.

[0078] In this application, the original reliability data files may originate from different airlines or other organizational structures and may have different types. The original reliability data files may be any two-dimensional table structure data file, with file formats including but not limited to ".xls", ".csv", and ".txt" with fixed delimiters. Preferably, the first line of the original reliability data file is the header line. For example, if the original reliability data file originates from a specific airline, its header line may include the following fields: aircraft number, aircraft type, flight date, flight hours, wheel chock hours, number of normal takeoffs and landings, number of consecutive takeoffs and landings, cumulative flight hours, cumulative wheel chock hours, cumulative takeoffs and landings, cumulative consecutive takeoffs and landings, whether grounded, and reason for grounding.

[0079] In this application, the CSDD data dictionary is pre-defined and constructed according to the ATASPEC2000 specification. It includes multiple elements, each typically including a field name, comment, type, length, value range, and default data cleaning or transformation rules. The total number of elements in the CSDD data dictionary can be expanded according to the relevant field. Furthermore, data cleaning or transformation rules can be written for the CSDD data dictionary through a graphical user interface according to business needs, and corresponding rule low-level code can be developed. Optionally, the completed CSDD data dictionary can also be displayed using a graphical user interface.

[0080] For data cleaning templates of raw reliability data tables, data cleaning rules typically include removing duplicate rows, padding missing data with zeros, validating data validity, unit conversion, and correcting numerical errors. When developing low-code rules, you only need to develop custom cleaning and transformation rule code for input instances, following the format of the input class, including data calculation, type conversion, and data validation.

[0081] In step S1, obtaining the raw reliability data table includes reading the raw reliability data file to generate a raw reliability data table containing raw table fields and their corresponding data, including data source, title field, file name, and upload time. For example, for the aforementioned ".xls" format file named "Flight Hour Cycle," after reading and parsing, the following table is generated: Figure 2 The raw reliability data table shown on the left can be displayed using a graphical user interface. It should be understood that, depending on actual needs, the raw reliability data table may include more common raw table fields, which will not be listed here.

[0082] In step S1, after obtaining the original reliability data table, a text semantic similarity algorithm can be used to automatically match the original reliability data table with the data cleaning template and data transformation template stored in the database. If the matching fails, the original reliability data file is determined to be of a new type.

[0083] In step S2, when the original reliability data file is determined to be of a new type, the step of configuring the data cleaning template may include selecting original table fields in the original reliability data table displayed in the graphical user interface, selecting the elements mapped by the first mapping from the CSDD data dictionary of the selected original table fields as cache table fields through the graphical user interface, and obtaining the data cleaning rules associated with the cache table fields from the CSDD data dictionary, thereby completing the configuration of the data cleaning template for the original reliability data table. Optionally, the data cleaning template name is automatically named as a concatenation of the data source data and the original reliability data file name.

[0084] Optionally, based on business needs, elements are selected from the CSDD data dictionary as corresponding cache table fields for each original table field using a combination of drop-down lists and fuzzy search via a graphical user interface. This forms the first mapping between multiple original table fields in the reliability data original table and their corresponding cache table fields, and the first mapping is displayed through a graphical user interface. It should be understood that the first mapping may include the original table fields, the corresponding cache table fields, and the first mapping relationship between the original table fields and their corresponding cache table fields. Figure 2 In the example, the mapping relationship between the original table fields in the original reliability data table and the cache table fields in the reliability data cache table is a one-to-one mapping relationship.

[0085] In step S3, when it is determined that the original reliability data file is of a new type, the step of configuring the data conversion template includes selecting the cache table field to be processed for data conversion in the reliability data cache table displayed in the graphical user interface, selecting the element mapped by the selected cache table field according to the second mapping from the CSDD data dictionary as the canonical table field, and obtaining the data conversion rules associated with the canonical table field from the CSDD data dictionary, thereby completing the configuration of the data conversion template for the reliability data cache table.

[0086] Optionally, display such as in the graphical user interface Figure 2 The reliability data cache table shown illustrates a second mapping between multiple cache table fields and their corresponding canonical table fields. This second mapping includes the cache table fields, their corresponding canonical table fields, and the second mapping relationship between them. For example... Figure 2 As shown in the consecutive takeoff and landing count cache table field, the second mapping relationship can include a one-to-many mapping relationship. Furthermore, as... Figure 2 As shown in the fields of the normal takeoff and landing count cache table and the consecutive takeoff and landing count cache table, the second mapping relationship can also include a many-to-one mapping relationship. Data transformation rules are exemplarily shown below. Figure 2 As shown, it can include data summation, counting, and categorical counting.

[0087] Preferably, after configuring the data cleaning template and data transformation template and writing the CSDD data dictionary, the CSDD data dictionary, data cleaning template, and data transformation template are stored in the database.

[0088] Optionally, in step S1, a newly added raw reliability data file can be uploaded using a graphical user interface, a new raw reliability data table can be obtained through parsing, and a data cleaning template and data transformation template stored in the database can be automatically matched for the new raw reliability data table using a text semantic similarity algorithm, thereby completing the automatic configuration of the data cleaning template and data transformation template. Specifically, the step of automatically matching the data cleaning template and data transformation template includes reading the data source data in the raw reliability data table, and using a text semantic similarity algorithm to search for and obtain the data cleaning template and data transformation template that include the data source data.

[0089] Alternatively, in step S2, when a data cleaning template including data source data from the original reliability data table is automatically matched (i.e., the original reliability data file is not a new type), the step of automatically matching (i.e., configuring) the data cleaning template includes, for each data cleaning template obtained through the search, reading the existing original table fields mapped by the data cleaning template, and sequentially comparing the read existing original table fields with the original table fields in the original reliability data table. The data cleaning template with the most consistent comparison result is taken as the most similar data cleaning template. Subsequently, the most similar data cleaning template is used to perform data cleaning processing on the original reliability data table to obtain the reliability data cache table. Optionally, in step S2, all known original table fields mapped by the most similar data cleaning template can also be read, and the original table fields can be automatically matched sequentially by text comparison. When the automatic matching of fields fails, the corresponding known original table fields are displayed.

[0090] Alternatively, in step S3, when a data transformation template including the data source data in the original reliability data table is automatically matched, i.e., the original reliability data file is not a new type, the step of automatically matching (i.e. configuring) the data transformation template may include, for each data transformation template obtained by the search, reading the existing cache table fields mapped by the data transformation template, comparing the read existing cache table fields with the cache table fields in the reliability data cache table in turn, and taking the data transformation template with the most consistent comparison results as the most similar data transformation template.

[0091] Once the data cleaning template is configured, in step S2, the specific steps for data cleaning using the data cleaning template may include reconstructing the original reliability data table according to a preset reliability data cache table structure to obtain a first object dataset; obtaining the cache table fields corresponding to the original table fields in the first object dataset according to the first mapping in the data cleaning template; replacing the corresponding original table fields in the first object dataset with the cache table fields; then cleaning the reliability data in the first object dataset corresponding to the cache table fields according to the data cleaning rules associated with the cache table fields; and thereby obtaining... Figure 2 The reliability data cache table is shown in the middle. Optionally, the reliability data cache table has the same name as the data cleaning template.

[0092] Once the data transformation template is configured, in step S3, the specific steps for data transformation using the data transformation template may include reconstructing the reliability data cache table according to the preset reliability data specification table structure to obtain the second object dataset; obtaining the specification table fields corresponding to the cache table fields in the second object dataset according to the second mapping in the data transformation template, and replacing the corresponding cache table fields in the second object dataset; then performing data transformation on the reliability data corresponding to the specification table fields in the second object dataset according to the data transformation rules associated with the specification table fields, thereby obtaining, for example, the following steps. Figure 2 The reliability data specification table is shown on the right. Preferably, the structure of the reliability data specification table is designed according to the reliability data acquisition standard, conforms to the ATASPEC2000 specification, and the fields in the reliability data specification table structure have consistent naming and constraint rules.

[0093] Optionally, when reliability data cleaning or transformation fails, the array containing the reliability data can be sent to a table to be corrected, and the reliability data in the array that failed to be cleaned or transformed can be highlighted in different colors in the table to be corrected. Optionally, the reason for the data cleaning or transformation failure can be displayed in the last column of the table to be corrected. Preferably, the reliability data rows in the table to be corrected can be corrected through a graphical user interface, and the array containing the corrected reliability data can be sent to the first object dataset or the second object dataset for data cleaning or transformation to be performed again as needed. For the original reliability data table, the reliability data cache table, the reliability data specification table, the first object dataset, or the second object dataset, the reliability data of each row forms an array.

[0094] It is known that using a rule engine (Drools) can improve data processing efficiency; therefore, the processing method provided in this application may also include using a rule engine for data cleaning and data transformation. Specifically, the low-level rules associated with data cleaning rules and the low-level rules associated with data transformation rules are encapsulated in a batch processing template of the rule engine. This allows the batch processing template to be invoked for batch data cleaning and data transformation when performing data cleaning processing using the data cleaning template and when performing data transformation processing using the data transformation template.

[0095] When performing batch data cleaning and transformation, the batch processing template uses its input interface to obtain the original reliability data table and the cache table field types mapped to the original table fields. It then uses its output interface to output the cleaned and transformed reliability data and the status information of the data cleaning and transformation process. Preferably, for a single object dataset, the Drools batch processing template only needs to be called once during data cleaning and transformation. However, due to computer memory limitations, when the dataset size exceeds the memory limit, it can be divided into multiple batches for processing.

[0096] To facilitate data retrieval, in step S2, the data cleaning step further includes, after cleaning all the original reliability data tables, for each first object dataset, traversing the arrays in the first object dataset, and encapsulating all the traversed arrays into first CRUD objects according to a certain size, and then loading all the first CRUD objects into a reliability data cache table in batches. Alternatively, in step S3, the data transformation step further includes, after transforming all the reliability data cache tables, for each second object dataset, traversing the arrays in the second object dataset, and encapsulating all the traversed arrays into second CRUD objects according to a certain size, and then loading all the second CRUD objects into a reliability data specification table in batches.

[0097] This application also provides a reliability data processing apparatus for performing the aforementioned processing method, such as... Figure 3 As shown, it includes a data acquisition module, a graphical configuration module, a data processing module, and a database storage module. These modules are interconnected via data cables, wired networks, or wireless networks. Therefore, these templates can transfer data to each other as needed.

[0098] The data acquisition module acquires the uploaded raw reliability data file and parses it to obtain the corresponding raw reliability data table. The graphical configuration module allows configuring data cleaning and transformation templates, maintaining a CSDD data dictionary built according to the ATASPEC2000 specification, and displaying the raw reliability data table and the reliability data cache table. The data processing module uses the data cleaning template to clean the raw reliability data table to obtain the corresponding reliability data cache table, and uses the data transformation template to transform the reliability data cache table to obtain the corresponding reliability data specification table. The database storage module receives and stores the data cleaning template, data transformation template, and CSDD data dictionary from the graphical configuration module, receives and stores the reliability data cache table and reliability data specification table from the data processing module, and provides existing data cleaning and data transformation templates to the data acquisition module.

[0099] like Figure 3 As shown, the data acquisition module includes a file upload unit and a template auto-matching unit. The file upload unit allows the uploading of raw reliability data files and parses them to obtain the corresponding raw reliability data table. Furthermore, the file upload unit supports batch uploading of raw reliability data files, recording the filename, data source, and upload time as log information, and automatically backing up the raw reliability data file. For the file upload unit, files with duplicate names are not allowed in the same batch of uploaded raw reliability data files. If the names of raw reliability data files uploaded in the same batch are duplicated, a duplicate message will be displayed, and the file upload will be terminated. The template auto-matching unit retrieves existing data cleaning templates and data transformation templates from the database storage module and automatically matches the most similar existing data cleaning template and data transformation template for the raw reliability data table.

[0100] Furthermore, the graphical configuration module includes a data cleaning template configuration unit, a data transformation template configuration unit, and a data dictionary maintenance unit. The data cleaning template configuration unit allows the establishment of the first mapping of fields from the original table in the reliability data source table for data cleaning template configuration. The data transformation template configuration unit allows the establishment of the second mapping of fields from the cache table in the reliability data cache table for data transformation template configuration. The data dictionary maintenance unit allows the creation, editing, and deletion of field standards in the CSDD data dictionary as needed. It also allows for the modification and deletion of attributes such as business terminology names, definitions, examples, data types, default data cleaning rules, and data transformation rules for field standards.

[0101] The data processing module may include a batch processing unit for calling batch processing templates from the rules engine. These templates encapsulate low-level code for data cleaning and transformation rules, enabling batch and rapid data cleaning and transformation of the reliability data to be processed. The module also includes a data object loading unit, which encapsulates the cleaned or transformed reliability data into CRUD objects and loads these encapsulated CRUD objects in batches into a reliability data cache table or a reliability data specification table.

[0102] The database storage module may include table storage units, template storage units, and data dictionary storage units. Table storage units store reliability data cache tables obtained after data cleaning and reliability data specification tables obtained after data transformation. Template storage units store data cleaning templates and data transformation templates created (i.e., configured) for new raw reliability data files. Data dictionary storage units store the completed or updated CSDD data dictionary.

[0103] The reliability data processing device provided in this application can be deployed in a browser / server manner. Users can access the operation page of the reliability data processing device through a web browser, enabling multiple terminal users in different locations to operate simultaneously.

[0104] This application provides a reliability data processing method and apparatus that includes graphical templates, low-code rule development, and greater transparency in the data processing process. It is suitable for standardized processing and querying of multi-source and multi-type reliability data. The processing method provided in this application can process raw reliability data files from different airlines into high-quality, standardized reliability data tables, and can import the processed reliability data into a unified standardized table for use in fleet operation reliability analysis.

[0105] Although the specific embodiments described in this application only depict the processing methods and data processing apparatus for multi-source heterogeneous reliability data processing and reliability analysis, it should be understood that the aforementioned methods and data processing apparatus can also be applied to data processing in other fields, such as commodity order management and building material records. In other words, the aforementioned processing methods and data processing apparatus can be applied to specific scenarios where any raw data files from different data providers and with different formats are cleaned and transformed according to specifications, and the processed data is consolidated into a single data table.

[0106] Furthermore, this application also provides a non-transitory computer-readable storage medium storing computer-readable instructions thereon, which, when executed at a computer system, cause the computer system to perform any of the processing methods described herein. Examples of computer-readable storage media include random access memory (RAM), read-only memory (ROM), optical disk, flash memory, hard disk storage, and other memory devices that can use magnetic, optical, and other techniques to store instructions or other data and are machine-accessible.

[0107] Furthermore, the term "computer-readable instructions and rule-based low-code" as used herein refers to any kind of executable code that is executed by the processor of a computer system, including code expressed in machine language, interpreted language, or scripting language. Computer-readable instructions and rule-based low-code include binary code, machine code, bytecode, code for limited integrated circuits, and code expressed in programming languages ​​such as C, Java, or Python. Computer-readable instructions and rule-based low-code can be, for example, any kind of software, firmware, script, module, or library that, when properly executed, processed, interpreted, compiled, or run in a virtual machine or other software environment, causes the processor of a computer system that supports executable code to perform the tasks specified by the code.

[0108] While specific embodiments of this application have been described above, those skilled in the art should understand that these are merely illustrative examples, and the scope of protection of this application is defined by the appended claims. Those skilled in the art can make various changes or modifications to these embodiments without departing from the principles and essence of this application, but all such changes and modifications fall within the scope of protection of this application.

Claims

1. A method for processing multi-fleet reliability data based on the ATASPEC2000 specification, comprising the following steps: Obtain raw reliability data files of different types from multiple fleets, and parse the raw reliability data files to obtain a raw reliability data table with a two-dimensional table structure; The CSDD data dictionary, constructed according to the ATASPEC2000 specification, is used to configure a data cleaning template for each of the original reliability data tables and perform data cleaning processing therefrom, thereby obtaining the corresponding reliability data cache table; as well as The CSDD data dictionary is used to configure a data transformation template for each reliability data cache table, and data transformation processing is performed on each reliability data cache table to obtain reliability data and a reliability data specification table that conform to the ATASPEC2000 specification. The data cleaning template includes a first mapping between the original table fields of the original reliability data table and the cache table fields of the reliability data cache table, as well as the data cleaning rules associated with the cache table fields. The data transformation template includes a second mapping between the cache table fields and the specification table fields of the reliability data specification table, as well as the data transformation rules associated with the specification table fields.

2. The processing method according to claim 1, wherein, The steps to obtain the original table of reliability data include: The original reliability data file is read to generate a reliability data original table, which includes the original table fields and their corresponding data, such as data source, title field, file name, and upload time. The reliability data original table is then displayed using a graphical user interface.

3. The processing method according to claim 2 further includes: Based on business requirements, data cleaning rules or data transformation rules and corresponding rule codes are written for the CSDD data dictionary through the graphical user interface, and the completed CSDD data dictionary is displayed.

4. The processing method according to claim 3 further includes: The text semantic similarity algorithm is used to automatically match the original reliability data table with the data cleaning template and data transformation template stored in the database; as well as If a match fails, the original reliability data file is determined to be of a new type.

5. The processing method according to claim 4, wherein, When the original reliability data file is determined to be of a new type, the steps for configuring the data cleaning template include: Select an original table field from the reliability data original table displayed in the graphical user interface, and use the graphical user interface to select the element mapped by the first mapping from the CSDD data dictionary as a cache table field, and obtain the data cleaning rules associated with the cache table field from the CSDD data dictionary.

6. The processing method according to claim 5, wherein, When the original reliability data file is determined to be of a new type, the steps for configuring the data conversion template include: In the reliability data cache table displayed in the graphical user interface, select the cache table field for which data transformation processing is desired, select the element mapped by the selected cache table field according to the second mapping from the CSDD data dictionary as the canonical table field, and obtain the data transformation rule associated with the canonical table field from the CSDD data dictionary.

7. The processing method according to claim 6 further includes: The CSDD data dictionary, along with the created and configured data cleaning template and data transformation template, are stored in the database.

8. The processing method according to claim 7, wherein, The steps for automatically matching the data cleaning template and the data transformation template include: Read the data source data from the original reliability data table, and use a text semantic similarity algorithm to search and obtain a data cleaning template and a data transformation template that include the data source data.

9. The processing method according to claim 8, wherein, The step of automatically matching the data cleaning template and the data transformation template further includes: For each data cleaning template obtained by the search, the existing original table fields mapped by the data cleaning template are read, and the read existing original table fields are compared with the original table fields in the reliability data original table in turn. The data cleaning template with the most consistent comparison results is taken as the most similar data cleaning template. The original reliability data table is cleaned using the most similar data cleaning template to obtain a reliability data cache table; and For each data transformation template obtained from the search, read the existing cache table fields mapped by the data transformation template, and compare the read existing cache table fields with the cache table fields in the reliability data cache table in turn. The data transformation template with the most consistent comparison results is taken as the most similar data transformation template.

10. The processing method according to claim 9, further comprising: Read all known original table fields mapped by the most similar data cleaning template, and automatically match the read known original table fields with the original table fields through text comparison. as well as When automatic field matching fails, the corresponding known original table field will be displayed.

11. The processing method according to claim 1, further comprising: When reliability data cleaning or data transformation fails, the array containing the reliability data is sent to the correction table, and the reliability data in the array that failed to be cleaned or transformed is highlighted in the correction table with different colors.

12. The processing method according to claim 11, further comprising: The reliability data rows in the table to be corrected are modified through a graphical user interface, allowing for data cleaning or transformation as needed.

13. The processing method according to claim 1, further comprising: The rule low code associated with the data cleaning rule and the rule low code associated with the data transformation rule are encapsulated into the batch processing template of the rule engine, so that the batch processing template can be called to perform batch data cleaning and data transformation processing when using the data cleaning template for data cleaning processing and when using the data transformation template for data transformation processing.

14. The processing method according to claim 13, the batch data cleaning and data transformation processing steps include: The original reliability data table and the cache table field types mapped to the original table fields are obtained using the input class interface of the batch processing template, and the reliability data after data cleaning and data transformation processing, as well as the status information of data cleaning and data transformation processing, are output using the output class interface of the batch processing template.

15. The processing method according to any one of claims 1-14, wherein, Data cleaning processes include: The reliability data original table is reconstructed according to the preset reliability data cache table structure to obtain the first object dataset; Based on the first mapping in the data cleaning template, obtain the cache table fields corresponding to the original table fields in the first object dataset, and replace the corresponding original table fields in the first object dataset with the cache table fields; and The reliability data in the first object dataset corresponding to the fields of the cache table is cleaned according to the data cleaning rules associated with the fields of the cache table, thereby obtaining the reliability data cache table.

16. The processing method according to claim 15, wherein, The data cleaning rules include removing duplicate data rows, padding missing data with zeros, verifying the validity of data, and correcting data with numerical errors.

17. The processing method according to claim 15, wherein, The data cleaning step also includes: For each cache table field in the first object dataset, when there are no associated data cleaning rules, the cache table field and its corresponding reliability data are stored in an array, and the data in the array is directly returned to the first object dataset; and When there are associated data cleaning rules, the cache table fields and their corresponding reliability data are stored in an array as key-value pairs. Then, data cleaning is performed according to the associated data cleaning rules, and the cleaning result data in the array is directly returned to the first object dataset.

18. The processing method according to claim 15, wherein, The data cleaning process also includes: After cleaning all the original reliability data tables, for each first object dataset, the arrays in the first object dataset are traversed, and all the traversed arrays are encapsulated into first CRUD objects according to a certain size. Then, all the first CRUD objects are loaded into a reliability data cache table in batches.

19. The processing method according to any one of claims 1-14, wherein, The data transformation and processing steps include: The reliability data cache table is reconstructed according to the preset reliability data specification table structure to obtain the second object dataset; According to the second mapping in the data transformation template, obtain the canonical table field corresponding to the cache table field in the second object dataset, and replace the corresponding cache table field in the second object dataset; and The reliability data corresponding to the fields in the specification table in the second object dataset is transformed according to the data transformation rules associated with the fields in the specification table, thereby obtaining the reliability data specification table.

20. The processing method according to claim 19, further comprising: The reliability data specification table structure is designed according to the reliability data acquisition standard and conforms to the ATASPEC2000 specification. The fields in the reliability data specification table structure have consistent naming and constraint rules.

21. The processing method according to claim 19, wherein, The data conversion step also includes: When the second mapping contains a many-to-one mapping relationship and the canonical table field has no associated data transformation rule, multiple canonical table fields are concatenated with a separator, and the multiple canonical table fields obtained therefrom and their corresponding reliability data are stored in an array. Then, the data in the array is directly returned to the second object dataset. When the second mapping contains a many-to-one mapping relationship and the canonical table field has associated data transformation rules, the canonical table field and its corresponding reliability data are stored in an array in the form of multiple key-value pairs. Then, the data is transformed according to the data transformation rules corresponding to the canonical table field, and the transformation result data in the array is directly returned to the second object dataset. When the second mapping contains a one-to-many mapping relationship, and the canonical table fields have no associated data transformation rules, each canonical table field and its corresponding reliability data are stored in an array, and the data in the array is directly returned to the second object dataset; and When the second mapping contains a one-to-many mapping relationship and the normative table fields have associated data transformation rules, each normative table field and its corresponding reliability data are stored in an array in the form of key-value pairs. Then, data transformation is performed according to the data transformation rules corresponding to each normative table field, and the transformation result data in the array is directly returned to the second object dataset.

22. The processing method according to claim 19, wherein, The data transformation steps also include: After performing data transformation on all reliability data cache tables, for each second object dataset, the arrays in the second object dataset are traversed, and all traversed arrays are encapsulated into second CRUD objects according to a certain size. Then, all second CRUD objects are loaded into a reliability data specification table in batches.

23. A reliability data processing apparatus for performing the processing method according to any one of claims 1 to 22, the reliability data processing apparatus comprising: The data acquisition module is used to acquire the uploaded raw reliability data file and parse the raw reliability data file to obtain the corresponding raw reliability data table. A graphical configuration module is used to allow configuring data cleaning templates and data transformation templates for the raw reliability data table, maintaining the CSDD data dictionary built according to the ATASPEC2000 specification, and displaying the raw reliability data table and the reliability data cache table; The data processing module is used to perform data cleaning processing on the original reliability data table using the data cleaning template to obtain a corresponding reliability data cache table, and to perform data transformation processing on the reliability data cache table using the data transformation template to obtain a corresponding reliability data specification table. as well as The database storage module is used to receive and store the data cleaning template, the data transformation template, and the CSDD data dictionary from the graphical configuration module, receive and store the reliability data cache table and the reliability data specification table from the data processing module, and provide existing data cleaning templates and data transformation templates to the data acquisition module.

24. The reliability data processing apparatus according to claim 23, wherein, The data acquisition module includes: The file upload unit is used to allow the upload of raw reliability data files and parse the raw reliability data files to obtain the corresponding raw reliability data table; and The template auto-matching unit is used to obtain existing data cleaning templates and data transformation templates from the database storage module, and automatically match the most similar existing data cleaning templates and data transformation templates for the original reliability data table.

25. The reliability data processing apparatus according to claim 24, wherein, The graphical configuration module includes: The data cleaning template configuration unit is used to create and configure data cleaning templates, allowing the establishment of the first mapping of the original table fields in the original table of the reliability data. A data transformation template configuration unit is used to create and configure data transformation templates, allowing the establishment of a second mapping of cache table fields in the reliability data cache table; and The data dictionary maintenance unit is used to allow the creation, editing, and deletion of field standards in the CSDD data dictionary as needed. The maintainable attributes of the field standards include business term names, definitions, examples, data types, default data cleaning rules, and data transformation rules.

26. The reliability data processing apparatus according to claim 25, wherein, The data processing module includes: The data batch processing unit is used to call the batch processing template of the rule engine. The batch processing template encapsulates the rule low code associated with the data cleaning rules and data transformation rules, so that the batch processing template can be used to perform batch and fast data cleaning and data transformation processing on the reliability data to be processed.

27. The reliability data processing apparatus according to claim 26, wherein, The data processing module further includes: The data object loading unit is used to encapsulate the reliability data after data cleaning and / or data transformation into CRUD objects, and to load the encapsulated CRUD object data in batches into a reliability data cache table or a reliability data specification table.

Images