A multi-source data preprocessing method and system for aerospace discrete manufacturing platform

Abstract

The present application discloses a multi-source data preprocessing method and system for aerospace discrete manufacturing platform, which relates to the field of aerospace discrete manufacturing data preprocessing technology. The method includes: adapting and processing input multi-source heterogeneous data through at least one dynamically configured adapter component to unify and standardize the data format to obtain adapted data; transmitting the adapted data to a distributed cleaning engine, which performs error detection, pattern recognition and logical consistency correction on the adapted data set to obtain a cleaned data set; calculating the data quality index of the cleaned data set through a quality assessment module; dynamically adjusting the cleaning rules based on the data quality index; providing data traceability and correction guidance according to the needs and feedback of upper-level applications through a preset quality feedback control mechanism, and continuously optimizing data quality. The present application can effectively improve the extensibility, dynamic flexibility and inheritance reusability of the cleaning rules of the database system.

Description

Technical Field

[0001] The present application relates to the technical field of aerospace discrete manufacturing data preprocessing, and in particular to a multi-source data preprocessing method and system for an aerospace discrete manufacturing platform. Background Art

[0002] The manufacturing process of aerospace products is mostly discrete manufacturing, which leads to the data source of aerospace products covering multi-source heterogeneous data generated and summarized by key process equipment, single-machine systems, detection devices and other objects in the main process links such as casting, machining, welding, sheet metal, heat treatment, assembly, testing, and inspection. In the multi-sensing device data acquisition system, the information provided by each sensor is subject to the constraints of the environmental state and the characteristics of the sensing device itself, resulting in the data collection of the manufacturing process may cause inaccurate impact on the back-end application. To this end, in the relevant technology, the main method is to input data cleaning methods into the database system to reduce or eliminate the impact of inaccurate data on the back-end application caused by inaccurate multi-source heterogeneous data collection.

[0003] However, at present, traditional data cleaning methods mainly use hard-coding methods to execute cleaning logic, and there are problems with the scalability, dynamic flexibility and inheritance reusability of cleaning rules for multiple data sources. Summary of the invention

[0004] The embodiments of the present application provide a multi-source data preprocessing method and system for an aerospace discrete manufacturing platform, so as to improve the problems of poor scalability, dynamic flexibility and inheritance reusability of cleaning rules for multiple data sources.

[0005] A first aspect of an embodiment of the present application provides a multi-source data preprocessing method for an aerospace discrete manufacturing platform, the method comprising:

[0006] Adapting and processing the input multi-source heterogeneous data through at least one dynamically configured adapter component to unify and standardize the data format and obtain adapted data;

[0007] The adapted data is transmitted to a distributed cleaning engine, and the distributed cleaning engine performs error detection, pattern recognition and logic consistency correction on the adapted data to obtain a cleaned data set;

[0008] Calculating a data quality index for the cleaned data set by a quality assessment module;

[0009] Dynamically adjust cleaning rules based on the data quality index;

[0010] Through the preset quality feedback control mechanism, data traceability and correction guidance are provided according to the needs and feedback of upper-level applications to continuously optimize data quality.

[0011] In some embodiments of the present application, the step of adapting and processing the input multi-source heterogeneous data through at least one dynamically configured adapter component to unify and standardize the data format to obtain adapted data includes:

[0012] Automatically analyzing the data structure and content format of the received multi-source heterogeneous data;

[0013] Use pre-trained machine learning models to identify data features and generate adaptation rules;

[0014] Applying the generated adaptation rules to the multi-source heterogeneous data to convert them into the adapted data, wherein the adapted data has a unified format recognizable by the database system;

[0015] During the adaptation process, new data patterns are continuously learned to achieve real-time updating and optimization of adaptation rules.

[0016] In some embodiments of the present application, the distributed cleaning engine includes a dynamic response system for data volume, and the steps of transmitting the adapted data to the distributed cleaning engine, and the distributed cleaning engine performing error detection, pattern recognition, and logical consistency correction on the adapted data set to obtain a cleaned data set include:

[0017] Adjust computing resource allocation based on data traffic;

[0018] Real-time monitoring of various data anomalies and distribution to corresponding nodes for parallel processing;

[0019] Summarize the parallel processing results and integrate them to generate the final cleaned data set;

[0020] After data processing is completed, computing resources are automatically reduced.

[0021] In some embodiments of the present application, the step of calculating the data quality index of the cleaned data set by the quality assessment module includes:

[0022] Set different categories of quality control indicators and assign corresponding monitoring logic to each indicator;

[0023] After data cleaning, check each quality control indicator item by item and record its status;

[0024] Integrate all indicator statuses and calculate the data quality index;

[0025] Generate a quality assessment report for reference and further processing by system administrators or data cleaning engines.

[0026] In some embodiments of the present application, the factors that need to be considered in calculating the data quality index include:

[0027] The relevance of data items to business functions and their impact on business processes; the strength of association and logical consistency between data items in the data set; the matching degree between the frequency of data updates and freshness and the real-time response requirements of the business.

[0028] In some embodiments of the present application, the calculation formula of the data quality index is:

[0029]

[0030] Among them, W i represents the weight factor of the i-th data item, which is related to the manufacturing link relevance of the data item, the impact on system performance, or the cost of change; V i stands for data validation coefficient, which indicates the degree to which the data passes a series of validation procedures; Q i is the single quality score of the i-th data item, which measures the accuracy, completeness and other attributes of the data; D i It is a data dependency indicator, indicating the correlation between data items or the intensity of their dependency in the business process; i It is a data timeliness indicator, reflecting the freshness of the data or its closeness to the current operation time; E i represents the environmental factor score associated with the i-th data item, including data generation time, signal strength, signal-to-noise ratio, or other important environmental factors in aerospace discrete manufacturing; R i It is the score of the degree to which the data item complies with the established business rules; α i and β i is the adjustment coefficient, which is adjusted according to the characteristics of different data items and the requirements of business scenarios; Z is the normalization factor, which ensures that the CQI remains within a reasonable range; f() represents a mixed nonlinear function that combines multiple variables to comprehensively obtain the quality score of each data item; e is the base of the natural logarithm.

[0031] In some embodiments of the present application, the step of dynamically adjusting the cleaning rules based on the data quality index includes:

[0032] Conduct real-time data mining to identify possible abnormal patterns and potential data errors;

[0033] Based on the mining results, analyze the effectiveness of the rules in the cleaning process and their scope of application;

[0034] Generate a rule adjustment plan and verify the validity of the rule adjustment plan;

[0035] After determining that the rule adjustment solution is feasible, the existing cleaning rule base is updated and redeployed to the distributed cleaning engine.

[0036] In some embodiments of the present application, the steps performed by the quality feedback control mechanism include:

[0037] Set up a listening program on the cleaned data set to detect and record quality change events;

[0038] Automatically generate quality problem reports based on detected events and analyze possible causes against business rules;

[0039] Use statistical methods to predict the potential impact of quality events on subsequent data processing cycles;

[0040] Provide data cleaning and maintenance recommendations based on predictive analysis to guide subsequent operations.

[0041] In some embodiments of the present application, the step of providing data tracing and correction guidance according to the requirements and feedback of upper-layer applications and continuously optimizing data quality includes:

[0042] After data cleaning is completed, collect meta-information related to the cleaned data, including data source, cleaning time, etc.

[0043] Establish data metadata index and query interface to facilitate upper-level applications to conduct data traceability query;

[0044] Receive feedback from upper-level applications and perform data tracing and analysis based on issues identified in the feedback;

[0045] Based on the results of the retrospective analysis, make necessary corrections and updates to the data source or cleaning rules.

[0046] In a second aspect, an embodiment of the present application further provides a multi-source data preprocessing system for an aerospace discrete manufacturing platform, the system comprising:

[0047] An adapter component that can be dynamically configured to adapt to specific data source characteristics, including data format and structure, and can achieve data unification and standardization;

[0048] a distributed cleaning engine, communicating with the adapter component to receive the adapted data, the distributed cleaning engine being used to perform error detection, pattern recognition and logic consistency correction of the data, and supporting parallel processing of the data;

[0049] A quality assessment component, used to calculate a cleaning quality index of data processed by the distributed cleaning engine;

[0050] A cleaning rule optimization module dynamically adjusts the cleaning rules based on the cleaning quality index obtained from the quality assessment component and is capable of predicting future data cleaning needs based on feedback data;

[0051] Quality feedback control system that provides data traceability and correction guidance based on the needs and feedback from upstream systems to optimize data quality and provide insights into cleaning operations.

[0052] It can be seen that the method of the embodiment of the present application can bring the following beneficial effects:

[0053] The multi-source heterogeneous data covered in the aerospace manufacturing process is often difficult to be effectively processed in traditional database systems, resulting in the data quality of back-end applications being affected. The present application can quickly adapt to the data types generated by different process links (such as casting, welding, etc.) and key equipment through dynamically configured adapter components, thereby improving the system's ability to handle diversified data sources. Since the data in the aerospace manufacturing link is collected by multi-sensing devices, the data of these devices are often restricted by the environment and equipment characteristics, and are prone to errors. The present invention can significantly reduce data errors from key process equipment and stand-alone systems through efficient data processing of distributed cleaning engines, ensure the data accuracy of aerospace products, and enhance the data quality of back-end applications. Traditional methods often lack the ability to comprehensively evaluate data quality. In this solution, the quality assessment module provides a method for objective evaluation based on the data quality index, making the data cleaning process traceable and quantifiable, which is particularly important for ensuring the quality standards of aerospace products. In traditional technologies, in the face of cleaning tasks from different data sources, hard-coded cleaning rules are often inflexible and difficult to expand and reuse. The system can dynamically adjust and optimize the cleaning rules based on the data quality index calculated in real time. Such intelligent optimization is critical to dealing with the complex and changeable data characteristics in the aerospace discrete manufacturing process. In aerospace discrete manufacturing, the accuracy of manufacturing process data is critical to subsequent product testing and inspection. The quality feedback control mechanism provided by the present invention not only enhances the data traceability, but also provides data correction guidance according to the upper-level application requirements, optimizing the entire process from aerospace product manufacturing to data application. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0055] Figure 1 It is a schematic diagram of an application scenario between a terminal and a server provided in an embodiment of the present application;

[0056] Figure 2 It is a flowchart of a multi-source data preprocessing method for aerospace discrete manufacturing platform provided in an embodiment of the present application;

[0057] Figure 3 It is a structural schematic diagram of a multi-source data preprocessing system for aerospace discrete manufacturing platform provided in an embodiment of the present application;

[0058] Figure 4 It is a schematic diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0059] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.

[0060] The embodiment of the present application provides a multi-source data preprocessing method and system for aerospace discrete manufacturing platform, and also provides corresponding devices and storage media. Among them, the device can be an Internet of Things device, and the storage medium can be a computer storage medium. The multi-source data preprocessing method for aerospace discrete manufacturing platform can be integrated in a device, which can be a server, or a terminal or other device.

[0061] Among them, the server can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, network acceleration services (Content Delivery Network, CDN), as well as big data and artificial intelligence platforms.

[0062] The terminal may be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, etc., but is not limited thereto. The terminal and the server may be directly or indirectly connected via wired or wireless communication, and this application does not limit this.

[0063] For example, Figure 1As shown, the server can adapt and process the input multi-source heterogeneous data through at least one dynamically configured adapter component to unify and standardize the data format to obtain adapted data; transmit the adapted data to a distributed cleaning engine, and the distributed cleaning engine performs error detection, pattern recognition and logical consistency correction on the adapted data set to obtain a cleaned data set; calculate the data quality index for the cleaned data set through a quality assessment module; dynamically adjust the cleaning rules based on the data quality index; and provide data traceability and correction guidance according to the needs and feedback of upper-level applications through a preset quality feedback control mechanism to continuously optimize data quality.

[0064] It should be noted that the above steps performed in the server may also be performed in the terminal, and this embodiment does not limit this.

[0065] In the embodiments of the present application, "plurality" refers to two or more than two. In the embodiments of the present application, "first" and "second" are used to distinguish descriptions and should not be understood as implying relative importance.

[0066] It should be noted that the description order of the following embodiments is not intended to limit the preferred order of the embodiments.

[0067] See also Figure 2 In this embodiment, a multi-source data preprocessing method for aerospace discrete manufacturing platform is provided, which is applied to a data processing system. The data processing system includes a perception layer, a network layer, a service layer and an application layer, such as Figure 2 As shown, the specific process of the multi-source data preprocessing method for aerospace discrete manufacturing platform can be as follows:

[0068] S1. Adapting and processing input multi-source heterogeneous data through at least one dynamically configured adapter component to unify and standardize the data format and obtain adapted data;

[0069] S2, transmitting the adapted data to the distributed cleaning engine, which performs error detection, pattern recognition and logical consistency correction on the adapted data to obtain a cleaned data set;

[0070] S3, calculating the data quality index of the cleaned data set through the quality assessment module;

[0071] S4. Dynamically adjust cleaning rules based on data quality index;

[0072] S5. Through the preset quality feedback control mechanism, data traceability and correction guidance are provided according to the needs and feedback of upper-level applications to continuously optimize data quality.

[0073] The technical solution provided by the present application is mainly through the cooperation between the adapter component, the distributed cleaning engine, the quality assessment module and the quality feedback control mechanism, so that the database system can face multiple data sources, and the extensibility, dynamic flexibility and inheritance reusability of the cleaning rules of the database system are better in the cross-related technologies. In detail, the multi-source heterogeneous data covered in the aerospace manufacturing process is often difficult to effectively process in the traditional database system, resulting in the data quality of the back-end application being affected. Through the dynamically configured adapter component, the present application can quickly adapt to the data types generated by different process links (such as casting, welding, etc.) and key equipment, thereby improving the system's ability to handle diversified data sources. Since the data in the aerospace manufacturing link is collected by multi-sensing devices, the data of these devices are often restricted by the environment and equipment characteristics, and are prone to errors. The present invention can significantly reduce the data errors from key process equipment and stand-alone systems through the efficient data processing of the distributed cleaning engine, ensure the data accuracy of aerospace products, and enhance the data quality of back-end applications. Traditional methods often lack the ability to comprehensively evaluate data quality. In this solution, the quality assessment module provides a method for objective evaluation based on the data quality index, so that the data cleaning process is traceable and quantifiable, which is particularly important for ensuring the quality standards of aerospace products. In traditional technology, when faced with cleaning tasks from different data sources, hard-coded cleaning rules are often inflexible and difficult to expand and reuse. However, this system can dynamically adjust and optimize cleaning rules based on the data quality index calculated in real time. Such intelligent optimization is critical for dealing with the complex and changeable data characteristics in the aerospace discrete manufacturing process. In aerospace discrete manufacturing, the accuracy of manufacturing process data is critical for subsequent product testing and inspection. The quality feedback control mechanism provided by the present invention not only enhances the data traceability, but also provides data correction guidance according to the upper-level application requirements, optimizing the entire process from aerospace product manufacturing to data application.

[0074] In some embodiments, the step of adapting and processing the input multi-source heterogeneous data through at least one dynamically configured adapter component to unify and standardize the data format to obtain adapted data includes:

[0075] Automatically analyzing the data structure and content format of the received multi-source heterogeneous data;

[0076] Use pre-trained machine learning models to identify data features and generate adaptation rules;

[0077] Applying the generated adaptation rules to the multi-source heterogeneous data to convert them into the adapted data, wherein the adapted data has a unified format recognizable by the database system;

[0078] During the adaptation process, new data patterns are continuously learned to achieve real-time updating and optimization of adaptation rules.

[0079] This embodiment mainly introduces a dynamically configured adapter component, relying on the adapter component to intelligently and automatically process multi-source heterogeneous data, that is, data from different sensors and systems. In detail, first, the structure and content format of the received multi-source heterogeneous data are analyzed by an automated tool. The method adopted is to use natural language processing (NLP) technology and pattern matching algorithms to classify and archive data elements, such as separating and standardizing timestamps, reading values, units, etc. Next, a pre-trained machine learning model is used to identify the characteristics of these data, and adaptation rules are generated based on the identification results. The machine learning model in this step may include a decision tree algorithm, which enables it to generate a decision logic based on the pattern in the historical data for identifying and adapting new data. These automatically generated rules are then applied to data from various sources, converting them into a consistent format that can be effectively processed by the database system, thereby completing the data adaptation process. The conversion process focuses on eliminating incompatibilities between data, such as differences in measurement units or differences in time representation in different data sources. During the adaptation process, the system will continue to learn and identify new data patterns and update optimization rules in real time. This means that the adapter is not only a static match, but also has dynamic learning capabilities and can be adaptively adjusted as the data source is updated and changed. Through the above scheme, the information collected by the database system from multiple different data sources can be effectively integrated and used, effectively avoiding a large number of data processing problems caused by data inconsistency, such as data redundancy, format mismatch, and data incompleteness.

[0080] It should be noted that the process of dynamically configured adapter components processing multi-source heterogeneous data is specifically as follows: the adapter component can automatically adapt to different types of data sources without human intervention, such as through programmable interfaces or predetermined configuration files, allowing adaptive loading of rules and algorithms for processing different data formats. When receiving data from different sources, the adapter component first identifies and determines the data type, source, and format. This process involves checking the structure of the data, such as whether the data is a key-value pair, an array, or a nested object. The adapter component then applies one or more preset or dynamically generated rules to convert the data, which define how to map the source data to the target format, such as uniformly transforming a date representation method to a standard format, or performing unit conversion. The processed data is converted into a unified format so that other components in the database system can interact with it more easily, abandoning the specificity of each data source and maintaining the consistency and interoperability of the system. The data structure and content format analysis of the multi-source heterogeneous data processing automation tool is specifically as follows: the automation tool uses text parsing and pattern recognition technology to first read the data sample and then infer the structural characteristics of the data, such as whether the data contains lists, dictionaries, or other data structures. Content format analysis involves identifying specific data patterns, such as the format of dates, the representation of numbers, etc. It identifies specific elements in the data by comparing data samples with a series of known data patterns. The tool uses these analysis results to build a set of intermediate layer representations (i.e. intermediate formats), which provide the basis for the final data conversion. When encountering data formats that cannot be automatically matched, the tool can provide prompts and require manual intervention or learning new data patterns. The specific method of pre-trained machine learning models to identify data features and generate adaptation rules is as follows: pre-trained machine learning models have been trained on a large number of historical data sets and have learned the ability to identify various data features. For example, the model can distinguish between text, numbers, timestamps, JSON objects, etc. When new data arrives, these models will apply their learning results to analyze the data and find patterns and regularities in the data, such as the model can identify the sensor number or measurement value contained in the data. Once the features are identified, the model will automatically generate adaptation rules based on this information. These rules may include data conversion instructions, such as certain field values ​​need to be converted to uppercase, certain values ​​need to be converted to international units, etc. These adaptation rules are then saved and applied to the incoming data. The model continues to monitor the effectiveness of the adaptation and adjusts the rules based on new data samples to ensure continuous adaptation to changes and unknown patterns in the data source.

[0081] To make it easier to understand this solution, let's take an example: In the aerospace field, monitoring the rocket launch process involves hundreds of different sensors, which may output data using a variety of communication protocols and data formats, such as some using XML format, while others may be JSON or simple text files. The pre-trained machine learning model automatically analyzes these data as they are received and creates adaptation rules to standardize the data format so that it can meet the requirements of the central database system, so that all data looks as structured and consistent as if it came from a single source. As data continues to flow in, the system can discover new data patterns or anomalies through machine learning and update the adaptation rules in real time to ensure that all data is correctly understood and processed.

[0082] It should also be noted that multi-source heterogeneous data refers to data from different input sources with different formats and structures. "Multi-source" means that the data comes from multiple different places, such as different sensors, instruments or databases, and "heterogeneous" means that the data are different in format and structure. For example, a temperature sensor may output data in plain text format, while a pressure sensor may output structured JSON data. A dynamically configured adapter component refers to a software component or module that can automatically adjust processing rules according to different types of input data without manual settings. The adapter component can adapt to new data types without manual intervention to update the configuration each time. The data structure in data structure and content format analysis refers to the organization of data, such as lists, dictionaries, tables, etc. Content format analysis refers to determining the representation of data content, such as date format, number display method (such as whether there is a decimal point, what currency symbol to use, etc.).

[0083] In some embodiments, the distributed cleaning engine includes a dynamic response system for data volume, and the steps of transmitting the adapted data to the distributed cleaning engine, and the distributed cleaning engine performing error detection, pattern recognition, and logical consistency correction on the adapted data to obtain a cleaned data set include:

[0084] Adjust computing resource allocation based on data traffic;

[0085] Real-time monitoring of various data anomalies and distribution to corresponding nodes for parallel processing;

[0086] Summarize the parallel processing results and integrate them to generate the final cleaned data set;

[0087] After data processing is completed, computing resources are automatically reduced.

[0088] The technical solution provided in this embodiment is mainly implemented by performing error detection, pattern recognition and logical consistency correction on the adapted data to obtain the process of cleaning the data set. In detail, first, a dynamic response system is designed through a distributed cleaning engine to adjust the allocation of computing resources according to the changes in real-time data flow. In the application scenario of aerospace discrete manufacturing, this means that if the spacecraft sensor data we receive suddenly increases, such as in a launch countdown or emergency, the dynamic response system can instantly add more computing resources to ensure that all data can be processed quickly and accurately. Next, the distributed cleaning engine implements real-time monitoring of abnormal conditions of various types of data. This includes problems such as data interruption, out-of-bounds values, and format anomalies. Once abnormal data is detected, the system distributes it to a dedicated processing node for parallel processing. For example, if the temperature data measured from a certain aerospace engine exceeds a preset threshold, the data will be marked as abnormal and sent to the appropriate cleaning engine node for further analysis and processing. The results of the parallel processing will then be aggregated and integrated to form a complete cleaning data set. In this step, it may be necessary to merge the processing results from different nodes and resolve possible data conflicts to ensure the consistency and accuracy of the entire data set. For example, the same abnormal data may be detected and attempted to be cleaned by multiple nodes, and the system needs to coordinate these nodes to ensure that the results of data processing are unified and non-contradictory. Finally, after the data processing is completed, the dynamic response system of the distributed cleaning engine will automatically reduce the computing resources. This means that when the data processing demand decreases, the system can reduce resource allocation on demand, improve efficiency, and save resources. For example, during the stable flight phase of the spacecraft, the sensor data becomes stable and less abnormal, and the cleaning engine can reduce the computing resources used and use them for other needs. By dynamically adjusting the computing resources, the system can adapt to the data processing needs of different stages, as well as the changing data volume and abnormal patterns, ensuring that the multi-source data preprocessing system of the aerospace discrete manufacturing database can process data efficiently and reliably under various operating conditions.

[0089] In some embodiments, the step of calculating the data quality index of the cleaned data set by the quality assessment module includes:

[0090] Set different categories of quality control indicators and assign corresponding monitoring logic to each indicator;

[0091] After data cleaning, check each quality control indicator item by item and record its status;

[0092] Integrate all indicator statuses and calculate the data quality index;

[0093] Generate a quality assessment report for reference and further processing by system administrators or data cleaning engines.

[0094] In this embodiment, a quality assessment module is set to perform a series of quality control processes on the cleaned data set and calculate the data quality index. In detail, setting quality control indicators and monitoring logic In aerospace discrete manufacturing, the quality of data is the key to ensuring the reliability of the manufacturing process and product performance. Therefore, a series of quality control indicators need to be set for different types of data. These indicators may include data integrity, accuracy, consistency and timeliness. For each indicator, the quality assessment module needs to write and assign corresponding monitoring logic. For example, for temperature data, the monitoring logic may include checking whether the temperature value is within the normal operating range and whether the data points are continuously recorded at a predetermined interval. After the data cleaning process is completed, the assessment module performs checks on each preset quality control indicator. This includes applying predefined monitoring logic to each data item, checking the data, and recording the status of each indicator. For example, the system will verify that all temperature data points from engine monitoring do not exceed the limit value, while ensuring that the time interval of data sampling meets the predetermined specification. Calculating data quality index Each quality control indicator generates a status value. By comprehensively evaluating these status values, the quality assessment module calculates the comprehensive data quality index of the entire data set. This index provides a quantitative measure of data quality, which can be a comprehensive score or grade, such as a 1 to 10 scale, or a percentage standard. Generate a quality assessment report After all quality control indicators are checked, the quality assessment module generates a detailed quality assessment report. This report not only lists the status of each indicator and the final data quality index, but also includes an overall assessment of the quality of the data set, as well as suggestions for areas where the data processing program may need to be improved. This report will be used as a reference for system administrators or data cleaning engines to determine whether the data cleaning process needs to be optimized or adjusted. Through these implementation steps, the quality assessment module can improve the management capabilities of data quality control, ensure the high-quality data standards required for aerospace discrete manufacturing, and help ensure the safety and reliability of the entire production process. At the same time, the detailed records and reports generated during the quality assessment process can also play a certain auxiliary role in the continuous improvement of the data processing process.

[0095] In some embodiments, the calculation of the data quality index needs to consider factors including: the relevance of the data item to the business function and its influence on the business process; the strength of association and logical consistency between the data items in the data set; the matching degree between the frequency of data update and freshness and the demand for real-time response of the business. In detail, the implementation method of the relevance evaluation of the data item and the business function involves quantitatively evaluating the association between each data item and the aerospace discrete manufacturing business function. This process needs to start from the perspective of the business process and identify which data has a direct impact on the business process. For example, if the data is closely related to the security monitoring function, its relevance score is high; if the data is not clearly related to the process, the relevance score is low. The impact assessment needs to analyze the contribution of the data item to the decision-making and business operation, such as navigation data that directly contributes to the navigation path optimization decision will have a higher impact score. The analysis of the strength of association and logical consistency in the data set quantifies the logical relationship and strength of association between the data items in the data set. In the aerospace field, data should generally show certain expected patterns, such as temperature and pressure readings collected in the same time period should have certain physical correlations. Statistical methods such as correlation analysis and regression analysis can be used to evaluate these associations. Data that does not meet expected associations, such as temperature changes that do not meet expectations, will reduce the overall quality score of the data set. The evaluation of data update frequency and freshness examines whether the frequency of data updates meets the real-time requirements of aerospace discrete manufacturing business. For businesses that require highly real-time responses, such as launch countdowns or fault response in emergency scenarios, the data update frequency should be fast enough to ensure that decision makers receive the latest information. The calculation of the data quality index will use update frequency and freshness as an important parameter. If data items that have a greater impact on business processes are not updated in a timely manner, their DQI will be affected.

[0096] Furthermore, the calculation formula of the data quality index is:

[0097]

[0098] Among them, W i represents the weight factor of the i-th data item, which may be related to the manufacturing link relevance of the data item, the impact on system performance, or the cost of change; V i stands for data validation coefficient, which indicates the degree to which the data passes a series of validation procedures; Q i is the single quality score of the i-th data item, which measures the accuracy, completeness and other attributes of the data; D i It is a data dependency indicator, indicating the correlation between data items or the intensity of their dependency in the business process; i It is a data timeliness indicator, reflecting the freshness of the data or its closeness to the current operation time; E irepresents the environmental factor score associated with the ith data item, which may include data generation time, signal strength, signal-to-noise ratio, or other important environmental factors in aerospace discrete manufacturing; R i It is the score of the degree to which the data item complies with the established business rules; α i and β i is the adjustment coefficient, which is adjusted according to the characteristics of different data items and the requirements of business scenarios; Z is the normalization factor, which ensures that the CQI remains within a reasonable range; f() represents a mixed nonlinear function that combines multiple variables to comprehensively obtain the quality score of each data item; e is the base of the natural logarithm.

[0099] To facilitate a better understanding of the formula, the following example is given based on the formula:

[0100] About the weight factor W i , if a data item is the engine temperature monitoring data, due to its importance to flight safety, its weight factor W i It may be set higher, such as 0.9. It should be noted that the weight factor ranges from 0 to 1. The weight factor can be determined through expert evaluation and historical data analysis, or it may be set based on the predictive value of the data in previous failure cases.

[0101] About the data verification coefficient V i If a specific data item passes multiple verification steps, such as range checking, trend analysis, and historical data comparison, the data verification coefficient may be higher, such as 0.95. The data verification coefficient is determined based on whether the data item passes a series of designed verification processes. The more verifications and the more complex they are, the higher the coefficient.

[0102] About single quality score Q i , a data item such as acceleration data, which was recorded with high accuracy and no missing values ​​in the last flight test, may receive a high quality score, such as 85 points (assuming the score is between 0 and 100). The quality score of a single item can be determined by comparing the data item's record with the expected value or historical average, and checking its accuracy, number of interruptions, and other attributes.

[0103] About the data dependency index D i ,If there are two sets of data items, such as spacecraft temperature and pressure,that must physically maintain a certain correlation, and the actual data match is high, Di will get a higher score.,Data dependency indicators can be used to evaluate the degree of dependency,of data items through correlation analysis or causal models.

[0104] About the data timeliness index T i,Consider a data item that must be updated once a second. If the actual situation almost meets this, its timeliness index will get a high score.,The data timeliness index shell determines the timeliness of the update by comparing the gap between the timestamp of data generation and the processing or use time.

[0105] About Environmental Factor Score E i For a data item, if during the data collection period, environmental conditions such as temperature and vibration are within acceptable ranges, then E i Higher. i The environmental conditions information recorded during data production can be used, such as the instrument's operating temperature, vibration level, electromagnetic interference, etc.

[0106] Business rule compliance score R i If the flight data fully complies with the pre-defined flight mission parameters, its business rule consistency score R i Will be higher. i It can be obtained by checking whether the data items comply with preset business rules and logic.

[0107] About the adjustment factor α i , β i and normalization factor Z, depending on the context and needs of the specific implementation. For example: adjustment factor α i and β i , if the anomaly of a data item in the data set has a greater impact on the final data quality index, then the corresponding α i and β i It may be adjusted higher. i and β i It can be obtained through model training or experimental parameter tuning according to the characteristics of the data items and business scenario requirements. Normalization factor Z, if the calculated range of the comprehensive score exceeds the set maximum and minimum limits, Z will be used to scale the score to a reasonable range. The normalization factor can be determined based on the maximum and minimum values ​​of the calculated score.

[0108] In some embodiments, the step of dynamically adjusting the cleaning rules based on the data quality index includes:

[0109] Conduct real-time data mining to identify possible abnormal patterns and potential data errors;

[0110] Based on the mining results, analyze the effectiveness of the rules in the cleaning process and their scope of application;

[0111] Generate a rule adjustment plan and verify the validity of the rule adjustment plan;

[0112] After determining that the rule adjustment solution is feasible, the existing cleaning rule base is updated and redeployed to the distributed cleaning engine.

[0113] In this embodiment, in order to achieve real-time data mining, the system will continuously monitor the data stream and use specific data mining algorithms, such as clustering, outlier detection, association rule mining, etc., to identify abnormal patterns and potential errors that may exist in the data. For example, a sudden change in temperature reading or a data combination that does not conform to physical laws may be marked as anomalies. Based on the results of data mining, the system will evaluate the effectiveness of existing data cleaning rules. This may include analyzing the performance of the rules on different types of data sets and whether they can effectively identify and correct errors in the data. Through this analysis process, the scope and limitations of each rule can be evaluated. The system will formulate a recommended plan for rule adjustment based on the analysis results. The adjustment plan may include modifying the parameters of existing rules, adding new cleaning rules, or removing rules that do not work well. For example, if a rule fails to accurately identify the latest abnormal pattern, it may be proposed to adjust the algorithm or parameters of the rule. Before implementing a new rule, the effectiveness of the adjustment plan needs to be verified. This is usually done through a test data set, including repeatedly applying the improved rules, evaluating their actual impact on data quality, and ensuring that the new rules can be executed correctly and do not produce unexpected side effects. Once the rule adjustment plan is verified to be effective and feasible, the system will update the existing cleaning rule base. Updates include overwriting old rules, adding new rules, or deleting invalid rules. The updated rule base is then redeployed to the distributed cleaning engine to implement data cleaning tasks. Redeployment involves using automated deployment tools to synchronously update the rule base on different cleaning engine nodes so that the new rules can take effect immediately without interrupting the data processing flow. Through this series of steps, the database system of the R&D platform will be able to dynamically adjust the data cleaning rules to cope with changes in the data environment, thereby maintaining the flexibility and effectiveness of the cleaning process and ensuring that the quality of the processed data can meet the high standards of aerospace discrete manufacturing.

[0114] In some embodiments, the steps performed by the quality feedback control mechanism include:

[0115] Set up a listening program on the cleaned data set to detect and record quality change events;

[0116] Automatically generate quality problem reports based on detected events and analyze possible causes against business rules;

[0117] Use statistical methods to predict the potential impact of quality events on subsequent data processing cycles;

[0118] Provide data cleaning and maintenance recommendations based on predictive analysis to guide subsequent operations.

[0119] Furthermore, the step of providing data tracing and correction guidance according to the requirements and feedback of upper-layer applications and continuously optimizing data quality includes:

[0120] After data cleaning is completed, collect meta-information related to the cleaned data, including data source, cleaning time, etc.

[0121] Establish data metadata index and query interface to facilitate upper-level applications to conduct data traceability query;

[0122] Receive feedback from upper-level applications and perform data tracing and analysis based on issues identified in the feedback;

[0123] Based on the results of the retrospective analysis, make necessary corrections and updates to the data source or cleaning rules.

[0124] In this embodiment, a listening program is deployed on the cleaned data set so that the program can continuously monitor the data and detect quality change events. For example, when a newly added data batch causes the overall quality index of the data set to change beyond a preset threshold, the listening program will identify and record this event. When the listening program detects a quality change event, the system automatically generates a quality problem report. The report will include the details of the event and analyze the possible causes in combination with relevant business rules, such as abnormal data sources or improper cleaning rules. Use statistical methods to analyze and predict the impact of quality change events on subsequent data processing cycles. By statistically analyzing the data of historical quality events, the model may warn of possible quality degradation in advance, allowing preventive measures to be taken. Based on the results of the predictive analysis, guidance is provided to the data cleaning and maintenance team, such as adjusting the data cleaning algorithm or improving the data monitoring mechanism for foreseen problems. After completing the data cleaning, the system will collect and record the meta-information generated during the cleaning process, such as the source of data, cleaning time, rules and methods used for cleaning, etc. The system needs to build a mechanism that allows upper-level applications to easily trace data through indexing and query interfaces. This can be achieved through a user interface where users can enter query requests and retrieve relevant meta-information. The platform receives feedback from upper-level applications. For example, when a space instrument gives unexpected results, the system can reversely query the data and analyze the possible problems. Once the problem is traced and identified, the system administrator or relevant personnel can correct the data source or adjust the cleaning rules and workflow based on the problem found.

[0125] Through the steps of the above embodiment, the R&D platform database system can realize a closed-loop data quality management and maintenance mechanism. This mechanism will monitor data quality, automatically propose improvement suggestions, respond to upper-layer application feedback, and continuously optimize the data processing process to ensure that the data in the database system is constantly maintained in a high-quality state to meet the strict requirements of aerospace discrete manufacturing.

[0126] like Figure 3 As shown, Figure 3 A schematic diagram of the structure of a multi-source data preprocessing system for aerospace discrete manufacturing platform provided in an embodiment of the present application, the system includes:

[0127] An adapter component 10 that can be dynamically configured to adapt to specific data source characteristics, including data format and structure, and can achieve data unification and standardization;

[0128] A distributed cleaning engine 20, communicating with the adapter component to receive the adapted data, the distributed cleaning engine is used to perform error detection, pattern recognition and logic consistency correction of the data, and supports parallel processing of the data;

[0129] A quality assessment component 30, for calculating a cleaning quality index of data processed by the distributed cleaning engine;

[0130] A cleaning rule optimization module 40 dynamically adjusts the cleaning rules based on the cleaning quality index obtained from the quality assessment component and can predict future data cleaning needs based on feedback data;

[0131] The quality feedback control system 50 can provide data traceability and correction guidance based on the needs and feedback from upstream systems to optimize data quality and provide insights into cleaning operations.

[0132] The adapter component 10 is designed as a highly modular and configurable software component, allowing it to dynamically adapt to various types of data sources. These adapters can identify and convert multiple data formats and structures, achieve data unification and standardized processing, and create a consistent working foundation for subsequent operations.

[0133] The Distributed Cleaning Engine 20 is the core processing platform of the system, which interfaces with the adapter component to obtain the adapted data. It has high-performance data processing capabilities and can perform error detection, pattern recognition, and logic consistency correction. By utilizing parallel processing technology, the engine can process large amounts of data simultaneously at different nodes, improving cleaning efficiency and accuracy.

[0134] The quality assessment component 30 (Quality Assessment Component) is a module for real-time evaluation of the quality of data cleaning results. It uses advanced algorithms and models to evaluate data quality and generates a quality index to measure whether the cleaned data meets the specified standards.

[0135] The cleaning rules optimization module 40 dynamically adjusts the existing data cleaning rules based on the quality index provided by the quality assessment component. Based on the system feedback, the module can perform predictive analysis to infer the future data cleaning needs and improve the cleaning rules accordingly.

[0136] The Quality Feedback Control System 50 involves obtaining requirements and feedback from the upper-level business flow system and providing guidance on data traceability and correction. These guidance can optimize the quality of data processing and provide insights and optimization suggestions on data cleaning operations to the operation and maintenance team.

[0137] The implementation of the entire system starts with the adaptation of different data sources by the adapter component, improving the efficiency and quality of data processing through the distributed cleaning engine, then using the quality assessment component for quality inspection, and dynamically adjusting the cleaning strategy through the cleaning rule optimization module, and finally integrating upstream feedback through the quality feedback control system to form a closed-loop, self-optimizing, and efficient data processing platform. Such a platform can ensure the high standard quality requirements for the generation and use of data in the aerospace discrete manufacturing process, and ensure the accuracy and effectiveness of manufacturing and R&D activities.

[0138] Corresponding to the embodiments of the solution part above, an embodiment of the present invention further provides a computer device. The computer device described below and the multi-source data preprocessing method for aerospace discrete manufacturing platform described above can refer to each other.

[0139] The computer equipment includes:

[0140] Memory for storing computer programs;

[0141] The processor is used to implement the steps of the multi-source data preprocessing method of aerospace discrete manufacturing database of the above method embodiment when executing the computer program:

[0142] S1. Adapting and processing input multi-source heterogeneous data through at least one dynamically configured adapter component to unify and standardize the data format and obtain adapted data;

[0143] S2, transmitting the adapted data to a distributed cleaning engine, wherein the distributed cleaning engine performs error detection, pattern recognition and logic consistency correction on the adapted data to obtain a cleaned data set;

[0144] S3, calculating a data quality index for the cleaned data set through a quality assessment module;

[0145] S4. Dynamically adjust cleaning rules based on the data quality index;

[0146] S5. Through the preset quality feedback control mechanism, data traceability and correction guidance are provided according to the needs and feedback of upper-level applications to continuously optimize data quality.

[0147] Figure 4 A schematic diagram of the structure of a computer device provided in an embodiment of the present application, which may be a computing device with data processing capabilities.

[0148] The device may include: a processor 301 and a memory 302 .

[0149] The memory 302 is used to store programs, and the processor 301 calls the programs stored in the memory 302 to execute the above method embodiment. The specific implementation method and technical effect are similar and will not be repeated here.

[0150] The memory 302 stores program codes, and when the program codes are executed by the processor 301, the processor 301 executes various steps of the method according to various exemplary embodiments of the present application described in the above “Exemplary Method” section of this specification.

[0151] The processor 301 may be a general-purpose processor, such as a central processing unit (CPU), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, and may implement or execute the methods, steps and logic block diagrams disclosed in the embodiments of the present application. A general-purpose processor may be a microprocessor or any conventional processor, etc. The steps of the method disclosed in the embodiments of the present application may be directly embodied as being executed by a hardware processor, or may be executed by a combination of hardware and software modules in the processor.

[0152] As a non-volatile storage medium, the memory 302 can be used to store non-volatile software programs, non-volatile computer executable programs and modules. The memory may include at least one type of storage medium, such as flash memory, hard disk, multimedia card, card-type memory, random access memory (R andom Access Memory, RAM), static random access memory (Static Random Access Memory, SRAM), programmable read-only memory (ProgrammableRead Only Memory, PROM), read-only memory (Read Only Memory, ROM), electrically erasable programmable read-only memory (Elect rically Erasable Programmable Read-Only Memory, EEPROM), magnetic memory, disk, optical disk, etc. The memory is any other medium that can be used to carry or store the desired program code in the form of an instruction or data structure and can be accessed by a computer, but is not limited thereto. The memory 302 in the embodiment of the present application can also be a circuit or any other device that can realize a storage function, for storing program instructions and / or data.

[0153] Optionally, the present application also provides a program product, such as a storage medium, including a program, which is used to execute the above method embodiment when executed by a processor.

[0154] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0155] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0156] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0157] The above-mentioned integrated unit implemented in the form of a software functional unit can be stored in a computer-readable storage medium. The above-mentioned software functional unit is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor (English: processor) to perform some steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (English: Read-Only Memo ry, abbreviated: ROM), random access memory (English: Random Access Memory, abbreviated: RAM), disk or optical disk and other media that can store program codes.

Claims

1. A multi-source data preprocessing method for aerospace discrete manufacturing database, characterized in that: Solutions include: Adapting and processing the input multi-source heterogeneous data through at least one dynamically configured adapter component to unify and standardize the data format and obtain adapted data; The adapted data is transmitted to a distributed cleaning engine, and the distributed cleaning engine performs error detection, pattern recognition and logic consistency correction on the adapted data to obtain a cleaned data set; Calculating a data quality index for the cleaned data set by a quality assessment module; Dynamically adjust cleaning rules based on the data quality index; Through the preset quality feedback control mechanism, data traceability and correction guidance are provided according to the needs and feedback of upper-level applications to continuously optimize data quality; It is characterized in that the calculation formula of the data quality index is: Among them, W i represents the weight factor of the i-th data item, which may be related to the manufacturing link relevance of the data item, the impact on system performance, or the cost of change; V i stands for data validation coefficient, which indicates the degree to which the data passes a series of validation procedures; Q i is the single quality score of the i-th data item, which measures the accuracy and completeness of the data; D i It is a data dependency indicator, indicating the correlation between data items or the intensity of their dependency in the business process; i It is a data timeliness indicator, reflecting the freshness of the data or its closeness to the current operation time; E i represents the environmental factor score associated with the ith data item, which may include data generation time, signal strength, signal-to-noise ratio, or other important environmental factors in aerospace discrete manufacturing; R i It is the score of the degree to which the data item complies with the established business rules; α i and β i is the adjustment coefficient, which is adjusted according to the characteristics of different data items and the requirements of business scenarios; Z is the normalization factor, which ensures that the CQI remains within a reasonable range; f() represents a mixed nonlinear function that combines multiple variables to comprehensively obtain the quality score of each data item; e is the base of the natural logarithm.

2. The multi-source data preprocessing method of aerospace discrete manufacturing database according to claim 1 is characterized in that: The step of adapting and processing the input multi-source heterogeneous data through at least one dynamically configured adapter component to unify and standardize the data format and obtain adapted data includes: Automatically analyzing the data structure and content format of the received multi-source heterogeneous data; Use pre-trained machine learning models to identify data features and generate adaptation rules; Applying the generated adaptation rules to the multi-source heterogeneous data to convert them into the adapted data, wherein the adapted data has a unified format recognizable by the database system; During the adaptation process, new data patterns are continuously learned to achieve real-time updating and optimization of adaptation rules.

3. The multi-source data preprocessing method of aerospace discrete manufacturing database according to claim 1 is characterized in that: The distributed cleaning engine includes a dynamic response system for data volume, and the steps of transmitting the adapted data to the distributed cleaning engine, and the distributed cleaning engine performing error detection, pattern recognition and logical consistency correction on the adapted data to obtain a cleaned data set include: Adjust computing resource allocation based on data traffic; Real-time monitoring of various data anomalies and distribution to corresponding nodes for parallel processing; Summarize the parallel processing results and integrate them to generate the final cleaned data set; After data processing is completed, computing resources are automatically reduced.

4. The multi-source data preprocessing method of aerospace discrete manufacturing database according to claim 1 is characterized in that: The step of calculating the data quality index of the cleaned data set by the quality assessment module includes: Set different categories of quality control indicators and assign corresponding monitoring logic to each indicator; After data cleaning, check each quality control indicator item by item and record its status; Integrate all indicator statuses and calculate the data quality index; Generate a quality assessment report for reference and further processing by system administrators or data cleaning engines.

5. The multi-source data preprocessing method of aerospace discrete manufacturing database according to any one of claims 1 to 4, characterized in that: The factors to be considered in calculating the data quality index include: The relevance of data items to business functions and their impact on business processes; the strength of association and logical consistency between data items in the data set; the matching degree between the frequency of data updates and freshness and the real-time response requirements of the business.

6. The multi-source data preprocessing method of aerospace discrete manufacturing database according to claim 5 is characterized in that: The step of dynamically adjusting the cleaning rules based on the data quality index includes: Conduct real-time data mining to identify possible abnormal patterns and potential data errors; Based on the mining results, analyze the effectiveness of the rules in the cleaning process and their scope of application; Generate a rule adjustment plan and verify the validity of the rule adjustment plan; After determining that the rule adjustment solution is feasible, the existing cleaning rule base is updated and redeployed to the distributed cleaning engine.

7. The multi-source data preprocessing method of aerospace discrete manufacturing database according to claim 5 is characterized in that: The steps performed by the quality feedback control mechanism include: Set up a listening program on the cleaned data set to detect and record quality change events; Automatically generate quality problem reports based on detected events and analyze possible causes against business rules; Use statistical methods to predict the potential impact of quality events on subsequent data processing cycles; Provide data cleaning and maintenance recommendations based on predictive analysis to guide subsequent operations.

8. The multi-source data preprocessing method of aerospace discrete manufacturing database according to claim 7 is characterized in that: The steps of providing data traceability and correction guidance based on the needs and feedback of upper-level applications and continuously optimizing data quality include: After data cleaning is completed, metadata related to the cleaned data is collected, including data source and cleaning time; Establish data metadata index and query interface to facilitate data traceability query by upper-layer applications; Receive feedback from upper-level applications and perform data tracing and analysis based on issues identified in the feedback; According to the retrospective analysis results, the data source or cleaning rules are corrected and updated.

Images