An engineering project multi-source data collection verification and storage backtracking method and system

Abstract

The application discloses an engineering project multi-source data collection, verification, storage and backtracking method and system, relates to the technical field of data processing, and comprises the following steps: collecting multi-source heterogeneous data through real-time synchronization and batch extraction, performing multiple verifications on the collected data, storing the verified data in a data warehouse to form a standardized data set after configuring a unique identification ID, simultaneously associating collection sources, cleaning rules and summary processing node information to the unique identification ID and storing the unique identification ID as a full-link flow log, and reversely searching and outputting original data sources in the full-link flow log; establishing a joint index in multiple dimensions, receiving a dimension combination condition, searching and outputting original detailed records, and aggregating and exporting complete data records by taking the unique identification ID as an associated primary key. The application realizes unified collection, quality verification, tracing and export of engineering project multi-source data through dual-mode collection, full-link logs and multi-dimensional indexes.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, specifically to a method and system for the acquisition, verification, storage, and backtracking of multi-source data in engineering projects. Background Technology

[0002] Currently, large group enterprises generally deploy multiple business information systems such as project management, contract management, financial management, and materials management in their engineering project management processes. These systems operate independently, accumulating massive amounts of heterogeneous data from multiple sources. Existing technologies include some data collection and verification solutions, such as using periodic extraction tools to unify data from various business systems into a data warehouse, or employing rule bases for field-level format verification of data. There are also knowledge graph-based engineering quality inspection data traceability solutions. However, these existing solutions still have the following shortcomings:

[0003] First, data collection often adopts a full-volume, timed extraction method, which cannot adaptively select the collection mode according to the actual change frequency of the data table. This results in insufficient timeliness of high-frequency data and waste of resources due to repeated extraction of low-frequency data. Second, verification methods are mostly limited to field-level verification, lacking a systematic hierarchical and progressive verification mechanism for primary key conflicts, format and null values. Data errors and omissions are difficult to identify automatically, and data consistency cannot be guaranteed.

[0004] Second, the data analysis dimensions are fixed, usually only supporting the generation of statistical reports according to preset dimensions, and it is not possible to freely combine and drill down between dimensions such as time, organization, project, and personnel, making it difficult to penetrate from the summary indicators to the original business details; existing traceability solutions are mostly for specific scenarios (such as construction quality traceability) and have not yet formed a general data traceability system covering the entire chain of collection, cleaning, and processing, making it impossible to quickly locate the cause after abnormal indicators appear.

[0005] Third, project benefit calculation, cost risk identification, and resource allocation optimization still rely on human experience judgment and lack systematic quantitative evaluation methods based on historical data. Project risks are mostly discovered after the fact, and decision-making lacks quantitative data support, making it difficult to meet the group's needs for refined and intelligent project management. Summary of the Invention

[0006] To address the aforementioned issues, this invention proposes a method and system for the collection, verification, storage, and backtracking of multi-source data in engineering projects. This method employs a dual-mode collection approach, adaptively switching between real-time synchronization and batch extraction based on data change frequency. It combines primary key conflict / format / null value triple verification and cleaning to form a standardized dataset. Furthermore, it utilizes unique identifiers (IDs) linked to end-to-end log flows, along with indicator coding for reverse tracing and multi-dimensional index queries based on time, organization, project, and personnel. This achieves high-quality standardized integration of multi-source heterogeneous engineering project data, accurate end-to-end tracing, and efficient multi-dimensional retrieval and utilization, thereby providing reliable data support for comprehensive quality assessment of engineering projects.

[0007] On the one hand, a method for collecting, verifying, storing, and tracing back multi-source data in engineering projects includes:

[0008] S1. Obtain the change frequency of each data table. Based on the change frequency, select real-time synchronization and batch extraction dual modes to collect multi-source heterogeneous data. Perform primary key conflict verification, format verification and null value verification on the collected multi-source heterogeneous data. Mark and isolate the data that fails the verification. Clean the data that passes the verification and store it in the data warehouse to form a standardized dataset.

[0009] S2, configure a unique identifier ID for each data point in the standardized dataset, and associate the data collection source, the cleaning rules executed, and the information of the aggregation and processing nodes that each data point passes through with the unique identifier ID, and store it as a full-link flow log;

[0010] S3 receives externally input indicator codes, performs reverse retrieval in the end-to-end flow log based on the indicator codes, locates the collection source, cleaning rules and summary processing nodes associated with the indicator codes, and outputs the retrieved original data source information.

[0011] S4: Based on the time field, organization field, project field and personnel field in the standardized dataset, a dimension combination query index is built. It receives the dimension combination conditions from external input, calls the dimension combination query index to retrieve data records that meet the dimension combination conditions from the standardized dataset, performs statistics on the retrieved data records according to their respective bound rules, and outputs the corresponding original detailed records.

[0012] S5 uses the unique identifier ID as the primary key to aggregate and associate the original data source information and original detail records belonging to the same data record, and exports the aggregated complete data record.

[0013] Furthermore, in S1, multi-source heterogeneous data is collected through a dual-mode approach of real-time synchronization and batch extraction, specifically including:

[0014] Obtain the data tables to be collected from the databases of each business system, and calculate the average number of change records for each data table within the past preset number of statistical periods;

[0015] Data tables with an average number of change records exceeding a preset threshold are defined as high-frequency change data tables. Data in high-frequency change data tables are classified into real-time synchronization mode, and incremental change data is captured in real time by listening to the database change log.

[0016] Data tables with an average number of change records lower than or equal to a preset threshold are defined as low-frequency batch data tables. Data in low-frequency batch data tables are categorized into batch extraction mode, and query requests with time range conditions are sent to each business system according to a preset time window to batch extract data.

[0017] Furthermore, in S1, primary key conflict checks, format checks, and null value checks are performed on the collected multi-source heterogeneous data, specifically including:

[0018] S11, iterate through each field of the collected data, determine whether the current field is allowed to be empty according to the preset field non-empty rule library, and mark the data record with the required field value being empty as an empty value exception;

[0019] S12, extract the source system identifier and data table identifier of the data that passed the null value check, load the corresponding field format template from the format rule library, compare the values ​​of each field in the data record with the format template, and mark the data that does not meet the format requirements as format abnormal;

[0020] S13, extract the primary key field value of the data record that has passed the format validation, divide the data to be validated into multiple primary key area segments according to the hash distribution of the primary key field value, sort the data records in each primary key area segment according to the primary key, compare the primary key values ​​of adjacent records after sorting, and mark the data records with the same primary key value as primary key conflict anomalies.

[0021] S14. Data records marked as null value exception, format exception, and primary key conflict exception are isolated and stored. Data records that pass all three types of checks are cleaned and stored in the data warehouse.

[0022] Furthermore, S1 also includes the process of marking and isolating data that fails verification and then repairing it, specifically including:

[0023] Data records marked as null value exceptions, format exceptions, and primary key conflict exceptions are isolated and stored in the exception data temporary storage area. Each exception record in the exception data temporary storage area is traversed, and repair processing is performed in the following manner:

[0024] For fields that fail to trigger validation, firstly, search for related records with the same primary key or the same business entity in the local related dimension table to complete the data; if no results are found in the local search, send an HTTP asynchronous request to the external data service platform for a global search and receive the callback results to complete the data; if the external search times out or has no valid results, push the data to the administrator for manual entry.

[0025] If any one of the repair levels succeeds, the repair of that field will be terminated and the operation log will be recorded. If all three levels fail, the repair of that field will be marked as failed.

[0026] After all abnormal fields have been processed, if any field fails to be repaired, the entire record will be re-isolated. If all fields are successfully repaired, primary key conflict, format and null value checks will be re-executed on the repaired record. If the checks pass, the record will be stored in the data warehouse; otherwise, it will be isolated again for the next round of repair.

[0027] Furthermore, in S3, the indicator encoding and the summary processing node have a preset mapping relationship, which is pre-configured in the system parameter table.

[0028] Furthermore, in S3, reverse retrieval is performed in the end-to-end flow log based on the indicator code. Specifically, the system parameter table is queried based on the externally input indicator code to locate the aggregation processing node bound to the indicator code in the system parameter table. The end-to-end flow log is retrieved based on the identifier of the aggregation processing node to locate all data records processed by the aggregation processing node, as well as the collection source and cleaning rules corresponding to each data record.

[0029] Furthermore, in S4, a dimensional combination query index is built based on the time field, organization field, project field, and personnel field in the standardized dataset, specifically including:

[0030] S41, traverse the table structure information of the standardized dataset, obtain the date field from the table structure information, and mark the date field as a time dimension field;

[0031] S42, retrieve the field names from the table structure information, including fields related to organization, department, and branch, and mark them as organization dimension fields;

[0032] S43, retrieve the fields whose names include keywords such as project, engineering, and section from the table structure information, and mark them as project dimension fields;

[0033] S44, retrieve the fields whose field names include personnel, manager, and person in charge from the table structure information, and mark them as personnel dimension fields;

[0034] S45. The marked time dimension field, organization dimension field, project dimension field and personnel dimension field are used as composite index columns. A composite index is built in ascending order of cardinality and stored in the index storage area.

[0035] Furthermore, the process of retrieving data records from the standardized dataset that satisfy the dimension combination conditions by calling the dimension combination query index also includes:

[0036] Monitor the retrieval response time of the index for combined monitoring dimensions;

[0037] When the retrieval response time exceeds the preset threshold, the filter fields not included in the composite index in the current query conditions are parsed, and the frequency of each filter field appearing as a condition in historical query records is counted.

[0038] Fields that appear more frequently than a preset frequency threshold will be marked as recommended custom peripheral dimensions and pushed to the user's device.

[0039] Receive user confirmation or modification of recommended custom peripheral dimensions, and generate effective custom peripheral dimension configurations;

[0040] Add the custom peripheral dimension to the composite index, recalculate the cardinality of all index columns, rearrange the column order of the composite index in ascending order of cardinality, and then store it in the index storage area.

[0041] Furthermore, following S5, it also includes, based on the aggregated complete data records, performing a comprehensive quality score on the project to be evaluated, specifically including:

[0042] S61, Obtain the project type, planned duration, and planned cost of the project to be evaluated;

[0043] S62, from the aggregated complete data records, filter historical project records that belong to the same project type as the project to be evaluated, and calculate the schedule deviation rate, cost deviation rate and resource utilization rate of each historical project;

[0044] S63, take the average of the schedule deviation rate, cost deviation rate and resource utilization rate of each historical project to obtain the schedule deviation benchmark value, cost deviation benchmark value and resource utilization benchmark value of this type of project;

[0045] S64. The planned duration and the baseline value of the duration deviation of the project to be evaluated are superimposed to obtain the predicted duration; the planned cost and the baseline value of the cost deviation of the project to be evaluated are superimposed to obtain the predicted cost.

[0046] S65 generates a comprehensive quality score for the project to be evaluated based on the predicted construction period, predicted cost, and resource utilization benchmarks.

[0047] On the other hand, a system for the acquisition, verification, storage, and traceability of multi-source data in engineering projects includes:

[0048] The data preprocessing module is used to obtain the change frequency of each data table, select real-time synchronization and batch extraction dual modes to collect multi-source heterogeneous data based on the change frequency, perform primary key conflict verification, format verification and null value verification on the collected multi-source heterogeneous data, mark and isolate the data that fails the verification, and clean the data that passes the verification and store it in the data warehouse to form a standardized dataset.

[0049] The end-to-end flow log generation module is used to configure a unique identifier ID for each piece of data in the standardized dataset, and associate the collection source, the cleaning rules executed, and the information of the aggregation and processing nodes that each piece of data passes through with the unique identifier ID, and store them as an end-to-end flow log;

[0050] The reverse retrieval module is used to receive externally input indicator codes, perform reverse retrieval in the end-to-end flow log based on the indicator codes, locate the collection source, cleaning rules and summary processing nodes associated with the indicator codes, and output the original data source information retrieved.

[0051] The original detailed record output module is used to build a dimension combination query index based on the time field, organization field, project field and personnel field in the standardized dataset. It receives external input dimension combination conditions, calls the dimension combination query index to retrieve data records that meet the dimension combination conditions from the standardized dataset, performs statistics on the retrieved data records according to their respective bound rules, and outputs the corresponding original detailed records.

[0052] A dimensional combination query index is built based on the time, organization, project, and personnel fields in the standardized dataset, specifically including:

[0053] Traverse the table structure information of the standardized dataset, extract the date fields from the table structure information, and mark the date fields as time dimension fields;

[0054] Extract field names from the table structure information, including those related to organization, department, and branch office, and mark them as organization dimension fields;

[0055] Retrieve fields from the table structure information whose names include keywords such as "project," "engineering," and "section," and mark them as project dimension fields.

[0056] Retrieve fields from the table structure information whose names include personnel, managers, and responsible persons, and mark them as personnel dimension fields;

[0057] The marked time dimension field, organization dimension field, project dimension field and personnel dimension field are used as composite index columns. A composite index is built in ascending order of cardinality and stored in the index storage area.

[0058] The data export module is used to aggregate and associate the original data source information and original detailed records belonging to the same data record using a unique identifier ID as the association primary key, and export the aggregated complete data record.

[0059] The present invention adopts the above technical solution and has the following beneficial effects:

[0060] (1) This invention collects data in two modes: real-time synchronization and batch extraction. It combines a three-layer progressive verification mechanism of primary key conflict verification, format verification and null value verification. Data that fails verification is marked and isolated, and data that passes verification is stored in a data warehouse to form a standardized dataset, thereby achieving unified collection and quality assurance of multi-source heterogeneous data.

[0061] (2) This invention configures a unique identifier ID for each piece of data and associates the collection source, cleaning rules and summary processing node information with the ID and stores them as a full-link flow log; the mapping relationship between the indicator code and the summary processing node is pre-configured in the system parameter table. When tracing the source, the system parameter table is queried according to the indicator code to locate the bound summary processing node, and the full-link flow log is retrieved with the node identifier to return all data records processed by the node and their collection source and cleaning rules. There is no need to maintain the binding relationship between the indicator and the source and cleaning rules, which reduces the number of table associations and realizes the rapid tracing and location of abnormal indicators.

[0062] (3) This invention constructs a joint index through four dimensions: time, organization, project, and personnel, which supports fast retrieval and statistical summary under any combination of dimensions. It also aggregates and associates the original data source information and original detailed records with the unique identifier ID as the association primary key, and exports complete data records, realizing flexible multi-dimensional query and full-link verifiable data export.

[0063] (4) This invention automatically matches and identifies field names by traversing the table structure information of the standardized dataset and using a preset keyword thesaurus. Date fields and fields whose names contain keywords such as organization, department, branch, project, engineering, section, personnel, manager, and person in charge are marked as four types of dimension fields: time, organization, project, and personnel. This achieves automatic identification of dimension fields with zero manual intervention. Then, the marked four types of fields are used as joint index columns. A joint index is constructed in order of cardinality from low to high and stored in an independent index storage area. This makes the index naturally cover multiple dimension combination query scenarios, supports drill-down analysis of summary indicators from any dimension combination and efficiently penetrates to the original detailed records. This significantly improves the query efficiency of multi-source heterogeneous engineering project data and effectively solves the technical problem of solidified multi-dimensional analysis dimensions and inability to trace back to the original details from abnormal indicators in the prior art. Attached Figure Description

[0064] Figure 1 This is a flowchart of a method for collecting, verifying, storing, and tracing back multi-source data in an engineering project, according to an embodiment of the present invention.

[0065] Figure 2 This is a schematic diagram illustrating the acquisition of multi-source data for an engineering project according to an embodiment of the present invention;

[0066] Figure 3 This is a diagram of a system for the acquisition, verification, storage, and backtracking of multi-source data in an engineering project, according to an embodiment of the present invention. Detailed Implementation

[0067] The present invention will be further described in detail below with reference to the embodiments and accompanying drawings, but the embodiments of the present invention are not limited thereto.

[0068] like Figure 1 As shown, the present invention provides a method for the acquisition, verification, storage, and backtracking of multi-source data in engineering projects, comprising:

[0069] S1 collects multi-source heterogeneous data through a dual-mode approach of real-time synchronization and batch extraction (i.e., obtains the change frequency of each data table, and selects the dual-mode approach of real-time synchronization and batch extraction to collect multi-source heterogeneous data based on the change frequency). It performs primary key conflict verification, format verification, and null value verification on the collected multi-source heterogeneous data, marks and isolates the data that fails the verification, and cleans and stores the data that passes the verification into the data warehouse to form a standardized dataset.

[0070] Specifically, multi-source heterogeneous data is collected through a dual-mode approach of real-time synchronization and batch extraction, including:

[0071] Obtain the data tables to be collected from the databases of each business system, and calculate the average number of change records for each data table within the past preset number of statistical periods (default is 3 statistical periods);

[0072] Data tables with an average number of change records exceeding a preset threshold are defined as high-frequency change data tables. Data in high-frequency change data tables are classified into real-time synchronization mode, and incremental change data is captured in real time by listening to the database change log.

[0073] Data tables with an average number of change records lower than or equal to a preset threshold are defined as low-frequency batch data tables. Data in low-frequency batch data tables are categorized into batch extraction mode, and query requests with time range conditions are sent to each business system according to a preset time window to batch extract data.

[0074] Specifically, the collected multi-source heterogeneous data undergoes primary key conflict verification, format verification, and null value verification, including:

[0075] S11, iterate through each field of the collected data, determine whether the current field is allowed to be empty according to the preset field non-empty rule library, and mark the data record with the required field value being empty as an empty value exception;

[0076] S12, extract the source system identifier and data table identifier of the data that passed the null value check, load the corresponding field format template from the format rule library, compare the values ​​of each field in the data record with the format template, and mark the data that does not meet the format requirements as format abnormal;

[0077] S13, extract the primary key field value of the data record that has passed the format validation, divide the data to be validated into multiple primary key area segments according to the hash distribution of the primary key field value, sort the data records in each primary key area segment according to the primary key, compare the primary key values ​​of adjacent records after sorting, and mark the data records with the same primary key value as primary key conflict anomalies.

[0078] S14. Data records marked as null value exception, format exception, and primary key conflict exception are isolated and stored. Data records that pass all three types of checks are cleaned and stored in the data warehouse.

[0079] Specifically, S1 also includes marking and isolating data that fails verification and then repairing it, specifically including:

[0080] Data records marked as null value exceptions, format exceptions, and primary key conflict exceptions are isolated and stored in the exception data temporary storage area. Each exception record in the exception data temporary storage area is traversed, and repair processing is performed in the following manner:

[0081] For fields that fail to trigger validation, firstly, search for related records with the same primary key or the same business entity in the local related dimension table to complete the data; if no results are found in the local search, send an HTTP asynchronous request to the external data service platform for a global search and receive the callback results to complete the data; if the external search times out or has no valid results, push the data to the administrator for manual entry.

[0082] If any one of the repair levels succeeds, the repair of that field will be terminated and the operation log will be recorded. If all three levels fail, the repair of that field will be marked as failed.

[0083] After all abnormal fields have been processed, if any field fails to be repaired, the entire record will be re-isolated. If all fields are successfully repaired, primary key conflict, format and null value checks will be re-executed on the repaired record. If the checks pass, the record will be stored in the data warehouse; otherwise, it will be isolated again for the next round of repair.

[0084] Specifically, in the context of this application, the local relational dimension table refers to the pre-stored basic reference data tables in the data warehouse, such as projects, organizations, personnel, and contracts. These tables have relatively stable data quality and complete field information, serving as a reference source for repairing abnormal data. When a record is isolated due to null values ​​or formatting issues, the repair program first identifies the fields that triggered the validation failure. Simultaneously, it checks if there are other related fields with normal values ​​in the same record, such as project number, organization code, or personnel number. Then, based on the mapping relationship between fields, it uses the values ​​of these normal fields to match records with the same primary key or business entity in the local relational dimension table. If a match is successful, the corresponding field value is directly retrieved and filled into the abnormal record.

[0085] For example, a project schedule record might be isolated because the responsible person field is null, but the project number in the record is valid. The fix program can use this project number to find the same project in the project dimension table, and then retrieve the value of the responsible person field corresponding to that project and fill it back into the responsible person field of the abnormal record. The entire process records the source of the completion and the values ​​before and after, and the completed field will enter the subsequent re-validation stage.

[0086] Specifically, in this embodiment, the repair process is divided into three levels of repair:

[0087] Level 1 Repair: For fields in abnormal records that trigger validation failures, parse their field names and the data tables they belong to. Based on preset field association mapping rules, search the related dimension tables in the local data warehouse for related records that belong to the same primary key or the same business entity identifier as the field. If a corresponding related record is found, extract the corresponding field value from the related record and supplement it to the corresponding field in the abnormal record, mark the field as successfully repaired, and record the repair operation log. If no corresponding record is found in the local association table, proceed to Level 2 Repair.

[0088] Level 2 Repair: Based on the field name and exception type that triggered the verification failure in the exception record, a data completion request containing the field identifier, the name of the data table to which it belongs, and the current value of the exception field is constructed. This completion request is sent to a preset external data service platform via HTTP protocol. After receiving the request, the external data service platform performs a global asynchronous search in its multiple connected business systems. After the search is completed, the search results are returned via callback. After receiving the returned search results, the corresponding field value is extracted and added to the corresponding field in the exception record, the field is marked as successfully repaired, and the repair operation log is recorded. If no search results are received within the preset timeout period or no valid data is found in the search results, the process proceeds to Level 3 Repair.

[0089] Level 3 Repair: The abnormal records that have not been repaired after Level 1 and Level 2 repairs are pushed to the administrator. Through the manual entry interface provided by the administrator, the administrator manually checks and enters the abnormal fields based on business experience. After receiving the entry data returned by the administrator, the data is filled into the corresponding fields of the abnormal records, the field is marked as successfully repaired, and the repair operation log is recorded.

[0090] For abnormal records where all fields have been successfully repaired after the above-mentioned repairs, the primary key conflict check, format check, and null value check in S11 to S13 will be re-executed. Data records that pass the re-check will be stored in the data warehouse, and data records that still fail the re-check will be marked and isolated again and returned to the abnormal data temporary storage area to wait for the next round of repair processing.

[0091] Specifically, this embodiment also supports the construction of a multi-protocol general data access framework to complete the integration and intelligent quality verification of multi-source data from other heterogeneous business systems, forming a standardized and unified project dataset. The multi-protocol configuration includes three general interface protocols: HTTP, JSON, and WebService. It adopts a dual-mode approach of real-time synchronization and batch extraction to collect multi-source heterogeneous data from other systems, such as materials, projects, and finances. The end-to-end data latency of the real-time link is controlled within 5 seconds. An automated data verification rule base is built to perform intelligent verification for three types of anomalies: primary key conflicts, field format errors, and null values. The data consistency verification pass rate is ≥99.99%. The system automatically calculates the duplication rate, null value rate, and format error rate of statistics, generates a standardized data quality report, marks and isolates abnormal data, and stores the cleaned data in a unified data warehouse to form a standardized dataset.

[0092] Specifically, such as Figure 2 As shown, the data acquisition framework in this embodiment divides multi-source heterogeneous data into three categories—A, B, and C—based on the frequency of data changes, and each category uses a different acquisition channel. Category A data, being high-frequency change data, is the core data source for cost accounting and project management, and is therefore the most important. To ensure the integrity of Category A data acquisition, the system is configured with both real-time and scheduled upload methods: real-time upload continuously captures incremental change data through a real-time extraction process, with acquisition latency controlled within 5 seconds; scheduled upload serves as a fallback mechanism, actively pushing all change data from the data source side at a preset cycle (default 2 hours). When the real-time channel experiences network jitter or data backlog, scheduled upload can re-upload any missing data, ensuring that Category A data is not lost.

[0093] Category B data is medium-frequency change data, triggered by a timer within a preset time window. The data source receives the request and returns incremental data, which is then aggregated into the verification queue via the aggregation module. Category C data is low-frequency change data, actively pushed to the acquisition framework by the data source in a timed upload mode according to a preset period. After entering through their respective channels, all three types of data are uniformly aggregated into the verification queue, where null value verification, format verification, and primary key conflict verification are performed sequentially. After passing verification, the data is stored in the data warehouse to form a standardized dataset.

[0094] In this embodiment, the preset time window has two meanings: first, the execution time point when the system triggers the batch extraction task according to the preset scheduling plan, such as a fixed time every day; second, the data time range carried in a single request, that is, the start and end time interval of the data change queried from the business system. Taking Class B medium-frequency change data as an example, its typical configuration is that the system automatically sends a query request with time range conditions (such as querying data changed in the last 2 hours) to the business system according to the preset time window (such as triggering once every 2 hours), and batch pulls the data records that have changed within the time window to ensure the timeliness and completeness of data collection.

[0095] S2 configures a unique identifier ID for each data point in the standardized dataset, and associates the data collection source, the cleaning rules executed, and the information of the aggregation and processing nodes that each data point passes through with the unique identifier ID, storing them as a full-link flow log.

[0096] Specifically, in this embodiment, three types of intelligent decision analysis models can be constructed based on a standardized project dataset: a project benefit prediction model, a project risk warning model, and a resource optimization and allocation model. Specifically, the model training dataset is constructed by collecting historical engineering project data and raw material price data from the entire group; the project benefit prediction model is trained using time series algorithms; the project risk warning model is trained based on historical cost and construction period data; and the resource optimization and allocation model is trained based on the resource occupancy data of projects under construction. The internal parameters of each model are optimized using cross-validation and grid search, and the model accuracy is iteratively optimized using historical real project data.

[0097] S3 receives the indicator code from external input, performs reverse retrieval in the end-to-end flow log based on the indicator code, locates the collection source, cleaning rules and summary processing node associated with the indicator code, and outputs the retrieved original data source information.

[0098] Specifically, the indicator codes and summary processing nodes have a preset mapping relationship, which is pre-configured in the system parameter table.

[0099] Specifically, the reverse retrieval in the end-to-end flow log based on the indicator code is as follows: based on the externally input indicator code, query the system parameter table, locate the aggregation processing node bound to the indicator code in the system parameter table, and retrieve the end-to-end flow log based on the identifier of the aggregation processing node to locate all data records processed by the aggregation processing node, as well as the collection source and cleaning rules corresponding to each data record.

[0100] Specifically, in traditional solutions, establishing a traceability relationship between indicators and raw data typically requires configuring source system mapping tables, cleaning rule mapping tables, and processing node mapping tables for each indicator. Each new indicator necessitates maintaining multiple binding relationships, leading to a linear increase in system complexity with the number of indicators and poor performance for multi-table join queries. In this embodiment, the mapping relationship between indicator codes and summary processing nodes is pre-configured in the system parameter table. Taking the indicator code "COST_OVERR_001" (cost overrun rate) as an example, the system parameter table only records the binding summary processing node "NODE_AGG_COST_01" for this indicator; there is no need to separately configure the binding relationship between this indicator and the source system and cleaning rules.

[0101] When a user triggers source tracing, the system retrieves the end-to-end flow log using "NODE_AGG_COST_01" as the condition. This log records the unique identifier ID and collection source of each input data item during data processing, such as the source system identifier, source table identifier, collection timestamp, and the various cleaning rules it passed through, such as rule codes and cleaning action descriptions. Since this information is naturally included in the end-to-end flow log, the system can obtain all the necessary data with a single log retrieval, eliminating the need to maintain additional binding relationships between indicators and sources, and between indicators and cleaning rules. This reduces the number of table associations and lowers system complexity.

[0102] Specifically, in this embodiment, the indicator is encoded as an internal string used to uniquely identify a business indicator, allowing users to trace the original source from seeing abnormal data.

[0103] Specifically, this embodiment also supports establishing a multi-dimensional arbitrary combination of penetrating query architecture based on four basic analysis dimensions: time, organization, project name, and project manager; it also supports establishing a query engine that can freely splice dimensions based on the four analysis dimensions: time, organization, project name, and project manager; with a data volume of less than 100,000 records, the query response time for drilling down to a single original business record under the aggregated indicators is ≤3s; and it supports multi-dimensional statistics and year-on-year and month-on-month analysis around the five core indicators of progress, cost, quality, safety, and compliance.

[0104] S4 builds a dimension combination query index based on the time field, organization field, project field, and personnel field in the standardized dataset. It receives external input dimension combination conditions, calls the dimension combination query index to retrieve data records that meet the dimension combination conditions from the standardized dataset, performs statistics on the retrieved data records according to their respective bound rules, and outputs the corresponding original detailed records.

[0105] Specifically, a dimensional combination query index is built based on the time, organization, project, and personnel fields in the standardized dataset, including:

[0106] S41, traverse the table structure information of the standardized dataset, obtain the date field from the table structure information, and mark the date field as a time dimension field;

[0107] S42, retrieve the field names from the table structure information, including fields related to organization, department, and branch, and mark them as organization dimension fields;

[0108] S43, retrieve the fields whose names include keywords such as project, engineering, and section from the table structure information, and mark them as project dimension fields;

[0109] S44, retrieve the fields whose field names include personnel, manager, and person in charge from the table structure information, and mark them as personnel dimension fields;

[0110] S45. The marked time dimension field, organization dimension field, project dimension field and personnel dimension field are used as composite index columns. A composite index is built in ascending order of cardinality and stored in the index storage area.

[0111] Specifically, in a concrete case, suppose a group has multiple regional branches, such as those in East China, South China, North China, and Southwest China, each running three heterogeneous business systems: a project management system, a contract management system, and a materials management system. The data table field names differ between these systems. The project information table in the project management system contains the fields project_code, project_name, org_name, project_manager, start_date, and end_date; the contract master table in the contract management system contains the fields contract_no, project_id, dept_id, responsible_person, and sign_date; and the materials order table in the materials management system contains the fields order_no, project_name, branch_code, procurement_person, and order_date. Although the three tables mentioned above belong to different business systems, they all contain four core dimensions of information necessary for project management: time, organization, project, and personnel. This solution uses S41 to S44 to automatically identify and aggregate dimension fields: S41 traverses the table structure of the three tables and marks all date type fields start_date, end_date, sign_date, and order_date as time dimension fields; S42 uses keyword matching of field names to aggregate fields org_name, dept_id, and branch_code containing "organization," "department," and "branch" as organization dimension fields. Although the three field names are different, their semantics are all the same: the organization to which the data belongs; S43 aggregates fields project_code, project_id, and project_name containing "project," "engineering," and "bid section" as project dimension fields; S44 aggregates fields project_manager, responsible_person, and procurement_person containing "personnel," "manager," and "responsible person" as personnel dimension fields. Through the above steps, the fields in the three heterogeneous tables, which originally had different names, are automatically categorized into four unified business dimensions, eliminating the need for manual configuration of mapping relationships for each table.Next, cardinality statistics are performed on the four aggregated dimension fields using S45: the organization dimension field `org_name` has only 4 different values—East China, South China, North China, and Southwest China—with the lowest cardinality; the personnel dimension field `project_manager` has 80 different values, with a medium cardinality; the project dimension field `project_code` has 200 different values, with a relatively high cardinality; the time dimension field `order_date` spans multiple months, containing 365 different values ​​at the daily granularity, with the highest cardinality. However, since it is the rightmost column in the composite index and is only used for the final range scan, it does not affect the layer-by-layer filtering efficiency of the index tree. The four dimension fields are organized into a composite index (`org_name`, `project_manager`, `project_code`, `order_date`) in ascending order of cardinality and stored in a separate index storage area. This index does not rely on the user entering query conditions in a specific order; the system automatically maps the user-input dimension conditions to the index columns for layer-by-layer retrieval during query execution. Taking a typical engineering management query scenario as an example, group managers want to query the material overspending details of Zhang San, the person in charge of a certain type of project in the South China branch, from January to March 2026. The query conditions include four dimensions: time range limited to January to March 2026, organization limited to the South China branch, project limited to residential projects, and personnel limited to Zhang San. After receiving the above combination of conditions, the system calls the composite index for retrieval: the organization field org_name with the lowest cardinality is used as the first-level filtering condition of the B+ tree index. This field has only four different values, and after a very small number of comparisons, it can directly locate the index subtree corresponding to the South China branch, excluding about 75% of the data in the East China, North China, and Southwest regions; then, in the index subtree within the scope of the South China branch, project_manager is used for further filtering to locate all records handled by Zhang San; then project_code is used to locate the specific residential projects within this scope; finally, in the remaining data, order_date is used to perform a time interval scan to accurately locate all target records from January to March 2026. After four layers of filtering, the system performs statistics on the retrieved data records according to their respective bound rules, and outputs the corresponding original material order details, realizing a lossless penetrating query from summary indicators to original details.

[0112] Furthermore, while the order of the composite index is automatically generated by the system based on cardinality statistics, in practice, the system also supports manual adjustment of the index column order through a visual configuration interface or system parameter table. For example, if business personnel find that over 80% of queries within the group use "project" as the primary filter condition instead of "organization," they can adjust the index order from the default (org_name, project_manager, project_code, order_date) to (project_code, org_name, project_manager, order_date) through the configuration interface, making the query path more aligned with actual business access patterns. The system prioritizes reading the currently active index column order configuration each time a query is executed and dynamically builds the corresponding composite index based on the configuration. This design retains automated construction capabilities while also considering the differentiated needs of different business scenarios, further enhancing the flexibility and adaptability of the indexing strategy.

[0113] Specifically, the process of retrieving data records from a standardized dataset that meet the dimension combination conditions by calling the dimension combination query index also includes:

[0114] Monitor the retrieval response time of the index for combined query dimensions;

[0115] When the retrieval response time exceeds the preset threshold, the filter fields not included in the composite index in the current query conditions are parsed, and the frequency of each filter field appearing as a condition in historical query records is counted.

[0116] Fields that appear more frequently than a preset frequency threshold will be marked as recommended custom peripheral dimensions and pushed to the user's device.

[0117] Receive user confirmation or modification of recommended custom peripheral dimensions, and generate effective custom peripheral dimension configurations;

[0118] Add the custom peripheral dimension to the composite index, recalculate the cardinality of all index columns, rearrange the column order of the composite index in ascending order of cardinality, and then store it in the index storage area.

[0119] Specifically, during the process of retrieving data records that meet the dimension combination conditions from the standardized dataset by calling the dimension combination query index, the system launches an independent monitoring process in parallel to monitor the response time of each index retrieval in real time. This monitoring process runs decoupled from the query execution thread and will not block the main query process. When the retrieval response time of a query exceeds a preset threshold (e.g., more than 3 seconds), the monitoring process automatically triggers the slow query analysis process. The system first parses the syntax structure of the current query statement, extracts all fields used as filtering conditions, and identifies the set of fields that have not yet been included in the current composite index. Subsequently, the system queries historical query log records to count the total number and frequency of the above-mentioned unindexed filtering fields as query conditions over a period of time (e.g., the past 30 days). Fields with a frequency exceeding a preset frequency threshold (e.g., more than 10 times per day on average) are marked as candidate recommended dimensions. The system pushes the candidate recommended dimensions to the user in the form of a list, along with the frequency statistics of the field and example query statements for the user's reference. The user can confirm and adopt a recommended dimension, modify it to another dimension, or ignore the recommendation in the list. Upon receiving user confirmation or modification, the system generates an effective custom peripheral dimension configuration. This field is added to the existing four-dimensional composite index (Time, Organization, Project, Personnel) as an additional dimension column. The cardinality of all index columns, including this new dimension, is recalculated, and the column order of the composite index is rearranged in ascending order of cardinality. The adjusted index structure is then stored in the index storage area. Furthermore, as a further optimization, the system can analyze the correlation between the recommended custom peripheral dimension and the existing four-dimensional index columns. If a field has a strong correlation with an existing index column (e.g., "supplier" and "project" frequently appear together in query conditions), its recommendation priority is increased; conversely, if a field rarely appears in combined queries with an existing index column, its recommendation priority is decreased. The system can also automatically create a test index after recommendation, simulating several typical queries in a test environment to verify the index optimization effect, and presenting the verification results to the user to assist in decision-making. The system also supports configuring expiration dates for recommended dimensions—for example, temporary dimensions that are only valid for the current project phase and automatically expire after the project phase is switched, thus avoiding storage bloat and write performance degradation caused by the infinite accumulation of index dimensions.

[0120] S5 uses the unique identifier ID as the primary key to aggregate and associate the original data source information and original detail records belonging to the same data record, and exports the aggregated complete data record.

[0121] Specifically, in this embodiment, the original data source information and the original detailed records describe data attributes from different dimensions. The original data source information is a traceability record at the metadata level, including the data record's collection source, such as: source system identifier, source data table identifier and collection timestamp, the executed cleaning rules, such as: rule code and cleaning action description, and the aggregation and processing node information it has passed through. Its function is to depict where the data comes from and what kind of processing it has undergone. The original detailed records, on the other hand, are the specific data content at the business level, including the values ​​of core business fields such as the project's actual cost, planned duration, and resource input. Their function is to present what the data is. Together, they constitute a complete description of a data record. The source information provides the background and processing chain of the data, while the detailed records provide the actual business meaning of the data.

[0122] Specifically, in S2, each data record in the standardized dataset is configured with a unique identifier ID. The data record's acquisition source, the cleansing rules executed, and the aggregation and processing nodes it passed through are all associated with this unique identifier ID and stored as a full-link flow log. Therefore, each data record in the standardized dataset has a unique identity, and this unique identifier ID also serves as a key index field in the full-link flow log. Based on this, the original data source information output from the full-link flow log in S3, retrieved based on indicator codes, and the original detailed records retrieved from the standardized dataset in S4, retrieved based on dimension combination query indexes, both naturally carry the corresponding unique identifier ID. Based on this, in S5, using the unique identifier ID as the association primary key, the original data source information (stored in the full-link flow log) and the original detailed records (stored in the standardized dataset) belonging to the same data record are aggregated and associated. This achieves complete aggregation and export of data records without requiring the original data source information and original detailed records to be physically stored in the same location; as long as both carry the common association attribute, the unique identifier ID, complete aggregation and export of data records can be achieved.

[0123] Specifically, following S5, it also includes conducting a comprehensive quality score for the project to be evaluated based on the aggregated complete data records, specifically including:

[0124] S61, Obtain the project type, planned duration, and planned cost of the project to be evaluated;

[0125] S62, from the aggregated complete data records, filter historical project records that belong to the same project type as the project to be evaluated, and calculate the schedule deviation rate, cost deviation rate and resource utilization rate of each historical project;

[0126] S63, take the average of the schedule deviation rate, cost deviation rate and resource utilization rate of each historical project to obtain the schedule deviation benchmark value, cost deviation benchmark value and resource utilization benchmark value of this type of project;

[0127] S64. The planned duration and the baseline value of the duration deviation of the project to be evaluated are superimposed to obtain the predicted duration; the planned cost and the baseline value of the cost deviation of the project to be evaluated are superimposed to obtain the predicted cost.

[0128] S65 generates a comprehensive quality score for the project to be evaluated based on the predicted construction period, predicted cost, and resource utilization benchmarks.

[0129] Specifically, to ensure the accuracy of the comprehensive quality score, when the number of historical project records in the same project set is less than a preset threshold (less than 3 by default), a tiered downgrade strategy is automatically executed: Level 1: The project size matching range is widened from ±30% to ±50%, and the same project set is re-filtered; Level 2: If the threshold is still not met after widening, the project size restriction is further removed, and matching is done only by project type; Level 3: If the threshold is still not met after matching by project type, the average deviation rate of all historical projects in the group is used as a reference value, and the evaluation result is marked with the message "Insufficient reference data, low confidence level". For example, if a certain engineering project has no historical project records of the same type, after the system executes the three-level strategy sequentially, it uses the average of all projects in the group as a reference value and marks the evaluation report with "Currently, there is no similar historical project data; the following predicted values ​​are for reference only".

[0130] Specifically, the schedule deviation rate, cost deviation rate, and resource utilization rate of each historical project can be calculated as follows: Schedule deviation rate = (actual schedule - planned schedule) / planned schedule × 100%, which measures the degree of deviation between the actual schedule and the planned schedule. A positive value indicates a delay, and a negative value indicates an early completion. Cost deviation rate = (actual cost - budgeted cost) / budgeted cost × 100%, which measures the degree of deviation between the actual cost and the budgeted cost. A positive value indicates a cost overrun, and a negative value indicates a cost saving. Resource utilization rate = actual amount of resources used / total available resources × 100%, which measures the actual efficiency of the use of various resources during project implementation. The basic data involved in the above indicators, such as the actual schedule, planned schedule, actual cost, budgeted cost, actual resource usage, and total available resources, can all be directly obtained or statistically obtained from the aggregated complete data records.

[0131] Specifically, in this embodiment, the comprehensive quality score adopts a 100-point system, with 100 points as the base score. Points are deducted based on the deviation rate between the predicted and planned project duration, the deviation rate between the predicted and planned costs, and the degree to which the resource utilization benchmark value deviates from the reasonable range of industry standards. The higher the project duration deviation rate and cost deviation rate, the more points are deducted. Points are also deducted for excessively high or low resource utilization. The score after deduction is the comprehensive quality score.

[0132] Furthermore, this embodiment can also be applied to situations where, when a user triggers abnormal indicators or requires a solution decision, the system calls upon the intelligent decision-making model using this method to perform calculations, automatically identifying abnormal indicators such as cost overruns and project delays, and calculating the benefits and prioritizing multiple alternative project solutions. The intelligent decision-making suggestion output delay is controlled within 5 seconds, and quantitative suggestions for the allocation of idle resources and the rectification of risky projects are automatically output. At the same time, the system automatically calculates the deviation between the actual project cost and the model-predicted cost, the deviation between the actual project duration and the planned project duration, calculates the actual utilization rate of human and material resources, quantifies the level of project change control, and finally integrates multi-dimensional analysis, traceability verification, and model calculation data to automatically output a project operation evaluation report.

[0133] Specifically, the complete process of applying this method to the group's actual engineering projects is as follows:

[0134] First, heterogeneous data from multiple sources, including contracts, finances, and materials, are collected through a dual-mode approach of real-time synchronization and batch extraction. Null value verification, format verification, and primary key conflict verification are then performed sequentially. If the arrival date of a material entry record is empty, it is marked as isolated. If the amount of a change order record contains illegal characters, it is isolated due to format anomaly, corrected, and re-entered into the warehouse. Duplicate payment records are deduplicated through hash sharding comparison. Data records that pass all three types of verification are configured with unique identifiers and stored in the data warehouse to form a standardized dataset. At the same time, the data collection source, the cleaning rules executed, and the information of the aggregation and processing nodes that each data point passes through are associated with a unique identifier and stored as a full-link flow log.

[0135] After the completion of a residential development project, the system automatically calculated the cost overrun rate based on the actual cost and planned cost, and marked it as an anomaly. When a user clicked on this indicator, it triggered reverse tracing. The system queried the system parameter table to locate the aggregation and processing node bound to this indicator, and retrieved the entire process log using this node identifier. This returned the original data records involved in the indicator calculation, along with their source systems and cleaning rules. Upon investigation, it was found that a payment record for pile foundation work exceeded the contractual amount. This change was not promptly updated in the contract management system, resulting in the cost overrun not being identified in advance. The user then used a combination of "time + organization + project" dimensions to retrieve a standardized dataset using a composite index. After the system aggregated the data according to five major indicators—progress, cost, quality, safety, and compliance—the user drilled down to the corresponding cost details, further verifying the original documents for the abnormal payment.

[0136] The system uses a unique identifier ID as the primary key to aggregate the source information from the traceability output and the detailed records from the query output, exporting complete data records. Based on this complete data, the system filters historical projects of the same type to calculate benchmark values ​​for schedule deviation rate and cost deviation rate, and then overlays the planned schedule and planned cost of the project to output predicted schedule and predicted cost. The actual execution results are significantly higher than the predicted values. Combined with the uncertified change order issues identified in the traceability, the assessment report recommends adding a mandatory change order-related early warning mechanism for subsequent projects. The entire process, from data collection to assessment report output, takes approximately 15 minutes, compared to 3-5 working days using traditional methods, significantly improving management efficiency and accuracy.

[0137] like Figure 3 As shown in the figure, this embodiment also discloses a system for the acquisition, verification, storage, and traceability of multi-source data in engineering projects, including:

[0138] Data preprocessing module 31 is used to collect multi-source heterogeneous data through real-time synchronization and batch extraction dual modes, perform primary key conflict verification, format verification and null value verification on the collected multi-source heterogeneous data, mark and isolate the data that fails the verification, and clean and store the data that passes the verification into the data warehouse to form a standardized dataset.

[0139] The end-to-end flow log generation module 32 is used to configure a unique identifier ID for each piece of data in the standardized dataset, and associate the collection source, the cleaning rules executed and the information of the aggregation and processing nodes that each piece of data passes through with the unique identifier ID, and store it as an end-to-end flow log.

[0140] The reverse retrieval module 33 is used to receive externally input indicator codes, perform reverse retrieval in the full-link flow log based on the indicator codes, locate the collection source, cleaning rules and summary processing nodes associated with the indicator codes, and output the retrieved original data source information.

[0141] The original detailed record output module 34 is used to build a dimension combination query index based on the time field, organization field, project field and personnel field in the standardized dataset. It receives external input dimension combination conditions, calls the dimension combination query index to retrieve data records that meet the dimension combination conditions from the standardized dataset, performs statistics on the retrieved data records according to their respective bound rules, and outputs the corresponding original detailed records.

[0142] A dimensional combination query index is built based on the time, organization, project, and personnel fields in the standardized dataset, specifically including:

[0143] Traverse the table structure information of the standardized dataset, extract the date fields from the table structure information, and mark the date fields as time dimension fields;

[0144] Extract field names from the table structure information, including those related to organization, department, and branch office, and mark them as organization dimension fields;

[0145] Retrieve fields from the table structure information whose names include keywords such as "project," "engineering," and "section," and mark them as project dimension fields.

[0146] Retrieve fields from the table structure information whose names include personnel, managers, and responsible persons, and mark them as personnel dimension fields;

[0147] The marked time dimension field, organization dimension field, project dimension field and personnel dimension field are used as composite index columns. A composite index is built in ascending order of cardinality and stored in the index storage area.

[0148] The data export module 35 is used to aggregate and associate the original data source information and original detail records belonging to the same data record using a unique identifier ID as the association primary key, and export the aggregated complete data record.

[0149] A specific implementation of a system for collecting, verifying, storing, and tracing back multi-source data for engineering projects. The same method for collecting, verifying, storing, and tracing back multi-source data for engineering projects will not be described again in this embodiment.

[0150] Although the invention has been specifically shown and described in conjunction with preferred embodiments, those skilled in the art should understand that various changes in form and detail may be made to the invention without departing from the spirit and scope of the invention as defined in the appended claims, all of which shall be within the scope of protection of the invention.

Claims

1. A method for collecting, verifying, storing, and tracing back multi-source data in engineering projects, characterized in that, Includes the following steps: S1. Obtain the change frequency of each data table. Based on the change frequency, select real-time synchronization and batch extraction dual modes to collect multi-source heterogeneous data. Perform primary key conflict verification, format verification and null value verification on the collected multi-source heterogeneous data. Mark and isolate the data that fails the verification. Clean the data that passes the verification and store it in the data warehouse to form a standardized dataset. S2, configure a unique identifier ID for each data point in the standardized dataset, and associate the data collection source, the cleaning rules executed, and the information of the aggregation and processing nodes that each data point passes through with the unique identifier ID, and store it as a full-link flow log; S3 receives externally input indicator codes, performs reverse retrieval in the end-to-end flow log based on the indicator codes, locates the collection source, cleaning rules and summary processing nodes associated with the indicator codes, and outputs the retrieved original data source information. S4: Based on the time field, organization field, project field and personnel field in the standardized dataset, a dimension combination query index is built. It receives the dimension combination conditions from external input, calls the dimension combination query index to retrieve data records that meet the dimension combination conditions from the standardized dataset, performs statistics on the retrieved data records according to their respective bound rules, and outputs the corresponding original detailed records. A dimensional combination query index is built based on the time, organization, project, and personnel fields in the standardized dataset, specifically including: S41, traverse the table structure information of the standardized dataset, obtain the date field from the table structure information, and mark the date field as a time dimension field; S42, retrieve the field names from the table structure information, including fields related to organization, department, and branch, and mark them as organization dimension fields; S43, retrieve the fields whose names include keywords such as project, engineering, and section from the table structure information, and mark them as project dimension fields; S44, retrieve the fields whose field names include personnel, manager, and person in charge from the table structure information, and mark them as personnel dimension fields; S45, use the marked time dimension field, organization dimension field, project dimension field and personnel dimension field as composite index columns, build a composite index in order of cardinality from low to high, and store it in the index storage area; S5 uses the unique identifier ID as the primary key to aggregate and associate the original data source information and original detail records belonging to the same data record, and exports the aggregated complete data record.

2. The method for collecting, verifying, storing, and tracing back multi-source data in engineering projects according to claim 1, characterized in that, In S1, the change frequency of each data table is obtained. Based on the change frequency, a dual-mode approach of real-time synchronization and batch extraction is selected to collect multi-source heterogeneous data. Specifically, this includes: Obtain the data tables to be collected from the databases of each business system, and calculate the average number of change records for each data table within the past preset number of statistical periods; Data tables with an average number of change records exceeding a preset threshold are defined as high-frequency change data tables. Data in high-frequency change data tables are classified into real-time synchronization mode, and incremental change data is captured in real time by listening to the database change log. Data tables with an average number of change records lower than or equal to a preset threshold are defined as low-frequency batch data tables. Data in low-frequency batch data tables are categorized into batch extraction mode, and query requests with time range conditions are sent to each business system according to a preset time window to batch extract data.

3. The method for collecting, verifying, storing, and tracing back multi-source data in engineering projects according to claim 1, characterized in that, In S1, primary key conflict verification, format verification, and null value verification are performed on the collected multi-source heterogeneous data, specifically including: S11, iterate through each field of the collected data, determine whether the current field is allowed to be empty according to the preset field non-empty rule library, and mark the data record with the required field value being empty as an empty value exception; S12, extract the source system identifier and data table identifier of the data that passed the null value check, load the corresponding field format template from the format rule library, compare the values ​​of each field in the data record with the format template, and mark the data that does not meet the format requirements as format abnormal; S13, extract the primary key field value of the data record that has passed the format validation, divide the data to be validated into multiple primary key area segments according to the hash distribution of the primary key field value, sort the data records in each primary key area segment according to the primary key, compare the primary key values ​​of adjacent records after sorting, and mark the data records with the same primary key value as primary key conflict anomalies. S14. Data records marked as null value exception, format exception, and primary key conflict exception are isolated and stored. Data records that pass all three types of checks are cleaned and stored in the data warehouse.

4. The method for collecting, verifying, storing, and tracing back multi-source data in engineering projects according to claim 1, characterized in that, S1 also includes the process of marking and isolating data that fails verification and then repairing it, specifically including: Data records marked as null value exceptions, format exceptions, and primary key conflict exceptions are isolated and stored in the exception data temporary storage area. Each exception record in the exception data temporary storage area is traversed, and repair processing is performed in the following manner: For fields that fail to trigger validation, firstly, search for related records with the same primary key or the same business entity in the local related dimension table to complete the data; if no results are found in the local search, send an HTTP asynchronous request to the external data service platform for a global search and receive the callback results to complete the data; if the external search times out or has no valid results, push the data to the administrator for manual entry. If any one of the repair levels succeeds, the repair of that field will be terminated and the operation log will be recorded. If all three levels fail, the repair of that field will be marked as failed. After all abnormal fields have been processed, if any field fails to be repaired, the entire record will be re-isolated. If all fields are successfully repaired, primary key conflict, format and null value checks will be re-executed on the repaired record. If the checks pass, the record will be stored in the data warehouse; otherwise, it will be isolated again for the next round of repair.

5. The method for collecting, verifying, storing, and tracing back multi-source data in engineering projects according to claim 1, characterized in that, In S3, the indicator codes and summary processing nodes have a preset mapping relationship, which is pre-configured in the system parameter table.

6. The method for collecting, verifying, storing, and tracing back multi-source data in engineering projects according to claim 4, characterized in that, In S3, reverse retrieval is performed in the end-to-end flow log based on the indicator code. Specifically, the system parameter table is queried based on the externally input indicator code to locate the aggregation processing node bound to the indicator code in the system parameter table. The end-to-end flow log is retrieved based on the identifier of the aggregation processing node to locate all data records processed by the aggregation processing node, as well as the collection source and cleaning rules corresponding to each data record.

7. The method for collecting, verifying, storing, and tracing back multi-source data in engineering projects according to claim 1, characterized in that, In S4, the process of retrieving data records from the standardized dataset that meet the dimension combination conditions by calling the dimension combination query index also includes: Monitor the retrieval response time of the index for combined query dimensions; When the retrieval response time exceeds the preset threshold, the filter fields not included in the composite index in the current query conditions are parsed, and the frequency of each filter field appearing as a condition in historical query records is counted. Fields that appear more frequently than a preset frequency threshold will be marked as recommended custom peripheral dimensions and pushed to the user's device. Receive user confirmation or modification of recommended custom peripheral dimensions, and generate effective custom peripheral dimension configurations; Add the custom peripheral dimension to the composite index, recalculate the cardinality of all index columns, rearrange the column order of the composite index in ascending order of cardinality, and then store it in the index storage area.

8. The method for collecting, verifying, storing, and tracing back multi-source data in engineering projects according to claim 1, characterized in that, Following S5, it also includes, based on the aggregated complete data records, a comprehensive quality score for the project to be evaluated, specifically including: S61, Obtain the project type, planned duration, and planned cost of the project to be evaluated; S62, from the aggregated complete data records, filter historical project records that belong to the same project type as the project to be evaluated based on the project field, and calculate the schedule deviation rate, cost deviation rate and resource utilization rate of each historical project; S63, take the average of the schedule deviation rate, cost deviation rate and resource utilization rate of each historical project to obtain the schedule deviation benchmark value, cost deviation benchmark value and resource utilization benchmark value of this type of project; S64. The planned duration and the baseline value of the duration deviation of the project to be evaluated are superimposed to obtain the predicted duration; the planned cost and the baseline value of the cost deviation of the project to be evaluated are superimposed to obtain the predicted cost. S65 generates a comprehensive quality score for the project to be evaluated based on the predicted construction period, predicted cost, and resource utilization benchmarks.

9. A system for the acquisition, verification, storage, and traceability of multi-source data in engineering projects, characterized in that, include: The data preprocessing module is used to obtain the change frequency of each data table, select real-time synchronization and batch extraction dual modes to collect multi-source heterogeneous data based on the change frequency, perform primary key conflict verification, format verification and null value verification on the collected multi-source heterogeneous data, mark and isolate the data that fails the verification, and clean the data that passes the verification and store it in the data warehouse to form a standardized dataset. The end-to-end flow log generation module is used to configure a unique identifier ID for each piece of data in the standardized dataset, and associate the collection source, the cleaning rules executed, and the information of the aggregation and processing nodes that each piece of data passes through with the unique identifier ID, and store them as an end-to-end flow log; The reverse retrieval module is used to receive externally input indicator codes, perform reverse retrieval in the end-to-end flow log based on the indicator codes, locate the collection source, cleaning rules and summary processing nodes associated with the indicator codes, and output the original data source information retrieved. The original detailed record output module is used to build a dimension combination query index based on the time field, organization field, project field and personnel field in the standardized dataset. It receives external input dimension combination conditions, calls the dimension combination query index to retrieve data records that meet the dimension combination conditions from the standardized dataset, performs statistics on the retrieved data records according to their respective bound rules, and outputs the corresponding original detailed records. A dimensional combination query index is built based on the time, organization, project, and personnel fields in the standardized dataset, specifically including: Traverse the table structure information of the standardized dataset, extract the date fields from the table structure information, and mark the date fields as time dimension fields; Extract field names from the table structure information, including those related to organization, department, and branch office, and mark them as organization dimension fields; Retrieve fields from the table structure information whose names include keywords such as "project," "engineering," and "section," and mark them as project dimension fields. Retrieve fields from the table structure information whose names include personnel, managers, and responsible persons, and mark them as personnel dimension fields; The marked time dimension field, organization dimension field, project dimension field and personnel dimension field are used as composite index columns. A composite index is built in ascending order of cardinality and stored in the index storage area. The data export module is used to aggregate and associate the original data source information and original detailed records belonging to the same data record using a unique identifier ID as the association primary key, and export the aggregated complete data record.

Images