Real-time synchronization processing method and system for engineering project data under multi-source heterogeneous integration

Abstract

This invention discloses a method and system for real-time synchronization of engineering project data under multi-source heterogeneous integration, relating to the field of data processing technology. The method includes: constructing a cloud-edge-device collaborative architecture; real-time collection of multi-source heterogeneous engineering project data, and aggregation and integration processing to obtain integrated engineering project data; performing consistency comparison and logical verification on the integrated engineering project data to generate data comparison and verification results; synchronously evaluating the confidence level of the integrated engineering project data using a weighted scoring and multi-threaded verification algorithm, identifying low-confidence project data, and triggering an anomaly handling mechanism based on the low-confidence project data for data processing and quality closed-loop control. This solves the technical problems of difficulty in real-time synchronization of multi-source heterogeneous engineering project data, low integration processing efficiency, and difficulty in effectively controlling data quality in existing technologies, achieving the technical effects of improving the efficiency of engineering project data synchronization and integration processing, data credibility, and overall data quality controllability.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, specifically to a method and system for real-time synchronization processing of engineering project data under multi-source heterogeneous integration. Background Technology

[0002] With the continuous expansion of engineering construction projects and the ongoing improvement of informatization levels, engineering projects have gradually generated multi-source heterogeneous data throughout their entire lifecycle, including planning, design, construction, and operation and maintenance. This data comes from design systems, progress management systems, quality and safety systems, IoT sensing devices, and third-party business platforms. Due to significant differences in data structure, collection frequency, transmission methods, and business semantics among these various data types, existing engineering project data management methods largely rely on independent collection and periodic aggregation by subsystems, making it difficult to achieve real-time synchronization and efficient integration of cross-system data. Furthermore, during data synchronization, issues such as data inconsistency, duplication, missing data, or logical conflicts frequently occur due to factors such as network environment, system interface differences, and human error. Moreover, existing technologies typically lack quantitative assessment and closed-loop control mechanisms for data quality and reliability, making it difficult to guarantee the reliability and availability of engineering project data. This hinders the implementation of refined management and intelligent decision-making in engineering projects. Summary of the Invention

[0003] This application provides a method and system for real-time synchronization of engineering project data under multi-source heterogeneous integration, which solves the technical problems in the prior art of difficulty in real-time synchronization of multi-source heterogeneous engineering project data, low integration processing efficiency, and difficulty in effectively controlling data quality.

[0004] The first aspect of this application provides a method for real-time synchronization and processing of engineering project data under multi-source heterogeneous integration, the method comprising: A cloud-edge-device collaborative architecture is constructed, comprising a cloud platform and edge nodes. A data hub module and a rule engine module are deployed on the cloud platform, and a lightweight data proxy module is deployed on the edge nodes. The lightweight data proxy module collects multi-source heterogeneous engineering project data in real time, encrypts and transmits this data to the data hub module for aggregation and integration, resulting in integrated engineering project data. The rule engine module invokes preset business logic association rules to perform consistency comparison and logical verification on the integrated engineering project data, generating data comparison and verification results. Based on these results, a weighted scoring and multi-threaded verification algorithm are used to synchronously assess the confidence level of the integrated engineering project data, identifying low-confidence project data. An anomaly handling mechanism is triggered based on these low-confidence project data to perform data processing and closed-loop quality control.

[0005] Furthermore, the information of each node in the edge nodes is initialized and acquisition tasks are assigned to obtain an edge node acquisition task assignment list; according to the lightweight data proxy module, a data configuration acquisition strategy is obtained, which includes timed acquisition and event-driven acquisition; the lightweight data proxy module monitors the project data in real time based on the edge node acquisition task assignment list and the data configuration acquisition strategy to acquire initial project data; the initial project data is then identified by its acquisition source, timestamp, and format to obtain multi-source heterogeneous project data.

[0006] Furthermore, the multi-source heterogeneous engineering project data is encrypted and transmitted to the data hub module for integrity verification to obtain a data transmission verification result. If the data transmission verification result is complete, a data preprocessing step is obtained through the data hub module, including data cleaning and structured format conversion. Based on the data preprocessing step, the multi-source heterogeneous engineering project data is preprocessed to obtain usable engineering project data. The usable engineering project data is then aggregated and integrated through the data hub module to obtain integrated engineering project data.

[0007] Furthermore, the data hub module invokes the engineering project data association map; based on the engineering project data association map, entity identification and association analysis are performed on the available engineering project data to obtain engineering project associated entity data; the engineering project associated entity data is then subjected to time-series alignment and aggregation integration processing to obtain integrated engineering project data.

[0008] Furthermore, based on the business requirements of the engineering project, a knowledge graph structure for the engineering project is defined, which includes entity types, entity attributes, and entity relationship types. Related business data of the engineering project is acquired, and entity extraction and knowledge fusion are performed on the related business data according to the knowledge graph structure to generate a basic engineering project knowledge graph. The encoding of each knowledge node in the basic engineering project knowledge graph is optimized to construct an engineering project data association graph, and the engineering project data association graph is stored in the data hub module.

[0009] Furthermore, the preset business logic association rules include data consistency rules, business logic compliance rules, and abnormal data detection rules; the preset business logic association rules are grouped by business domain and the execution mode is set to obtain business logic domain association rules; based on the business logic domain association rules, consistency comparison and logic verification are performed on the integrated data of the engineering project to generate data comparison and verification results.

[0010] Furthermore, based on the data comparison and verification results, the integrated data of the engineering projects is filtered to obtain the engineering project data to be evaluated; a weighted scoring and multi-threaded verification algorithm is used to evaluate the confidence level of the engineering project data to be evaluated, and a confidence level set of the engineering project data is obtained; according to the confidence level set of the engineering project data, data in the engineering project data to be evaluated that are lower than a preset confidence threshold are identified and marked to determine the low-confidence project data.

[0011] Furthermore, the project data to be evaluated is divided into blocks based on a multi-threaded verification algorithm to obtain multi-threaded project data blocks; a weighted scoring method is used to perform parallel confidence scoring on the multi-threaded project data blocks to obtain a project data confidence set.

[0012] Furthermore, a confidence assessment dimension is constructed, and weights are assigned to the confidence assessment dimension to determine multi-dimensional weight decision coefficients. Based on the confidence assessment dimension and the multi-dimensional weight decision coefficients, the multi-threaded engineering project data block is simultaneously weighted and scored to obtain the engineering project data confidence set.

[0013] A second aspect of this application provides a real-time synchronization and processing system for engineering project data under multi-source heterogeneous integration, the system comprising: Architecture Construction Components: A cloud-edge-device collaborative architecture is constructed, comprising a cloud platform and edge nodes. A data hub module and a rule engine module are deployed on the cloud platform, and a lightweight data proxy module is deployed on the edge nodes. Data Processing Components: The lightweight data proxy module collects multi-source heterogeneous engineering project data in real time and encrypts and transmits this data to the data hub module for aggregation and integration, resulting in integrated engineering project data. Data Verification Components: The rule engine module invokes preset business logic association rules to perform consistency comparison and logical verification on the integrated engineering project data, generating data comparison and verification results. Data Evaluation Components: Based on the data comparison and verification results, a weighted scoring and multi-threaded verification algorithm is used to synchronously evaluate the confidence level of the integrated engineering project data, identifying low-confidence project data and triggering an anomaly handling mechanism for data processing and quality closed-loop control based on the low-confidence project data.

[0014] One or more technical solutions provided in this application have at least the following technical effects or advantages: First, a cloud-edge-device collaborative architecture is constructed, comprising a cloud platform and edge nodes. A data hub module and a rules engine module are deployed on the cloud, while a lightweight data proxy module is deployed on the edge nodes. Next, the lightweight data proxy module collects multi-source heterogeneous engineering project data in real time and transmits this encrypted data to the data hub module for aggregation and integration, resulting in integrated engineering project data. Then, the rules engine module invokes preset business logic association rules to perform consistency comparison and logical verification on the integrated engineering project data, generating data comparison and verification results. Finally, based on the data comparison and verification results, a weighted scoring and multi-threaded verification algorithm is used to synchronously assess the confidence level of the integrated engineering project data, identifying low-confidence project data and triggering an anomaly handling mechanism for data processing and closed-loop quality control. This approach solves the technical problems of difficulty in real-time synchronization of multi-source heterogeneous engineering project data, low integration processing efficiency, and difficulty in effectively controlling data quality in existing technologies, achieving the technical effects of improving the efficiency of engineering project data synchronization and integration processing, data credibility, and overall data quality controllability. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 A schematic flowchart illustrating the real-time synchronization processing method for engineering project data under multi-source heterogeneous integration provided in this application embodiment; Figure 2 This is a schematic diagram of the structure of a real-time synchronization processing system for engineering project data under multi-source heterogeneous integration, provided in an embodiment of this application.

[0017] Figure labeling: Architecture building component 11, data processing component 12, data verification component 13, data evaluation component 14. Detailed Implementation

[0018] To further illustrate the technical means and effects of the present invention in achieving its intended purpose, the following detailed description of the specific implementation methods, structures, features, and effects of the present invention, in conjunction with the accompanying drawings and preferred embodiments, is provided below.

[0019] Example 1, as Figure 1 As shown, this application provides a method for real-time synchronization and processing of engineering project data under multi-source heterogeneous integration, wherein the method includes: A cloud-edge-device collaborative architecture is constructed, which includes a cloud and edge nodes. A data hub module and a rule engine module are deployed on the cloud, and a lightweight data proxy module is deployed on the edge nodes.

[0020] In this embodiment, a cloud-edge-device collaborative architecture is constructed. This architecture includes at least cloud computing nodes and multiple distributed edge nodes. The cloud serves as a centralized data processing and decision-making hub, while the edge nodes act as data access and preprocessing units for project sites or business systems. A data hub module and a rule engine module are deployed in the cloud. The data hub module is used to uniformly access, cache, preprocess, schedule, and aggregate project data uploaded from various edge nodes to form integrated project data. The rule engine module is logically connected to the data hub module and is used to load preset business logic association rules and perform consistency checks, logical verifications, and anomaly identification on the integrated project data based on these rules. A lightweight data proxy module is deployed on each edge node. This lightweight data proxy module runs as a resident process and is used to uniformly encapsulate and manage the engineering project data sources connected to the edge nodes. It collects engineering project data in real time according to the configured collection strategy and completes data format identification, timestamp annotation, and preliminary integrity verification during the collection process. At the same time, the lightweight data proxy module encrypts and transmits the collected engineering project data to the data hub module in the cloud through a secure communication channel, realizing real-time access and synchronous processing of multi-source heterogeneous data of engineering projects in a cloud-edge-device collaborative architecture.

[0021] The lightweight data proxy module collects multi-source heterogeneous engineering project data in real time, and transmits the multi-source heterogeneous engineering project data in encryption to the data hub module for aggregation and integration processing to obtain integrated engineering project data.

[0022] After the cloud-edge-device collaborative architecture is built, lightweight data proxy modules deployed on each edge node collect multi-source heterogeneous data from engineering projects in real time. Specifically, the lightweight data proxy modules continuously monitor or periodically poll the accessed engineering project data sources according to a pre-configured data collection strategy. These engineering project data sources include, but are not limited to, engineering management systems, construction progress systems, quality and safety systems, IoT sensing devices, and third-party business platform interfaces. During the collection process, the lightweight data proxy modules uniformly encapsulate the acquired raw engineering project data, including identifying the data source, adding timestamps, and labeling the data format, thereby forming structured or semi-structured multi-source heterogeneous engineering project data. Subsequently, the lightweight data proxy modules encrypt the multi-source heterogeneous engineering project data through an established secure communication channel and transmit the encrypted data to the cloud-based data hub module in real time. After receiving multi-source heterogeneous engineering project data, the data hub module performs data access caching and scheduling management, and cleans, converts, and aggregates the data according to the preset data preprocessing process to finally form integrated engineering project data for subsequent consistency verification, logical verification, and data quality assessment.

[0023] Furthermore, the lightweight data proxy module collects multi-source heterogeneous engineering project data in real time, including: The information of each node in the edge nodes is initialized and acquisition tasks are assigned to obtain an edge node acquisition task assignment list; according to the lightweight data proxy module, a data configuration acquisition strategy is obtained, which includes timed acquisition and event-driven acquisition; the lightweight data proxy module monitors the project data in real time based on the edge node acquisition task assignment list and the data configuration acquisition strategy to acquire initial project data; the initial project data is identified by acquisition source, timestamp, and format identifier to obtain multi-source heterogeneous project data.

[0024] First, initialization is performed on each edge node deployed at the project site or business system side. This initialization includes node registration, node communication status detection, and identification of accessible data sources. After initialization, based on the data source type, data size, and collection frequency requirements of each edge node, corresponding data collection tasks are assigned to each edge node, generating an edge node collection task allocation list. Subsequently, a lightweight data proxy module loads the data configuration collection strategy. This strategy includes at least a timed collection strategy and an event-driven collection strategy. The timed collection strategy periodically collects project data at preset time intervals, while the event-driven collection strategy immediately initiates data collection when changes in project data status or business events are detected. Based on this, the lightweight data proxy module monitors and collects project data in real time according to the edge node acquisition task allocation list and the data configuration acquisition strategy to obtain initial project data. Finally, it performs unified identification processing on the initial project data, including labeling the data acquisition source, acquisition timestamp, and data format type, thereby forming multi-source heterogeneous project data with traceable source and time consistency, providing basic data support for subsequent data transmission, integration processing, and quality assessment.

[0025] Furthermore, the integrated data obtained from the engineering project includes: The multi-source heterogeneous engineering project data is encrypted and transmitted to the data hub module for integrity verification to obtain a data transmission verification result. If the data transmission verification result is complete, the data hub module obtains data preprocessing steps, including data cleaning and structured format conversion. Based on the data preprocessing steps, the multi-source heterogeneous engineering project data is preprocessed to obtain usable engineering project data. The data hub module then aggregates and integrates the usable engineering project data to obtain integrated engineering project data.

[0026] A lightweight data proxy module encrypts and transmits multi-source heterogeneous engineering project data, already marked with source identifiers and timestamps, to a cloud-based data hub module via a secure communication channel. Upon receiving the multi-source heterogeneous engineering project data, the data hub module performs integrity verification on the data transmission process. This integrity verification includes checking the integrity of data packets, transmission consistency, and decryption validity, and generates a data transmission verification result accordingly. When the data transmission verification result is complete, the data hub module invokes preset data preprocessing steps. These preprocessing steps include at least data cleaning and structured format conversion. Data cleaning removes duplicate data, abnormal data, and data with missing key fields, while structured format conversion unifies engineering project data from different sources and in different formats into a parsable data structure. After completing the data preprocessing step, the multi-source heterogeneous engineering project data is preprocessed based on the data preprocessing step to obtain usable engineering project data. Subsequently, the usable engineering project data is uniformly aggregated and integrated through the data hub module, including data collection and association according to the business dimensions of engineering projects, thereby forming integrated engineering project data, which provides a unified data foundation for subsequent data consistency comparison, logical verification and confidence assessment.

[0027] Furthermore, the available engineering project data is aggregated and integrated through the data hub module to obtain integrated engineering project data, including: The data hub module invokes the engineering project data association map; based on the engineering project data association map, entity identification and association analysis are performed on the available engineering project data to obtain engineering project associated entity data; the engineering project associated entity data is then subjected to time-series alignment and aggregation integration processing to obtain integrated engineering project data.

[0028] First, the data hub module invokes a pre-built and stored engineering project data association graph, which describes the entity relationships and association rules between various business objects in the engineering project. Based on this, entity identification and association analysis are performed on the available engineering project data according to the engineering project data association graph. Specifically, this includes identifying business entities related to the engineering project from the available engineering project data and matching these entities according to the entity relationships defined in the engineering project data association graph, thereby forming engineering project associated entity data. Subsequently, the engineering project associated entity data undergoes time-series alignment processing, mapping entity data from different data sources and different collection times to a unified time base. After time-series alignment, the engineering project associated entity data undergoes aggregation and integration processing, including data integration and association summarization according to the engineering project business dimensions, ultimately resulting in structurally consistent and clearly defined integrated engineering project data.

[0029] Furthermore, by using the data hub module, the project data association map is invoked, including: Based on the business requirements of the engineering project, a knowledge graph structure for the engineering project is defined, which includes entity types, entity attributes, and entity relationship types. Related business data of the engineering project is acquired, and entity extraction and knowledge fusion are performed on the related business data according to the knowledge graph structure to generate a basic engineering project knowledge graph. The encoding of each knowledge node in the basic engineering project knowledge graph is optimized to construct an engineering project data association graph, which is then stored in the data hub module.

[0030] First, based on the business needs of engineering projects in terms of schedule management, quality management, cost control, and safety management, a knowledge graph structure for engineering projects is defined. This knowledge graph structure includes at least entity types representing business objects of the engineering project, entity attributes (information fields describing entity characteristics), and entity relationship types (describing the relationships between different entities). Based on this, business data related to the engineering project is acquired, and entity extraction and knowledge fusion are performed on the associated business data according to the knowledge graph structure. Entity extraction identifies various engineering project business entities from the business data, and knowledge fusion merges and disambiguates data from different data sources that point to the same business entity, thereby generating a basic engineering project knowledge graph. Subsequently, the knowledge nodes in the basic engineering project knowledge graph are encoded and optimized, mapping entities, attributes, and relationships into structured representations suitable for computation and rapid retrieval, constructing an engineering project data association graph. This constructed engineering project data association graph is stored in the data hub module for subsequent use when performing entity recognition, association analysis, and aggregation integration processing on available engineering project data.

[0031] The rule engine module invokes preset business logic association rules, and uses these rules to perform consistency comparison and logical verification on the integrated data of the project, generating data comparison and verification results.

[0032] After generating the integrated project data, the rule engine module invokes preset business logic association rules to perform consistency comparison and logical verification on the integrated project data. Specifically, the rule engine module pre-stores and manages multiple types of business logic association rules, which describe the rationality constraints and relationships of project data under different business dimensions. When the integrated project data is input to the rule engine module, the rule engine module performs consistency comparison and logical verification on the integrated project data item by item or in batches according to the preset rule execution order and execution mode. The consistency comparison is used to detect whether there are conflicts or inconsistencies in the data of the same project from different data sources or different business systems, and the logical verification is used to determine whether the project data meets the preset business logic association relationships. After the rules are executed, the rule engine module generates data comparison verification results based on the verification results of each rule. The data comparison verification results are used to characterize the verification status and verification conclusion of each data item in the integrated project data, and serve as the input basis for subsequent confidence assessment and anomaly handling.

[0033] Furthermore, the generated data comparison and verification results include: The preset business logic association rules include data consistency rules, business logic compliance rules, and abnormal data detection rules; the preset business logic association rules are grouped by business domain and the execution mode is set to obtain business logic domain association rules; based on the business logic domain association rules, consistency comparison and logic verification are performed on the integrated data of the engineering project to generate data comparison and verification results.

[0034] First, pre-defined business logic association rules are categorized and configured. These rules include at least data consistency rules for verifying data source consistency, business logic compliance rules for constraining the rationality of project business processes, and abnormal data detection rules for identifying anomalous or abnormal data patterns. Based on this, the pre-defined business logic association rules are grouped according to the different business domains involved in the project, and corresponding rule execution modes are configured for each business domain. These execution modes define the triggering conditions, execution order, and verification scope of the rules, thus forming business logic domain association rules. Subsequently, consistency comparison and logical verification are performed on the integrated data of the project based on these business logic domain association rules. Specifically, data consistency rules are used to compare and verify the data content of the same project across multiple data sources; business logic compliance rules are used to logically verify the business relationships of the project data; and abnormal data detection rules are used to identify abnormal data in the integrated data of the project. Finally, data comparison and verification results are generated based on the execution results of each type of rule. These results reflect the verification conclusions of the integrated data of the project in terms of consistency, compliance, and abnormality, and serve as input for subsequent confidence assessment and anomaly handling mechanisms.

[0035] Based on the data comparison and verification results, a weighted scoring and multi-threaded verification algorithm is used to simultaneously assess the confidence level of the integrated data of the engineering project, identify low-confidence project data, and trigger an anomaly handling mechanism based on the low-confidence project data to perform data processing and quality closed-loop management.

[0036] Specifically, firstly, based on the data comparison and verification results, the consistency verification results, logical verification results, and anomaly detection identifiers corresponding to each data item in the integrated project data are extracted and used as the basic input for confidence assessment. Then, a scoring model containing multiple confidence assessment dimensions is constructed, and corresponding weights are assigned to each confidence assessment dimension. The comprehensive confidence of each data item is calculated using a weighted scoring method. Simultaneously, a multi-threaded verification algorithm is used to process the integrated project data in parallel, dividing the data into multiple data subsets and performing confidence calculations synchronously to improve the efficiency of confidence assessment. After completing the confidence assessment, the integrated project data is filtered based on the obtained confidence results, identifying data with confidence levels below a preset threshold, thus determining low-confidence project data. When low-confidence project data is identified, an anomaly handling mechanism is triggered, performing data processing operations such as anomaly marking, backtracking analysis, or re-collection on the low-confidence project data. The processing results are then fed back to the data central module, forming a closed-loop data quality control process that links data verification, assessment, and correction.

[0037] Furthermore, identifying low-confidence project data includes: Based on the data comparison and verification results, the integrated data of the engineering projects is filtered to obtain the engineering project data to be evaluated; a weighted scoring and multi-threaded verification algorithm is used to evaluate the confidence level of the engineering project data to be evaluated, and a confidence level set of the engineering project data is obtained; according to the confidence level set of the engineering project data, the data in the engineering project data to be evaluated that are lower than the preset confidence threshold are identified and marked to determine the low-confidence project data.

[0038] First, based on the data comparison and verification results, the integrated data of the engineering projects is filtered to remove data that fails to meet the preset rules in consistency comparison, logical verification, or anomaly detection. This forms the engineering project data to be evaluated. Next, a weighted scoring and multi-threaded verification algorithm is used to assess the confidence level of the engineering project data. The multi-threaded verification algorithm divides the data into multiple parallel processing units to synchronously calculate confidence levels. A weighted scoring method is then used to comprehensively calculate the confidence level of each dimension, resulting in a confidence set for the engineering project data. Finally, based on the confidence set, data with confidence levels below a preset confidence threshold are identified and marked. These low-confidence project data will be the focus of subsequent anomaly handling mechanisms and data quality closed-loop management.

[0039] Furthermore, obtaining the confidence set of engineering project data includes: The project data to be evaluated is divided into blocks based on a multi-threaded verification algorithm to obtain multi-threaded project data blocks; a weighted scoring method is used to perform parallel confidence scoring on the multi-threaded project data blocks to obtain a project data confidence set.

[0040] First, the project data to be evaluated is divided into blocks based on a multi-threaded verification algorithm. According to data size, data type, or business relationships, the project data is divided into multiple independent project data blocks, and corresponding thread resources are allocated to each block, forming multi-threaded project data blocks for parallel processing. Then, a weighted scoring method is used to synchronously score the confidence level of the multi-threaded project data blocks in parallel. Specifically, for each project data block, the verification results of each data item in the block are comprehensively weighted according to pre-defined confidence evaluation dimensions and corresponding weights to obtain the corresponding confidence score value. Finally, the confidence score results output by each thread are summarized to form a project data confidence set, which is used to characterize the credibility of the project data in terms of overall consistency, logical rationality, and anomaly risk.

[0041] Furthermore, a weighted scoring method is used to synchronously perform parallel confidence scoring on the multi-threaded project data blocks to obtain a project data confidence set, including: Construct confidence assessment dimensions and assign weights to these dimensions to determine multi-dimensional weight decision coefficients. Based on the confidence assessment dimensions and the multi-dimensional weight decision coefficients, synchronously perform confidence-weighted scoring on the multi-threaded project data blocks to obtain a confidence set for the project data.

[0042] First, confidence assessment dimensions are constructed based on the business characteristics and verification requirements of the project data. These dimensions include at least data consistency, business logic compliance, and anomaly risk. After constructing these dimensions, weights are assigned to each dimension, determining the corresponding weight values ​​based on the impact of different dimensions on the credibility of the project data, thus forming multi-dimensional weight decision coefficients. Subsequently, based on the confidence assessment dimensions and multi-dimensional weight decision coefficients, a weighted confidence score is simultaneously performed on multi-threaded project data blocks. In each thread, data items in the project data block are calculated in parallel according to the weights corresponding to each confidence assessment dimension, and the scores of each assessment dimension are weighted and summarized to obtain the corresponding confidence score value. Finally, the confidence score results output by each thread are uniformly aggregated to obtain a project data confidence set, which characterizes the overall credibility of the project data under the multi-dimensional assessment dimensions.

[0043] In summary, the embodiments of this application have at least the following technical effects: First, a cloud-edge-device collaborative architecture is constructed, comprising a cloud platform and edge nodes. A data hub module and a rules engine module are deployed on the cloud, while a lightweight data proxy module is deployed on the edge nodes. Next, the lightweight data proxy module collects multi-source heterogeneous engineering project data in real time and transmits this encrypted data to the data hub module for aggregation and integration, resulting in integrated engineering project data. Then, the rules engine module invokes preset business logic association rules to perform consistency comparison and logical verification on the integrated engineering project data, generating data comparison and verification results. Finally, based on the data comparison and verification results, a weighted scoring and multi-threaded verification algorithm is used to synchronously assess the confidence level of the integrated engineering project data, identifying low-confidence project data and triggering an anomaly handling mechanism for data processing and closed-loop quality control. This approach solves the technical problems of difficulty in real-time synchronization of multi-source heterogeneous engineering project data, low integration processing efficiency, and difficulty in effectively controlling data quality in existing technologies, achieving the technical effects of improving the efficiency of engineering project data synchronization and integration processing, data credibility, and overall data quality controllability.

[0044] Example 2, based on the same inventive concept as the real-time synchronization processing method for engineering project data under multi-source heterogeneous integration in the foregoing examples, such as... Figure 2 As shown, this application provides a real-time synchronization and processing system for engineering project data under multi-source heterogeneous integration, wherein the system includes: Architecture Building Component 11: Constructs a cloud-edge-device collaborative architecture, which includes a cloud and edge nodes. A data hub module and a rule engine module are deployed on the cloud, and a lightweight data proxy module is deployed on the edge nodes. Data Processing Component 12: Collects multi-source heterogeneous engineering project data in real time through the lightweight data proxy module, and transmits the multi-source heterogeneous engineering project data to the data hub module for aggregation and integration processing to obtain integrated engineering project data. Data Verification Component 13: According to the rule engine module, it calls preset business logic association rules and uses the preset business logic association rules to perform consistency comparison and logical verification on the integrated engineering project data to generate data comparison and verification results. Data Evaluation Component 14: Based on the data comparison and verification results, it uses a weighted scoring and multi-threaded verification algorithm to synchronously evaluate the confidence level of the integrated engineering project data, identifies low-confidence project data, and triggers an anomaly handling mechanism based on the low-confidence project data for data processing and quality closed-loop control.

[0045] Furthermore, the data processing component 12 is used to perform the following methods: The information of each node in the edge nodes is initialized and acquisition tasks are assigned to obtain an edge node acquisition task assignment list; according to the lightweight data proxy module, a data configuration acquisition strategy is obtained, which includes timed acquisition and event-driven acquisition; the lightweight data proxy module monitors the project data in real time based on the edge node acquisition task assignment list and the data configuration acquisition strategy to acquire initial project data; the initial project data is identified by acquisition source, timestamp, and format identifier to obtain multi-source heterogeneous project data.

[0046] Furthermore, the data processing component 12 is used to perform the following methods: The multi-source heterogeneous engineering project data is encrypted and transmitted to the data hub module for integrity verification to obtain a data transmission verification result. If the data transmission verification result is complete, the data hub module obtains data preprocessing steps, including data cleaning and structured format conversion. Based on the data preprocessing steps, the multi-source heterogeneous engineering project data is preprocessed to obtain usable engineering project data. The data hub module then aggregates and integrates the usable engineering project data to obtain integrated engineering project data.

[0047] Furthermore, the data processing component 12 is used to perform the following methods: The data hub module invokes the engineering project data association map; based on the engineering project data association map, entity identification and association analysis are performed on the available engineering project data to obtain engineering project associated entity data; the engineering project associated entity data is then subjected to time-series alignment and aggregation integration processing to obtain integrated engineering project data.

[0048] Furthermore, the data processing component 12 is used to perform the following methods: Based on the business requirements of the engineering project, a knowledge graph structure for the engineering project is defined, which includes entity types, entity attributes, and entity relationship types. Related business data of the engineering project is acquired, and entity extraction and knowledge fusion are performed on the related business data according to the knowledge graph structure to generate a basic engineering project knowledge graph. The encoding of each knowledge node in the basic engineering project knowledge graph is optimized to construct an engineering project data association graph, which is then stored in the data hub module.

[0049] Furthermore, the data verification component 13 is used to perform the following methods: The preset business logic association rules include data consistency rules, business logic compliance rules, and abnormal data detection rules; the preset business logic association rules are grouped by business domain and the execution mode is set to obtain business logic domain association rules; based on the business logic domain association rules, consistency comparison and logic verification are performed on the integrated data of the engineering project to generate data comparison and verification results.

[0050] Furthermore, the data evaluation component 14 is used to perform the following methods: Based on the data comparison and verification results, the integrated data of the engineering projects is filtered to obtain the engineering project data to be evaluated; a weighted scoring and multi-threaded verification algorithm is used to evaluate the confidence level of the engineering project data to be evaluated, and a confidence level set of the engineering project data is obtained; according to the confidence level set of the engineering project data, the data in the engineering project data to be evaluated that are lower than the preset confidence threshold are identified and marked to determine the low-confidence project data.

[0051] Furthermore, the data evaluation component 14 is used to perform the following methods: The project data to be evaluated is divided into blocks based on a multi-threaded verification algorithm to obtain multi-threaded project data blocks; a weighted scoring method is used to perform parallel confidence scoring on the multi-threaded project data blocks to obtain a project data confidence set.

[0052] Furthermore, the data evaluation component 14 is used to perform the following methods: Construct confidence assessment dimensions and assign weights to these dimensions to determine multi-dimensional weight decision coefficients. Based on the confidence assessment dimensions and the multi-dimensional weight decision coefficients, synchronously perform confidence-weighted scoring on the multi-threaded project data blocks to obtain a confidence set for the project data.

[0053] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some modifications or alterations to the above-disclosed technical content to create equivalent embodiments without departing from the scope of the present invention. Any modifications, equivalent changes, and alterations made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the scope of the present invention.

Claims

1. A method for real-time synchronization and processing of engineering project data under multi-source heterogeneous integration, characterized in that, The method includes: A cloud-edge-device collaborative architecture is constructed, which includes a cloud and edge nodes. A data hub module and a rule engine module are deployed on the cloud, and a lightweight data proxy module is deployed on the edge nodes. The lightweight data agent module collects multi-source heterogeneous engineering project data in real time, and encrypts and transmits the multi-source heterogeneous engineering project data to the data hub module for aggregation and integration processing to obtain integrated engineering project data. According to the rule engine module, a preset business logic association rule is invoked, and the preset business logic association rule is used to perform consistency comparison and logical verification on the integrated data of the project, and generate data comparison and verification results. Based on the data comparison and verification results, a weighted scoring and multi-threaded verification algorithm is used to simultaneously assess the confidence level of the integrated data of the engineering project, identify low-confidence project data, and trigger an anomaly handling mechanism based on the low-confidence project data to perform data processing and quality closed-loop management.

2. The method for real-time synchronization processing of engineering project data under multi-source heterogeneous integration as described in claim 1, characterized in that, The lightweight data proxy module collects multi-source heterogeneous engineering project data in real time, including: The information of each node in the edge node is initialized and the acquisition task is assigned to obtain the edge node acquisition task assignment list; According to the lightweight data proxy module, the data configuration collection strategy is obtained, which includes timed collection and event-driven collection. The lightweight data proxy module monitors the project data in real time based on the edge node acquisition task allocation list and the data configuration acquisition strategy, and collects the initial project data. The initial project data is collected by identifying its source, timestamp, and format to obtain multi-source heterogeneous project data.

3. The method for real-time synchronization processing of engineering project data under multi-source heterogeneous integration as described in claim 1, characterized in that, Obtain integrated data for the engineering project, including: The multi-source heterogeneous engineering project data is encrypted and transmitted to the data hub module for integrity verification, and the data transmission verification result is obtained. If the data transmission verification result is complete, the data preprocessing steps, including data cleaning and structured format conversion, are obtained through the data hub module. Based on the data preprocessing steps, the multi-source heterogeneous engineering project data is preprocessed to obtain usable engineering project data; The available engineering project data is aggregated and integrated through the data hub module to obtain integrated engineering project data.

4. The method for real-time synchronization processing of engineering project data under multi-source heterogeneous integration as described in claim 3, characterized in that, The available engineering project data is aggregated and integrated through the data hub module to obtain integrated engineering project data, including: The data hub module is used to access the data association map of the engineering project; Based on the engineering project data association map, entity identification and association analysis are performed on the available engineering project data to obtain engineering project associated entity data; The associated entity data of the project is subjected to time-series alignment and aggregation integration processing to obtain integrated project data.

5. The method for real-time synchronization processing of engineering project data under multi-source heterogeneous integration as described in claim 4, characterized in that, The data hub module invokes the project data association map, including: Based on the business requirements of the engineering project, a knowledge graph structure for the engineering project is defined, which includes entity types, entity attributes, and entity relationship types. Obtain relevant business data for engineering projects, and perform entity extraction and knowledge fusion on the relevant business data for engineering projects according to the knowledge graph structure of the engineering projects to generate a basic engineering project knowledge graph; The knowledge nodes in the basic engineering project knowledge graph are encoded and optimized to construct an engineering project data association graph, and the engineering project data association graph is stored in the data hub module.

6. The method for real-time synchronization processing of engineering project data under multi-source heterogeneous integration as described in claim 1, characterized in that, Generate data comparison and verification results, including: The preset business logic association rules include data consistency rules, business logic compliance rules, and abnormal data detection rules; The preset business logic association rules are grouped by business domain and the execution mode is set to obtain business logic domain association rules; Based on the business logic domain association rules, consistency comparison and logical verification are performed on the integrated data of the engineering project to generate data comparison and verification results.

7. The method for real-time synchronization processing of engineering project data under multi-source heterogeneous integration as described in claim 1, characterized in that, Identify low-confidence project data, including: Based on the data comparison and verification results, the integrated data of the engineering projects is filtered to obtain the data of the engineering projects to be evaluated. A weighted scoring and multi-threaded verification algorithm is used to assess the confidence level of the project data to be evaluated, thereby obtaining a confidence set of the project data. According to the confidence set of the project data, the data in the project data to be evaluated that are lower than the preset confidence threshold are identified and marked to determine the low-confidence project data.

8. The method for real-time synchronization processing of engineering project data under multi-source heterogeneous integration as described in claim 7, characterized in that, Obtain the confidence set of the engineering project data, including: The data of the project to be evaluated is divided into blocks based on a multi-threaded verification algorithm to obtain multi-threaded project data blocks. The confidence scores of the multi-threaded project data blocks are scored in parallel using a weighted scoring method to obtain a confidence set of the project data.

9. The method for real-time synchronization processing of engineering project data under multi-source heterogeneous integration as described in claim 8, characterized in that, A weighted scoring method is used to synchronously perform parallel confidence scoring on the multi-threaded project data blocks to obtain a confidence set of the project data, including: Construct confidence assessment dimensions, assign weights to the confidence assessment dimensions, and determine the multi-dimensional weight decision coefficients; Based on the confidence assessment dimensions and the multi-dimensional weight decision coefficients, the confidence weight score of the multi-threaded engineering project data block is performed synchronously to obtain the confidence set of the engineering project data.

10. A real-time synchronization processing system for engineering project data under multi-source heterogeneous integration, characterized in that, The system is used to implement the real-time synchronization processing method for engineering project data under multi-source heterogeneous integration as described in any one of claims 1-9, the system comprising: Architecture building components: Constructing a cloud-edge-device collaborative architecture, which includes a cloud and edge nodes, deploying a data hub module and a rule engine module on the cloud, and deploying a lightweight data proxy module on the edge nodes; Data processing component: The lightweight data agent module collects multi-source heterogeneous engineering project data in real time, and encrypts and transmits the multi-source heterogeneous engineering project data to the data hub module for aggregation and integration processing to obtain integrated engineering project data; Data verification component: Based on the rule engine module, it calls preset business logic association rules, uses the preset business logic association rules to perform consistency comparison and logical verification on the integrated data of the project, and generates data comparison and verification results; Data evaluation component: Based on the data comparison and verification results, a weighted scoring and multi-threaded verification algorithm are used to synchronously evaluate the confidence level of the integrated data of the engineering project, identify low-confidence project data, and trigger an anomaly handling mechanism based on the low-confidence project data to perform data processing and quality closed-loop control.

Images