Big data system based on building industry multi-data source project information unification

Abstract

The invention discloses a big data system based on building industry multi-data source project information unification, and relates to the technical field of big data systems. The system comprises a data acquisition layer which is directly connected with an external data source and is responsible for acquiring data and providing original materials for subsequent processing; the data preprocessing and cleaning layer is used for receiving data input of the data acquisition layer and carrying out primary processing on original data; the data storage layer is used for receiving the data of the data preprocessing and cleaning layer, storing the cleaned data, providing a basis for data analysis and supporting the storage requirement of large-scale data; the data processing layer is used for directly reading the data of the data storage layer and deeply analyzing the stored data; the big data system provided by the invention solves the problem of project information management in the building industry through improvements in the aspects of data unified management, data integrated analysis, decision-making efficiency improvement, data security guarantee and the like, and brings better effects and positive influences.

Description

Technical Field

[0001] The present invention relates to the technical field of big data systems, and in particular to a big data system based on the unification of project information from multiple data sources in the construction industry. Background Art

[0002] In the construction industry, information management and data processing have become important means to improve efficiency and quality. Existing technologies mainly include three forms: building information modeling (BIM), construction management system and Internet of Things (IoT) technology.

[0003] The data in the BIM system is mainly concentrated in the design and construction stage, while the data in the operation and maintenance stage are difficult to integrate. In addition, the data formats between different BIM software are not unified, which makes data sharing and exchange difficult. The data of the construction management system are usually scattered in different subsystems, making it difficult to achieve centralized management. A large amount of construction data has not been effectively analyzed and utilized, and the potential value of the data has not been fully utilized. The data generated by devices using the Internet of Things (IoT) technology are of various types and in non-uniform formats, making it difficult to integrate with data from other systems. In addition, the existing system has performance bottlenecks when processing and analyzing IoT data. Therefore, the existing technologies and systems have many problems and shortcomings in data collection, storage, management, analysis and visualization. There is an urgent need for a big data system that can integrate multiple data sources and achieve unified data management and efficient utilization to solve these problems.

[0004] Therefore, a big data system based on the unification of project information from multiple data sources in the construction industry is proposed. Summary of the invention

[0005] The purpose of the present invention is to solve the problems mentioned in the above background technology. The present invention provides a big data system based on the unification of project information from multiple data sources in the construction industry.

[0006] In order to achieve the above-mentioned purpose, the present invention specifically adopts the following technical solutions:

[0007] A big data system based on the unification of project information from multiple data sources in the construction industry, including:

[0008] Data collection layer: directly connects with external data sources, is responsible for acquiring data and providing raw materials for subsequent processing;

[0009] Data preprocessing and cleaning layer: receives data input from the data collection layer and performs preliminary processing on the raw data;

[0010] Data storage layer: receives data from the data preprocessing and cleaning layer, and stores the cleaned data, providing a basis for data analysis and supporting the storage needs of large-scale data;

[0011] Data processing layer: directly reads data from the data storage layer and conducts in-depth analysis of the stored data;

[0012] Business application layer: converts the analysis results of the data processing layer into actionable business guidance or strategies, and implements specific business functions based on the data processing results;

[0013] Visualization and interaction layer: receives data output from the business application layer, intuitively displays analysis results, and provides a human-computer interface for end users;

[0014] Security and compliance layer: runs through the entire system to ensure data security and compliance;

[0015] System management and operation and maintenance layer: comprehensive management of all underlying architectures and upper-layer applications.

[0016] Furthermore, the data sources of the data collection layer include BIM system, ERP, CRM, IoT devices, drones and manual entry.

[0017] Furthermore, the data storage layer includes a distributed file system, a relational database, a NoSQL database and a data warehouse.

[0018] Furthermore, the data processing layer includes a real-time data processing engine and a batch processing framework.

[0019] Furthermore, the business application layer includes an artificial intelligence application layer and a business logic layer, the business logic layer includes project cost forecasting, schedule optimization algorithms and risk assessment models, and the artificial intelligence application layer includes machine learning models and deep learning predictions.

[0020] Furthermore, the data preprocessing and cleaning layer includes data cleaning (removing invalid, duplicate, and erroneous data), format conversion, standardization processing, etc. to ensure data quality.

[0021] The beneficial effects of the present invention are as follows:

[0022] 1. Through this system, the construction industry can unify the management of project information from different data sources and realize the centralized storage and management of data. Compared with the traditional manual integration method, this system can automatically integrate information from multiple data sources, greatly improving the efficiency and accuracy of data management.

[0023] 2. This system provides powerful data analysis and mining functions, which can conduct in-depth analysis and comprehensive evaluation of project information. Users can quickly obtain key indicators and trend analysis through the system to help decision makers make accurate decisions and plans.

[0024] 3. Through this system, decision makers in the construction industry can obtain project information more quickly and conduct accurate analysis and evaluation. This will greatly improve the efficiency and accuracy of decision-making and help companies better plan and manage projects.

[0025] 4. This system uses advanced data security technology to ensure the security and confidentiality of project information during transmission and storage. Through measures such as authority management and data encryption, project information is protected from access and disclosure by unauthorized personnel.

[0026] 5. Through this system, the construction industry can achieve comprehensive management and comprehensive analysis of project information, helping enterprises to better grasp the progress of projects, optimize resource allocation, improve construction efficiency, etc. This will help improve project management level and enhance corporate competitiveness. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 It is a schematic diagram of the system structure of the present invention; DETAILED DESCRIPTION

[0028] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0029] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0030] It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in the subsequent drawings. In addition, the terms "first", "second", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0031] In the description of the embodiments of the present invention, it should be noted that the terms "inside", "outside", "upper", etc. indicate orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, or are the orientations or positional relationships in which the inventive product is usually placed when used. They are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention.

[0032] like Figure 1 As shown, a big data system based on the unification of project information from multiple data sources in the construction industry includes:

[0033] Data collection layer: directly connects with external data sources, is responsible for acquiring data and providing raw materials for subsequent processing; by designing a highly scalable data collection module, through API interface, crawler technology, file import, etc., it collects data from multiple channels such as project management software (such as BIM system, ERP, CRM), Internet of Things sensors (monitoring construction environment, equipment status), drone mapping, handheld terminal reports, etc.

[0034] Data preprocessing and cleaning layer: Receive data input from the data collection layer and perform preliminary processing on the original data; ensure data quality by preprocessing the collected data, including removing redundancy, correcting errors, and unifying formats and units. Establish a unified data model and coding system to effectively integrate data from different sources.

[0035] Data storage layer: Receive data from the data preprocessing and cleaning layer, and store the cleaned data to provide a basis for data analysis and support the storage needs of large-scale data; use big data storage technology such as Hadoop HDFS or cloud storage services to build a highly available and scalable data warehouse to adapt to the characteristics of large data volumes and rapid growth in construction projects. In the specific implementation, a metadata management system can also be established to record metadata such as data source, data structure, data quality, etc., to facilitate data tracking, understanding and use.

[0036] Data processing layer: directly reads data from the data storage layer and conducts in-depth analysis on the stored data; including real-time monitoring, trend prediction, pattern recognition, etc.

[0037] Business application layer: converts the analysis results of the data processing layer into actionable business guidance or strategies, and implements specific business functions based on the data processing results; such as cost control, schedule management, risk assessment, supply chain optimization, etc.

[0038] Visualization and interaction layer: Receive data output from the business application layer, intuitively display analysis results, and provide a human-computer interface for end users; in the actual operation process, use charts, 3D models, etc. to display key indicators such as project progress, cost distribution, and safety compliance. Build Web or mobile applications so that project managers, engineers, suppliers and other parties can view, query and interact with data anytime and anywhere to promote collaboration and decision-making.

[0039] Security and compliance layer: It runs through the entire system to ensure data security and compliance; including data encryption, access control, audit tracking and other functions. It monitors and protects every link of data flow. It implements strict data encryption measures to ensure the security of data transmission and storage. It sets up multi-level access rights management to protect sensitive information. It complies with relevant data protection regulations (such as GDPR) to ensure that data processing activities are legal and compliant.

[0040] System management and operation layer: Comprehensive management of all underlying architectures and upper-layer applications. Responsible for system monitoring, maintenance, performance optimization, version updates, etc. to ensure stable operation of the system. In specific implementation, a user feedback system can also be established to continuously collect user needs and suggestions, and continuously optimize system functions and user experience. Regularly review system performance, and upgrade and optimize the system according to business growth and technological progress.

[0041] The data sources of the data collection layer include BIM systems, ERP, CRM, IoT devices, drones, and manual entry. After the data is acquired, it can be input through APIs, database connections, file imports, etc.

[0042] The data storage layer includes distributed file systems, relational databases, NoSQL databases, and data warehouses.

[0043] The data processing layer includes a real-time data processing engine and a batch processing framework. The real-time data processing engine uses Apache Kafka and Flink to process real-time data streams, such as construction site monitoring video analysis and environmental parameter warnings; the batch processing framework uses Spark for offline analysis.

[0044] The business application layer includes the artificial intelligence application layer and the business logic layer. The business logic layer includes project cost forecasting, schedule optimization algorithms, and risk assessment models. The artificial intelligence application layer includes machine learning models and deep learning forecasting. By integrating machine learning and deep learning algorithms, pattern recognition, anomaly detection, and intelligent recommendations can be performed. For example, price fluctuations of building materials can be predicted based on historical project data to optimize supply chain management.

[0045] The data preprocessing and cleaning layer includes data cleaning (removing invalid, duplicate, and erroneous data), format conversion, standardization processing, etc. to ensure data quality.

[0046] In summary: Through this system, the construction industry can unify the management of project information from different data sources and realize centralized storage and management of data. Compared with the traditional manual integration method, this system can automatically integrate information from multiple data sources, greatly improving the efficiency and accuracy of data management. This system provides powerful data analysis and mining functions, which can conduct in-depth analysis and comprehensive evaluation of project information. Users can quickly obtain key indicators and trend analysis through the system to help decision makers make accurate decisions and plans. Through this system, decision makers in the construction industry can obtain project information more quickly and conduct accurate analysis and evaluation. This will greatly improve the efficiency and accuracy of decision-making and help companies better plan and manage projects.

[0047] This system uses advanced data security technology to ensure the security and confidentiality of project information during transmission and storage. Through measures such as permission management and data encryption, project information is protected from access and leakage by unauthorized personnel. Through this system, the construction industry can achieve comprehensive management and comprehensive analysis of project information, helping companies to better understand project progress, optimize resource allocation, and improve construction efficiency. This will help improve project management and enhance corporate competitiveness.

[0048] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The above embodiments and descriptions only describe the principles of the present invention. The present invention may be subject to various changes and improvements without departing from the spirit and scope of the present invention. These changes and improvements fall within the scope of the present invention. The scope of protection claimed by the present invention is defined by the attached claims and their equivalents.

Claims

1. A big data system based on the unification of project information from multiple data sources in the construction industry, characterized in that: include: Data collection layer: directly connects with external data sources, is responsible for acquiring data and providing raw materials for subsequent processing; Data preprocessing and cleaning layer: receives data input from the data collection layer and performs preliminary processing on the raw data; Data storage layer: receives data from the data preprocessing and cleaning layer, and stores the cleaned data, providing a basis for data analysis and supporting the storage needs of large-scale data; Data processing layer: directly reads data from the data storage layer and conducts in-depth analysis of the stored data; Business application layer: converts the analysis results of the data processing layer into actionable business guidance or strategies, and implements specific business functions based on the data processing results; Visualization and interaction layer: receives data output from the business application layer, intuitively displays analysis results, and provides a human-computer interface for end users; Security and compliance layer: runs through the entire system to ensure data security and compliance; System management and operation and maintenance layer: comprehensive management of all underlying architectures and upper-layer applications.

2. A big data system based on unified project information from multiple data sources in the construction industry according to claim 1, characterized in that: The data sources of the data collection layer include BIM systems, ERP, CRM, IoT devices, drones and manual entry.

3. A big data system based on unified project information from multiple data sources in the construction industry according to claim 1, characterized in that: The data storage layer includes a distributed file system, a relational database, a NoSQL database and a data warehouse.

4. A big data system based on unified project information from multiple data sources in the construction industry according to claim 1, characterized in that: The data processing layer includes a real-time data processing engine and a batch processing framework.

5. The big data system based on unified project information from multiple data sources in the construction industry according to claim 1 is characterized in that: The business application layer includes an artificial intelligence application layer and a business logic layer. The business logic layer includes project cost forecasting, schedule optimization algorithms and risk assessment models. The artificial intelligence application layer includes machine learning models and deep learning forecasting.

6. A big data system based on unified project information from multiple data sources in the construction industry according to claim 2, characterized in that: The data preprocessing and cleaning layer includes data cleaning (removing invalid, duplicate, and erroneous data), format conversion, standardization processing, etc. to ensure data quality.

Images