Multi-source heterogeneous data acquisition method and system for construction site

Abstract

The invention relates to the technical field of construction engineering management, in particular to a multi-source heterogeneous data acquisition method and system for a construction site, and the method comprises the following steps: S1, data type recognition and classification; s2, data preprocessing; s3, data transmission and convergence; s4, performing data verification and quality evaluation; through various data acquisition terminals and classified acquisition strategies, structured, semi-structured and non-structured data in multiple aspects of personnel, equipment, environment, progress and the like on a construction site can be covered, comprehensive acquisition of the data is realized, and a rich data basis is provided for construction site management.

Description

Technical Field

[0001] The present invention relates to the technical field of construction project management, and in particular to a method and system for collecting multi-source heterogeneous data at a construction site. Background Art

[0002] Modern construction site management involves data on personnel, equipment, environment, and progress. This data comes from a wide range of sources and has diverse structures, including real-time monitoring data collected by sensors, structured data such as construction drawings, and unstructured data from video surveillance. Traditional data collection methods often target a single type of data and lack a unified collection and processing mechanism for multi-source, heterogeneous data. This results in fragmented data and an inability to effectively integrate and analyze it, making it difficult to meet the needs of refined construction site management, safety warnings, and progress control. For example, personnel attendance data and equipment operation data cannot be correlated and analyzed, affecting resource allocation efficiency; environmental monitoring data is disconnected from construction progress data and cannot provide timely support for construction decision-making. To this end, a method and system for collecting multi-source, heterogeneous data at construction sites is proposed. Summary of the Invention

[0003] In view of this, the present invention provides a method and system for collecting multi-source heterogeneous data at a construction site to solve or alleviate the technical problems existing in the prior art and at least provide a beneficial option.

[0004] The technical solution of the present invention is implemented as follows: a method for collecting multi-source heterogeneous data at a construction site, comprising the following steps:

[0005] S1. Data type identification and classification;

[0006] S2, data preprocessing;

[0007] S3, data transmission and aggregation;

[0008] S4. Data verification and quality assessment.

[0009] Further preferably, the S1 deploys a variety of data acquisition terminals at the construction site, including but not limited to sensors (such as temperature and humidity sensors, pressure sensors, displacement sensors), cameras, RFID readers, smart meters, etc. After each acquisition terminal collects the data, it first identifies the data type. According to the structural characteristics of the data, the data is divided into structured data (such as personnel attendance records, equipment operation parameter tables), semi-structured data (such as XML format documents of construction logs) and unstructured data (such as video surveillance images, PDF files of construction drawings). At the same time, according to the category of construction site information reflected by the data, it is further subdivided into personnel data, equipment data, environmental data, progress data, etc.

[0010] Further preferably, the S2 performs data cleaning on structured data to remove duplicate data and erroneous data (such as equipment operating parameters that exceed a reasonable range); interpolation methods (such as linear interpolation, Lagrange interpolation) or statistical model-based methods are used to fill in missing data; for semi-structured data, it is converted into a unified format through a parser, for example, a construction log in XML format is parsed into structured tabular data to extract key information (such as construction time, construction content, and person in charge); for unstructured data, key frames are extracted from video data, and image processing algorithms are used to identify key scenes in the video (such as illegal operations by personnel and abnormal operation of equipment); feature extraction is performed on files such as construction drawings to extract key information such as dimensions and materials in the drawings.

[0011] Further preferably, the S3 uses a variety of communication technologies to achieve data transmission. For short-distance data transmission, low-power wireless communication technologies such as Bluetooth and Zigbee are used to transmit the data collected by the sensor to the aggregation node; for long-distance data transmission, network communication technologies such as 4G / 5G and Wi-Fi are used to transmit the data from the aggregation node to the data center server. During the data transmission process, the data is encrypted and encryption algorithms such as AES (Advanced Encryption Standard) are used to ensure the security and privacy of the data. The data center server serves as a data aggregation point, receives data from various collection terminals and aggregation nodes, and stores them in categories according to data type and category.

[0012] Further preferably, the S4 establishes a data verification rule base to verify the collected data. For example, for equipment operation data, the normal range of parameters is set, and if the data exceeds the range, it is marked as abnormal data; for personnel attendance data, the rationality of the punch-in time is checked to prevent abnormal punch-in records. At the same time, a data quality assessment model is used to evaluate data quality from the dimensions of completeness (whether the data is missing key information), accuracy (whether the data is true and reliable), and consistency (the consistency of the same data in different data sources). A data quality report is generated, and feedback and re-collection are provided for data that does not meet the quality standards.

[0013] A multi-source heterogeneous data acquisition system for a construction site includes the following modules: a data acquisition module, a data processing module, a data transmission module, and a data management module.

[0014] Further preferably, the 1. data acquisition module is composed of a variety of data acquisition terminals and is responsible for collecting various data at the construction site. Specifically including:

[0015] Personnel data collection unit: This unit uses an RFID reader and a facial recognition camera. The RFID reader is used to read the identification card information worn by personnel to obtain personnel attendance, identity and other data. The facial recognition camera is used to identify personnel's facial features and, in combination with the access control system, implements personnel entry and exit management and identity authentication. At the same time, it can monitor personnel's working status (such as whether they are working fatigued) through video analysis technology.

[0016] Equipment data acquisition unit: Various sensors (such as vibration sensors, current sensors, and GPS positioning sensors) are installed on construction equipment to collect real-time data on the equipment's operating parameters (such as speed, temperature, and position) and working status (such as startup, shutdown, and faults). Furthermore, data such as the equipment's operation logs and maintenance records are obtained through the equipment's built-in communication interfaces (such as RS-485 and Ethernet interfaces).

[0017] Environmental data collection unit: Deploy temperature and humidity sensors, air quality sensors (to detect PM2.5, PM10, and harmful gas concentrations), noise sensors, rainfall sensors, etc. to collect environmental parameters at the construction site in real time, providing data support for construction safety and personnel health;

[0018] Progress data collection unit: Utilizes laser rangefinders, drone aerial photography, and other equipment to collect data on topographic changes in the construction area, the construction progress of buildings, and other data. At the same time, combined with construction drawings and plans, image recognition and data analysis technology are used to compare the actual construction progress with the planned progress and generate a progress report.

[0019] Further preferably, the data processing module pre-processes the collected data, specifically including:

[0020] Structured data processing submodule: Executes data cleaning and missing value processing algorithms. By establishing a data rule library (such as normal range rules for equipment parameters and personnel attendance time rules), it screens and repairs structured data to ensure data accuracy and completeness.

[0021] Semi-structured data processing submodule: It has built-in parsers for various formats (such as XML parser and JSON parser), which converts semi-structured data into structured data, extracts key information, and stores it in the corresponding database table.

[0022] Unstructured Data Processing Submodule: This module uses image processing, natural language processing, and other technologies to perform feature extraction and content analysis on unstructured data such as videos, images, and documents. For example, for video data, object detection algorithms (such as the YOLO algorithm) are used to identify people, equipment, and scenes in the video. For construction drawings, OCR (Optical Character Recognition) technology is used to extract text information, combined with drawing vectorization technology to extract graphic information.

[0023] Further preferably, the 1. Data transmission module, which enables data transmission from the acquisition terminal to the data center server, includes: a short-range transmission unit: which uses wireless communication technologies such as Bluetooth and Zigbee to build a wireless sensor network at the construction site and transmit sensor data distributed at different locations to the aggregation node. This unit has the characteristics of low power consumption and self-organizing network, can adapt to the complex environment of the construction site, and ensure stable data transmission;

[0024] Long-distance transmission unit: Utilizes network communication technologies such as 4G / 5G and Wi-Fi to transmit data from the aggregation node to the data center server. This unit supports real-time data transmission and resumable transmission. In the event of unstable network signals, it can cache data and automatically transmit it after the network is restored to ensure data loss. At the same time, it uses data compression technology (such as GZIP compression) to reduce data transmission volume and improve transmission efficiency.

[0025] Further preferably, the data storage submodule adopts a distributed storage architecture, combining relational databases (such as MySQL) and non-relational databases (such as MongoDB, HBase) to classify and store different types of data. Structured data is stored in relational databases, which facilitates complex queries and analysis; semi-structured and unstructured data is stored in non-relational databases, which can efficiently handle large-scale, highly concurrent data read and write operations. At the same time, a data backup and recovery mechanism is established to regularly back up data to prevent data loss.

[0026] Data verification submodule: Based on the preset data verification rule base, it performs real-time verification on the data stored in the database. By setting triggers and stored procedures, it automatically detects the legitimacy and integrity of the data, and promptly marks abnormal data and notifies relevant personnel to handle it.

[0027] Data quality assessment submodule: Use the data quality assessment model to regularly assess the quality of data in the database, generate data quality reports by calculating indicators such as data integrity, accuracy, and consistency, and make data optimization suggestions based on the report results, such as re-collecting low-quality data and improving the data collection process.

[0028] The embodiment of the present invention adopts the above technical solution, which has the following advantages:

[0029] 1. Through multiple data collection terminals and classified collection strategies, the present invention can cover structured, semi-structured and unstructured data on construction site personnel, equipment, environment, progress and other aspects, realize comprehensive data collection, and provide a rich data foundation for construction site management.

[0030] 2. The data preprocessing and verification process, as well as the data quality assessment mechanism of the present invention, can effectively remove erroneous data, fill in missing data, and convert data formats to ensure that the collected data is accurate, complete, and consistent, improve the availability and reliability of the data, and provide strong support for subsequent in-depth data analysis and decision-making.

[0031] 3. The present invention adopts a variety of communication technologies and encryption algorithms to achieve stable and secure data transmission at different transmission distances, prevent data from being stolen or tampered with during transmission, and protect the privacy and security of construction site data.

[0032] 4. The system of the present invention adopts a modular design, and each module has clear functions and cooperates with each other, which facilitates the expansion and maintenance of the system, improves the flexibility and adaptability of the system, and can meet the ever-changing data collection needs of the construction site.

[0033] The above summary is for illustrative purposes only and is not intended to be limiting in any way. In addition to the illustrative aspects, embodiments and features described above, further aspects, embodiments and features of the present invention will be readily apparent by reference to the accompanying drawings and the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0035] Figure 1 is a flow chart of the method of the present invention;

[0036] Figure 2 It is a system module diagram of the present invention. DETAILED DESCRIPTION

[0037] Hereinafter, only certain exemplary embodiments are briefly described. As will be appreciated by those skilled in the art, the described embodiments may be modified in various ways without departing from the spirit or scope of the present invention. Therefore, the drawings and description are to be considered as illustrative in nature and not restrictive.

[0038] The embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0039] like Figure 1-2 As shown, an embodiment of the present invention provides a method for collecting multi-source heterogeneous data at a construction site, comprising the following steps:

[0040] S1. Data type identification and classification;

[0041] S2, data preprocessing;

[0042] S3, data transmission and aggregation;

[0043] S4. Data verification and quality assessment.

[0044] In one embodiment, S1 deploys a variety of data acquisition terminals at the construction site, including but not limited to sensors (such as temperature and humidity sensors, pressure sensors, displacement sensors), cameras, RFID readers, smart meters, etc. After each acquisition terminal collects the data, it first identifies the data type. According to the structural characteristics of the data, the data is divided into structured data (such as personnel attendance records, equipment operation parameter tables), semi-structured data (such as XML format documents of construction logs) and unstructured data (such as video surveillance images, PDF files of construction drawings). At the same time, according to the category of construction site information reflected by the data, it is further subdivided into personnel data, equipment data, environmental data, progress data, etc.

[0045] In one embodiment, S2 performs data cleaning on structured data to remove duplicate data and erroneous data (such as equipment operating parameters that exceed a reasonable range); interpolation methods (such as linear interpolation and Lagrange interpolation) or statistical model-based methods are used to fill in missing data; for semi-structured data, it is converted into a unified format through a parser, for example, a construction log in XML format is parsed into structured tabular data to extract key information (such as construction time, construction content, and person in charge); for unstructured data, key frames are extracted from video data, and image processing algorithms are used to identify key scenes in the video (such as illegal operations by personnel and abnormal operation of equipment); feature extraction is performed on files such as construction drawings to extract key information such as dimensions and materials in the drawings.

[0046] In one embodiment, S3 uses multiple communication technologies to achieve data transmission. For short-distance data transmission, low-power wireless communication technologies such as Bluetooth and Zigbee are used to transmit the data collected by the sensor to the aggregation node. For long-distance data transmission, network communication technologies such as 4G / 5G and Wi-Fi are used to transmit the data from the aggregation node to the data center server. During the data transmission process, the data is encrypted using encryption algorithms such as AES (Advanced Encryption Standard) to ensure data security and privacy. The data center server serves as a data aggregation point, receiving data from various collection terminals and aggregation nodes, and classifying and storing them according to data type and category.

[0047] In one embodiment, S4 establishes a data verification rule base to verify the collected data. For example, for equipment operation data, the normal range of parameters is set, and if the data exceeds the range, it is marked as abnormal data. For personnel attendance data, the rationality of punch-in time is checked to prevent abnormal punch-in records. At the same time, a data quality assessment model is used to evaluate data quality from the dimensions of completeness (whether the data is missing key information), accuracy (whether the data is authentic and reliable), and consistency (the consistency of the same data in different data sources). A data quality report is generated, and feedback is provided and re-collection is carried out for data that does not meet the quality standards.

[0048] A multi-source heterogeneous data acquisition system for a construction site includes the following modules: a data acquisition module, a data processing module, a data transmission module, and a data management module.

[0049] In one embodiment, 1. Data acquisition module: consists of multiple data acquisition terminals and is responsible for collecting various data on the construction site. Specifically, it includes:

[0050] Personnel data collection unit: This unit uses an RFID reader and a facial recognition camera. The RFID reader is used to read the identification card information worn by personnel to obtain personnel attendance, identity and other data. The facial recognition camera is used to identify personnel's facial features and, in combination with the access control system, implements personnel entry and exit management and identity authentication. At the same time, it can monitor personnel's working status (such as whether they are working fatigued) through video analysis technology.

[0051] Equipment data acquisition unit: Various sensors (such as vibration sensors, current sensors, and GPS positioning sensors) are installed on construction equipment to collect real-time data on the equipment's operating parameters (such as speed, temperature, and position) and working status (such as startup, shutdown, and faults). Furthermore, data such as the equipment's operation logs and maintenance records are obtained through the equipment's built-in communication interfaces (such as RS-485 and Ethernet interfaces).

[0052] Environmental data collection unit: Deploy temperature and humidity sensors, air quality sensors (to detect PM2.5, PM10, and harmful gas concentrations), noise sensors, rainfall sensors, etc. to collect environmental parameters at the construction site in real time, providing data support for construction safety and personnel health;

[0053] Progress data collection unit: Utilizes laser rangefinders, drone aerial photography, and other equipment to collect data on topographic changes in the construction area, the construction progress of buildings, and other data. At the same time, combined with construction drawings and plans, image recognition and data analysis technology are used to compare the actual construction progress with the planned progress and generate a progress report.

[0054] In one embodiment, the data processing module pre-processes the collected data, specifically including:

[0055] Structured data processing submodule: Executes data cleaning and missing value processing algorithms. By establishing a data rule library (such as normal range rules for equipment parameters and personnel attendance time rules), it screens and repairs structured data to ensure data accuracy and completeness.

[0056] Semi-structured data processing submodule: It has built-in parsers for various formats (such as XML parser and JSON parser), which converts semi-structured data into structured data, extracts key information, and stores it in the corresponding database table.

[0057] Unstructured Data Processing Submodule: This module uses image processing, natural language processing, and other technologies to perform feature extraction and content analysis on unstructured data such as videos, images, and documents. For example, for video data, object detection algorithms (such as the YOLO algorithm) are used to identify people, equipment, and scenes in the video. For construction drawings, OCR (Optical Character Recognition) technology is used to extract text information, combined with drawing vectorization technology to extract graphic information.

[0058] In one embodiment, 1. The data transmission module enables data transmission from the acquisition terminal to the data center server, including: a short-range transmission unit: This unit uses wireless communication technologies such as Bluetooth and Zigbee to build a wireless sensor network at the construction site, transmitting sensor data distributed at different locations to the aggregation node. This unit has the characteristics of low power consumption and self-organizing network, can adapt to the complex environment of the construction site, and ensure stable data transmission;

[0059] Long-distance transmission unit: Utilizes network communication technologies such as 4G / 5G and Wi-Fi to transmit data from the aggregation node to the data center server. This unit supports real-time data transmission and resumable transmission. In the event of unstable network signals, it can cache data and automatically transmit it after the network is restored to ensure data loss. At the same time, it uses data compression technology (such as GZIP compression) to reduce data transmission volume and improve transmission efficiency.

[0060] In one embodiment, the data storage submodule adopts a distributed storage architecture, combining relational databases (such as MySQL) and non-relational databases (such as MongoDB and HBase) to categorize and store different types of data. Structured data is stored in relational databases, which facilitates complex queries and analysis; semi-structured and unstructured data is stored in non-relational databases, which can efficiently handle large-scale, highly concurrent data read and write operations. At the same time, a data backup and recovery mechanism is established to regularly back up data to prevent data loss.

[0061] Data verification submodule: Based on the preset data verification rule base, it performs real-time verification on the data stored in the database. By setting triggers and stored procedures, it automatically detects the legitimacy and integrity of the data, and promptly marks abnormal data and notifies relevant personnel to handle it.

[0062] Data quality assessment submodule: Use the data quality assessment model to regularly assess the quality of data in the database, generate data quality reports by calculating indicators such as data integrity, accuracy, and consistency, and make data optimization suggestions based on the report results, such as re-collecting low-quality data and improving the data collection process.

[0063] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various modifications and substitutions within the technical scope disclosed in the present invention, and such modifications and substitutions are intended to be within the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be subject to the scope of protection of the claims.

Claims

1. A method for collecting multi-source heterogeneous data at a construction site, characterized by: The following steps are involved: S1. Data type identification and classification; S2, data preprocessing; S3, data transmission and aggregation; S4. Data verification and quality assessment.

2. The method for collecting multi-source heterogeneous data at a construction site according to claim 1, characterized in that: The S1 deploys a variety of data acquisition terminals at the construction site, including but not limited to sensors (such as temperature and humidity sensors, pressure sensors, displacement sensors), cameras, RFID readers, smart meters, etc. After each collection terminal collects data, it first identifies the data type. Based on the data's structural characteristics, it is divided into structured data (such as personnel attendance records and equipment operating parameter tables), semi-structured data (such as construction logs in XML format), and unstructured data (such as video surveillance footage and PDF files of construction drawings). Furthermore, based on the type of construction site information reflected in the data, it is further subdivided into personnel data, equipment data, environmental data, progress data, and so on.

3. The method for collecting multi-source heterogeneous data at a construction site according to claim 1, wherein: The S2 performs data cleaning on structured data to remove duplicate data and erroneous data (such as equipment operating parameters that exceed a reasonable range); it uses interpolation methods (such as linear interpolation and Lagrange interpolation) or statistical model-based methods to fill in missing data; for semi-structured data, it converts it into a unified format through a parser, for example, parsing XML-formatted construction logs into structured tabular data to extract key information (such as construction time, construction content, and person in charge); for unstructured data, it extracts key frames from video data and uses image processing algorithms to identify key scenes in the video (such as illegal operations by personnel and abnormal equipment operation); it extracts features from files such as construction drawings to extract key information such as dimensions and materials in the drawings.

4. The method for collecting multi-source heterogeneous data at a construction site according to claim 1, wherein: The S3 uses a variety of communication technologies to achieve data transmission. For short-distance data transmission, low-power wireless communication technologies such as Bluetooth and Zigbee are used to transmit data collected by sensors to the aggregation node. For longer-distance data transmission, network communication technologies such as 4G / 5G and Wi-Fi are used to transmit data from the aggregation node to the data center server. During the data transmission process, the data is encrypted using encryption algorithms such as AES (Advanced Encryption Standard) to ensure data security and privacy. The data center server serves as a data aggregation point, receiving data from various collection terminals and aggregation nodes, and categorizing and storing them according to data type and category.

5. The method for collecting multi-source heterogeneous data at a construction site according to claim 1, wherein: S4 establishes a data verification rule base to verify the collected data. For example, for equipment operation data, the normal range of parameters is set, and if the data exceeds the range, it is marked as abnormal data. For personnel attendance data, the rationality of punch-in time is checked to prevent abnormal punch-in records. At the same time, a data quality assessment model is used to evaluate data quality from the dimensions of completeness (whether the data is missing key information), accuracy (whether the data is authentic and reliable), and consistency (the consistency of the same data in different data sources). A data quality report is generated, and feedback is provided and re-collection is carried out for data that does not meet the quality standards.

6. A multi-source heterogeneous data acquisition system for a construction site, equipped with a multi-source heterogeneous data acquisition method for a construction site according to any one of claims 1 to 5, characterized in that: It includes the following modules: data acquisition module, data processing module, data transmission module and data management module.

7. The multi-source heterogeneous data acquisition system for a construction site according to claim 6, characterized in that:

1. Data acquisition module: It is composed of multiple data acquisition terminals and is responsible for collecting various data on the construction site. Specifically, it includes: Personnel data collection unit: This unit uses an RFID reader and a facial recognition camera. The RFID reader is used to read the identification card information worn by personnel to obtain personnel attendance, identity and other data. The facial recognition camera is used to identify personnel's facial features and, in combination with the access control system, implements personnel entry and exit management and identity authentication. At the same time, it can monitor personnel's working status (such as whether they are working fatigued) through video analysis technology. Equipment data acquisition unit: Various sensors (such as vibration sensors, current sensors, and GPS positioning sensors) are installed on construction equipment to collect real-time data on the equipment's operating parameters (such as speed, temperature, and position) and working status (such as startup, shutdown, and faults). Furthermore, data such as the equipment's operation logs and maintenance records are obtained through the equipment's built-in communication interfaces (such as RS-485 and Ethernet interfaces). Environmental data collection unit: Deploy temperature and humidity sensors, air quality sensors (to detect PM2.5, PM10, and harmful gas concentrations), noise sensors, rainfall sensors, etc. to collect environmental parameters at the construction site in real time, providing data support for construction safety and personnel health; Progress data collection unit: Utilizes laser rangefinders, drone aerial photography, and other equipment to collect data on topographic changes in the construction area, the construction progress of buildings, and other data. At the same time, combined with construction drawings and plans, image recognition and data analysis technology are used to compare the actual construction progress with the planned progress and generate a progress report.

8. The multi-source heterogeneous data acquisition system for a construction site according to claim 6, characterized in that: The data processing module pre-processes the collected data, specifically including: Structured data processing submodule: Executes data cleaning and missing value processing algorithms. By establishing a data rule library (such as normal range rules for equipment parameters and personnel attendance time rules), it screens and repairs structured data to ensure data accuracy and completeness. Semi-structured data processing submodule: It has built-in parsers for various formats (such as XML parser and JSON parser), which converts semi-structured data into structured data, extracts key information, and stores it in the corresponding database table. Unstructured Data Processing Submodule: This module uses image processing, natural language processing, and other technologies to perform feature extraction and content analysis on unstructured data such as videos, images, and documents. For example, for video data, object detection algorithms (such as the YOLO algorithm) are used to identify people, equipment, and scenes in the video. For construction drawings, OCR (Optical Character Recognition) technology is used to extract text information, combined with drawing vectorization technology to extract graphic information.

9. The multi-source heterogeneous data acquisition system for a construction site according to claim 6, characterized in that: The 1. Data transmission module: This module transmits data from the acquisition terminal to the data center server and includes: a short-range transmission unit: This unit uses wireless communication technologies such as Bluetooth and Zigbee to build a wireless sensor network at the construction site, transmitting data from distributed sensors to the aggregation node. This unit features low power consumption and self-organizing networking, adapting to the complex environment of the construction site and ensuring stable data transmission; Long-distance transmission unit: Utilizes network communication technologies such as 4G / 5G and Wi-Fi to transmit data from the aggregation node to the data center server. This unit supports real-time data transmission and resumable transmission. In the event of unstable network signals, it can cache data and automatically transmit it after the network is restored to ensure data loss. At the same time, it uses data compression technology (such as GZIP compression) to reduce data transmission volume and improve transmission efficiency.

10. The multi-source heterogeneous data acquisition system for a construction site according to claim 6, characterized in that: The data storage submodule adopts a distributed storage architecture, combining relational databases (such as MySQL) and non-relational databases (such as MongoDB and HBase) to categorize and store different types of data. Structured data is stored in relational databases, facilitating complex queries and analysis; semi-structured and unstructured data is stored in non-relational databases, which can efficiently handle large-scale, highly concurrent data read and write operations. At the same time, a data backup and recovery mechanism is established to regularly back up data to prevent data loss. Data verification submodule: Based on the preset data verification rule base, it performs real-time verification on the data stored in the database. By setting triggers and stored procedures, it automatically detects the legitimacy and integrity of the data, and promptly marks abnormal data and notifies relevant personnel to handle it. Data quality assessment submodule: Use the data quality assessment model to regularly assess the quality of data in the database, generate data quality reports by calculating indicators such as data integrity, accuracy, and consistency, and make data optimization suggestions based on the report results, such as re-collecting low-quality data and improving the data collection process.

Images