A data cleaning and processing system and method for the whole process of building construction

Abstract

The present application provides a data cleaning and processing system and method for the whole process of construction, which relates to the field of data processing technology. The acquisition array acquisition module acquires the data acquisition unit array of the target building area; the monitoring data extraction module traverses the data acquisition unit to extract the monitoring data set; the asynchronous fluctuation identification module traverses the monitoring data set to determine the data asynchronous factor; the clustering division module clusters the data acquisition unit; the data processing branch construction module constructs the data processing branch based on the data acquisition asynchronous factor; the branch resource configuration module generates the branch resource configuration information according to the data information entropy of the data acquisition cluster; the data cleaning processing module constructs the data cleaning processing network layer based on the branch resource configuration information and the data processing branch. The present application solves the technical problem that the existing technology lacks pertinence in the processing of construction data and cannot perform effective abnormality identification and processing, and improves the accuracy and reliability of the data.

Description

Technical Field

[0001] The present application relates to the technical field of data processing, and in particular to a system and method for cleaning and processing data throughout the entire construction process. Background Art

[0002] In the process of modern construction, due to the complex environment of the construction site, the data collection process is easily interfered by various factors, resulting in asynchronous fluctuations and noise in the collected data. Data cleaning processing is a key link to ensure data quality and improve management efficiency.

[0003] Existing data cleaning and processing technologies mainly include rule-based methods, statistical methods, machine learning methods, etc. Rule-based methods use preset rules to identify and correct errors in data. This method is simple to operate, but it is difficult to deal with complex data problems. Statistical methods use data distribution characteristics to identify and eliminate abnormal data, which improves the accuracy of data cleaning to a certain extent, but has limited effect on nonlinear or non-uniformly distributed data processing. Machine learning methods, such as clustering and neural networks, can process more complex data patterns, but require a large amount of labeled data and computing resources, and the model selection and training process is relatively complicated. These technologies lack effective identification and processing of asynchronous data fluctuations, which easily leads to data error accumulation. Secondly, most of the existing data processing methods are general-purpose, and it is difficult to customize the data characteristics unique to construction. In addition, the problem of unreasonable resource allocation also causes some processing units to be overloaded, while other unit resources are idle, and system resources are not fully utilized. Summary of the invention

[0004] The present application provides a system and method for cleaning and processing data throughout the construction process, which solves the technical problem that existing data cleaning and processing technologies cannot effectively identify and process anomalies according to the data characteristics of construction, resulting in insufficient accuracy and efficiency of data processing, and achieves the technical effect of improving the accuracy and reliability of data.

[0005] In view of the above problems, on the one hand, the present application provides a data cleaning and processing system for the whole process of construction, the system comprising: an acquisition array acquisition module, the acquisition array acquisition module is used to acquire a data acquisition unit array of a target building area, wherein the data acquisition unit array comprises K data acquisition units and K unit layout positions; a monitoring data extraction module, the monitoring data extraction module is used to traverse the K data acquisition units within a preset monitoring window to extract monitoring data and obtain K monitoring data sets; an asynchronous fluctuation identification module, the asynchronous fluctuation identification module is used to traverse the K monitoring data sets to identify data asynchronous fluctuations and determine K data asynchronous factors; a clustering division module, the clustering division module is used to cluster the K data acquisition units based on the K data asynchronous factors and the K unit layout positions Cluster division, generating M data acquisition clusters, wherein the M data acquisition clusters include M data acquisition asynchronous factors; a data processing branch construction module, wherein the data processing branch construction module is used to construct M data processing branches for cleaning the collected data of the M data acquisition clusters based on the M data acquisition asynchronous factors; a branch resource configuration module, wherein the branch resource configuration module is used to perform resource configuration on the M data processing branches according to the data information entropy of the M data acquisition clusters, and generate M branch resource configuration information; a data cleaning processing module, wherein the data cleaning processing module is used to construct a data cleaning processing network layer based on the M branch resource configuration information and the M data processing branches, wherein the data cleaning processing network layer is used to perform data cleaning processing on the collected data of the data acquisition unit array of the target building area.

[0006] On the other hand, the present application also provides a method for cleaning and processing data of the entire construction process, the method comprising: obtaining a data acquisition unit array of a target building area, wherein the data acquisition unit array comprises K data acquisition units and K unit layout positions; traversing the K data acquisition units within a preset monitoring window to extract monitoring data and obtain K monitoring data sets; traversing the K monitoring data sets to identify data asynchronous fluctuations and determine K data asynchronous factors; clustering the K data acquisition units based on the K data asynchronous factors and the K unit layout positions to generate M data sets. The invention relates to a method for preparing a data collection cluster, wherein the M data collection clusters include M data collection asynchronous factors; based on the M data collection asynchronous factors, M data processing branches are constructed for cleaning the collected data of the M data collection clusters; according to the data information entropy of the M data collection clusters, resources are configured for the M data processing branches to generate M branch resource configuration information; based on the M branch resource configuration information and the M data processing branches, a data cleaning processing network layer is constructed, wherein the data cleaning processing network layer is used to perform data cleaning processing on the collected data of the data collection unit array of the target building area.

[0007] One or more technical solutions provided in this application have at least the following technical effects or advantages:

[0008] The acquisition array acquisition module acquires the data acquisition unit array of the target building area, wherein the data acquisition unit array includes K data acquisition units and K unit layout positions. This module determines the coverage and layout position of data acquisition, and provides a basis for subsequent data monitoring and processing. The monitoring data extraction module traverses the K data acquisition units within the preset monitoring window to extract monitoring data and obtain K monitoring data sets. This module can obtain real-time data in the construction process by regularly extracting monitoring data, and provide the original data basis for subsequent data processing. The asynchronous fluctuation identification module traverses the K monitoring data sets to identify data asynchronous fluctuations and determine K data asynchronous factors. This module can detect abnormal changes in the data by identifying asynchronous fluctuations, provide a basis for subsequent data clustering and cleaning, and improve the accuracy and reliability of the data. The clustering division module clusters the K data acquisition units based on the K data asynchronous factors and the K unit layout positions to generate M data acquisition clusters, wherein the M data acquisition clusters include M data acquisition asynchronous factors. This module can classify similar acquisition units through clustering, thereby improving the efficiency and accuracy of data processing. The data processing branch construction module constructs M data processing branches for cleaning the collected data of the M data acquisition clusters based on the M data acquisition asynchronous factors. This module constructs data processing branches, which can process the data of each data cluster in parallel, thereby improving the efficiency and processing capacity of data cleaning. The branch resource configuration module performs resource configuration on the M data processing branches according to the data information entropy of the M data acquisition clusters, and generates M branch resource configuration information. Through resource configuration, this module can reasonably allocate computing resources according to data complexity and importance, thereby improving processing efficiency. The data cleaning processing module constructs a data cleaning processing network layer based on the M branch resource configuration information and the M data processing branches, wherein the data cleaning processing network layer is used to perform data cleaning processing on the collected data of the data acquisition unit array of the target building area. This module achieves comprehensive and efficient data cleaning and improves data quality through the collaborative work of multiple branches.

[0009] To summarize, this application integrates the above modules, analyzes the distribution and characteristics of data in the entire construction process, effectively identifies and processes asynchronous fluctuations in the data, realizes efficient collection, accurate processing and comprehensive cleaning of data in the entire construction process, and improves the efficiency and accuracy of data cleaning and processing in the entire construction process, thereby significantly improving the reliability and quality of the data, and providing strong data support for the monitoring and management of construction.

[0010] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 A structural diagram of a data cleaning and processing system for the entire construction process provided by an embodiment of the present application;

[0012] Figure 2 A schematic diagram of a process of traversing K monitoring data sets to identify data asynchronous fluctuations in a data cleaning and processing system for the entire construction process provided in an embodiment of the present application;

[0013] Figure 3 A schematic diagram of a process for generating K data asynchronous factors in a data cleaning and processing system for the entire construction process provided by an embodiment of the present application;

[0014] Figure 4 A flowchart of a construction whole-process data cleaning and processing system provided in an embodiment of the present application.

[0015] Explanation of the reference numerals: acquisition array acquisition module 10 , monitoring data extraction module 20 , asynchronous fluctuation identification module 30 , cluster division module 40 , data processing branch construction module 50 , branch resource configuration module 60 , data cleaning processing module 70 . DETAILED DESCRIPTION

[0016] The embodiment of the present application provides a system and method for cleaning and processing data throughout the construction process, thereby solving the technical problem that the existing data cleaning and processing technology cannot effectively identify and process anomalies according to the data characteristics of construction, resulting in insufficient accuracy and efficiency of data processing, and achieves the technical effect of improving the accuracy and reliability of data.

[0017] Embodiment 1, as Figure 1 As shown, the embodiment of the present application provides a data cleaning and processing system for the entire construction process, the system comprising:

[0018] The acquisition array acquisition module 10 is used to acquire a data acquisition unit array of a target building area, wherein the data acquisition unit array includes K data acquisition units and K unit layout positions.

[0019] Specifically, the target building area refers to a specific construction area that needs to be monitored and data collected. The data acquisition unit refers to various sensors and monitoring devices installed in the target building area, which are responsible for collecting environmental parameters such as temperature, humidity, pressure, vibration, etc. in real time to monitor the status of the construction. The data acquisition unit array refers to the combination of these sensors and monitoring devices, which are distributed in key locations of the construction site according to a certain layout to ensure comprehensive coverage and collect necessary data. The unit layout location refers to the actual physical location of each data acquisition unit on the construction site. These locations are selected to maximize the coverage and relevance of the data. K represents the specific number of devices in the data acquisition unit array, and K is a positive integer.

[0020] The acquisition array acquisition module 10 identifies and locates the positions of all data acquisition units in the target building area. This is usually achieved through GPS positioning, Internet of Things technology or building information modeling (BIM). For example, each data acquisition unit may be equipped with a GPS sensor for transmitting its position information in real time. The module integrates these data acquisition units and their corresponding position information into a data acquisition unit array, which not only includes the K data acquisition units themselves, but also includes the specific locations of the K data acquisition units in the building space. The database system is used to record the layout location and equipment information of each data acquisition unit to ensure that the location and data of each unit can be accurately associated during subsequent data processing.

[0021] The monitoring data extraction module 20 is used to traverse the K data collection units within a preset monitoring window to extract monitoring data and obtain K monitoring data sets.

[0022] Specifically, a preset monitoring window is defined according to the monitoring requirements. The preset monitoring window refers to a set data extraction time period, which can be hourly, daily or weekly. For example, for real-time monitoring of a construction site, the preset monitoring window may be set to once an hour. Data collected within the preset monitoring window is extracted from each data acquisition unit using data communication protocols such as MQTT, HTTP, etc., and data extraction tools such as API, sensor data reading library, etc. The data extracted from each data acquisition unit is stored in a set to form K monitoring data sets. This process ensures the timeliness and relevance of data extraction and provides basic data for subsequent data analysis and processing. Among them, the monitoring data set is a data set extracted from each data acquisition unit, and each set contains the data of a data acquisition unit within the monitoring window. For example, if the monitoring window is set to 8 to 10 a.m. every day, then during this time period, the module will automatically access each data acquisition unit and extract the data collected by it. These data may include temperature, humidity, vibration intensity, etc., depending on the type of data acquisition unit. The extracted data is then organized into K monitoring data sets, each set corresponding to a data record of a data acquisition unit within the monitoring window.

[0023] The asynchronous fluctuation identification module 30 is used to traverse the K monitoring data sets to identify data asynchronous fluctuations and determine K data asynchronous factors.

[0024] Specifically, the asynchronous fluctuation identification module 30 visits each monitoring data set in turn and analyzes the asynchronous fluctuations in the data. These abnormal fluctuations may be caused by sensor failure, environmental changes or other factors. According to the identified asynchronous fluctuations, a data asynchronous factor is determined for each data acquisition unit. For K monitoring data sets, the module can determine K data asynchronous factors, and each monitoring data set corresponds to a data asynchronous factor. This data asynchronous factor is an indicator or parameter used to quantify and represent data asynchronous fluctuations, reflecting the degree and characteristics of the unit data fluctuations. These asynchronous factors are then used for clustering and data cleaning processing. Through this process, the asynchronous fluctuation identification module 30 can timely identify and understand abnormal fluctuations in the data, provide an important reference basis for data cleaning and processing, and ensure the accuracy and reliability of the data.

[0025] The clustering division module 40 is used to cluster the K data acquisition units based on the K data asynchronous factors and the K unit layout positions to generate M data acquisition clusters, wherein the M data acquisition clusters include the M data acquisition asynchronous factors.

[0026] Specifically, the clustering division module 40 obtains the layout positions of K data asynchronous factors and K data acquisition units. The data asynchronous factors and unit layout positions are standardized to ensure that data of different dimensions are clustered and analyzed on the same scale. A suitable clustering algorithm is selected to divide the K data acquisition units into M data acquisition clusters. Common clustering algorithms include K-means clustering, hierarchical clustering, and DBSCAN. The module analyzes the standardized data through a clustering algorithm and divides the K data acquisition units into several groups, each of which is called a data acquisition cluster. Each cluster contains data acquisition units with similar data fluctuation characteristics. In addition, the unit layout position is also used as a factor of clustering to ensure that spatially close units are classified into the same cluster. Each data acquisition cluster contains a group of data acquisition units, corresponding to a data acquisition asynchronous factor, wherein the data acquisition asynchronous factor refers to the fluctuation index of each data acquisition cluster after clustering, which comprehensively reflects the data fluctuation of all units in the cluster. Finally, M data acquisition clusters and M data acquisition asynchronous factors are output, wherein M is a positive integer, and M<K, indicating the number of generated data acquisition clusters. Through this process, the clustering module 40 effectively organizes and classifies the data collection units, providing a structured and targeted data set for subsequent data cleaning and processing.

[0027] The data processing branch construction module 50 is used to construct M data processing branches for cleaning the collected data of the M data collection clusters based on the M data collection asynchronous factors.

[0028] Specifically, the data processing branch construction module 50 obtains M data acquisition clusters and their corresponding asynchronous factors. The data characteristics of each data acquisition cluster are analyzed according to its asynchronous factor. For example, a high asynchronous factor may indicate that the data fluctuates greatly, requiring more cleaning steps and stricter anomaly detection. A dedicated data processing branch is constructed for each data acquisition cluster. The data processing branch refers to a unique data cleaning and processing path designed for each data acquisition cluster, which is intended to solve the data quality problems unique to the cluster. The cleaning method and process of each processing branch are customized according to the data characteristics and asynchronous factors. For example, if the data asynchronous factor of a data acquisition cluster indicates that its data has large fluctuations, the module may construct a processing branch containing a filtering algorithm for this cluster to remove noise from the data. On the other hand, if the data asynchronous factor of another cluster indicates that its data is relatively stable, the module may construct a simple data calibration branch for this cluster. Through this process, the data processing branch construction module 50 ensures that each data cluster can receive the most suitable processing, improving the efficiency and accuracy of data cleaning and processing.

[0029] The branch resource configuration module 60 is used to configure resources for the M data processing branches according to the data information entropy of the M data acquisition clusters, and generate M branch resource configuration information.

[0030] Specifically, branch resource configuration information refers to the quantity and type of computing resources, storage resources or other necessary resources allocated to each data processing branch, including CPU, memory, storage space, etc.

[0031] The branch resource configuration module 60 first obtains M data acquisition clusters and evaluates the data information entropy of each data acquisition cluster. This can be achieved through complexity analysis, such as entropy calculation in information theory, to determine the quality and processing difficulty of the data. Data information entropy is an indicator that reflects the complexity and uncertainty of the data within the data acquisition cluster. High information entropy means that the data is complex and changeable, and low information entropy means that the data is relatively simple and stable. Then, according to the data information entropy of each data acquisition cluster, a resource requirement calculation tool or a custom algorithm, such as a linear model or a machine learning model, is used to estimate the resource requirements and calculate the computing resources required for each data processing branch. Data acquisition clusters with high information entropy usually require more computing resources for processing. Then, a resource allocation strategy is formulated to reasonably allocate available computing resources to each data processing branch based on the calculated resource requirements. The resource allocation strategy may include priority allocation, balanced allocation and other methods to ensure that each branch can obtain sufficient resources for data processing. Finally, according to the resource allocation strategy, specific resource configuration information is generated for each data processing branch. Including the number of CPU cores, memory size, storage space, etc. allocated to each branch. The purpose of this resource allocation is to optimize the data processing process, ensure that resources are used effectively, and at the same time improve the speed and efficiency of data processing.

[0032] For example, there are two data processing branches, branch A processes relatively pure data, while branch B processes noisy data. The module uses a decision tree algorithm based on information entropy to analyze the information entropy of each data cluster, and evaluates that branch B has higher information entropy, so it allocates more resources to it, such as a more powerful processor and larger storage space, to ensure that branch B can effectively process and clean data.

[0033] Through the above process, the branch resource configuration module 60 can reasonably allocate computing resources to each data processing branch according to the data information entropy of the data acquisition cluster, more effectively utilize system resources, ensure efficient data cleaning and processing, and thus improve the overall processing capacity and efficiency of the system.

[0034] A data cleaning processing module 70, wherein the data cleaning processing module 70 is used to construct a data cleaning processing network layer based on the M branch resource configuration information and the M data processing branches, wherein the data cleaning processing network layer is used to perform data cleaning processing on the collected data of the data collection unit array of the target building area.

[0035] Specifically, the data cleaning processing network layer refers to an integrated data cleaning framework that includes multiple data processing branches and resource configurations for processing data from different data acquisition clusters. The design purpose of this network layer is to ensure that the collected data of all data acquisition unit arrays can be effectively cleaned.

[0036] The data cleaning processing module 70 obtains M branch resource configuration information and M data processing branches. According to the branch resource configuration information and the data processing branches, a multi-level network layer is constructed. This network will take into account the needs of each data processing branch to ensure the effectiveness and efficiency of data cleaning. The specific construction process includes: designing a network structure, including a data input layer, multiple processing layers and an output layer, and each processing layer corresponds to a data processing branch. Each processing layer must be able to handle specific data problems and can also work with other layers. According to the branch resource configuration information, necessary resources are allocated to each processing layer, including computing resources, storage resources and network bandwidth. In each processing layer, specific data cleaning steps and data cleaning algorithms are configured, such as missing value processing, outlier detection and processing, data standardization and data smoothing. Use data processing tools and machine learning tools to achieve data cleaning. After the construction is completed, an integrated test is performed to ensure that the various processing layers can work seamlessly and can process data correctly.

[0037] In the data cleaning processing network layer, tasks are assigned to perform data cleaning tasks in parallel. Each data processing branch processes a data collection cluster and cleans the data according to the configured cleaning steps. Parallel processing frameworks such as Python's multiprocessing module are used to implement parallel processing and improve cleaning efficiency. Then, the operating status and resource usage of each data processing branch are monitored in real time to ensure the smooth progress of the data cleaning process. The cleaned data is stored in a unified data storage system for subsequent data analysis and processing.

[0038] Through this process, the data cleaning processing module 70 processes the collected data efficiently and accurately, improves the quality and availability of the data, and thus provides more reliable data support for construction management.

[0039] Further, such as Figure 2 As shown, the asynchronous fluctuation identification module 30 in the embodiment of the present application is also used to perform the following steps:

[0040] K monitoring data boxes are constructed based on the K monitoring data sets; K fluctuation identification scales and K fluctuation identification directions are determined according to the distances of the K medians of the K monitoring data boxes relative to the centers of the K monitoring data boxes; edge values ​​of the data exceeding the preset edges in the K monitoring data boxes are removed to generate K cleaning monitoring data sets; data asynchronous fluctuations are identified for the K cleaning monitoring data sets according to the K fluctuation identification scales and the K fluctuation identification directions to determine K data asynchronous factors.

[0041] Specifically, the monitoring data box is a statistical representation of the data set, which is used to describe the data distribution. The median is the middle value in the data set, that is, the value in the middle after the data is sorted. The fluctuation identification scale is a quantitative indicator to measure the degree of data fluctuation, reflecting the amplitude of data fluctuation. The fluctuation identification direction is the direction of data fluctuation, which can be rising or falling. The cleaning monitoring data set is the data set after edge value removal and asynchronous fluctuation identification processing.

[0042] The asynchronous fluctuation identification module 30 uses a box plot and a statistical analysis method to identify asynchronous fluctuations in the data. First, K monitoring data sets are obtained, and a box plot is constructed for each data set. The box plot is a statistical chart that describes the data distribution, which shows the median, quartiles, minimum and maximum values ​​of the data. Next, the distance between the median and the center of each monitoring data box is calculated. The box center is usually the middle position of the box, that is, the midpoint of the upper and lower quartiles. The fluctuation identification scale is determined by the distance between the median and the center of the box. The larger the distance, the more obvious the fluctuation, the more asymmetric the data distribution, and the larger the amplitude of a single movement during identification. At the same time, according to the position of the median relative to the center of the box, the fluctuation identification direction is determined, that is, whether the data fluctuation is upward or downward, and which side the median is close to, which side is the fluctuation identification direction. Then, the edge value of the data exceeding the preset edge in the K monitoring data boxes is eliminated to generate K cleaning monitoring data sets. For example, if the edge value is set to a value outside the first quartile and the third quartile of the box, then these values ​​will be eliminated. Finally, data asynchronous fluctuations of K cleaning monitoring data sets are identified according to K fluctuation identification scales and fluctuation identification directions, and K data asynchronous factors are determined.

[0043] Preferably, Figure 3 As shown, the asynchronous fluctuation identification module 30 in the embodiment of the present application is also used to perform the following steps:

[0044] K identification spaces are constructed based on the K cleaning monitoring data sets, wherein the K identification spaces include K identification particle sets, and each identification particle corresponds to one cleaning monitoring data; K central particles of the K identification spaces are extracted, and the K central particles are used as starting points to move in the K identification spaces according to the K fluctuation identification scales and the K fluctuation identification directions to obtain K iterative identification particles; particle iterative update is performed according to the K iterative identification particles and the K central particles to determine K stage identification particles; iterates in the K identification space starting from the K stage identification particles until a preset number of iterations is met or the density difference of the stage identification particles obtained from two adjacent iterations is less than a preset density difference, and the K stage identification particles obtained from the last iteration are used as K target identification particles; the difference between the K cleaning monitoring data corresponding to the K target identification particles and the K preset standard monitoring data is calculated, and the calculation results are respectively compared with the K preset standard monitoring data to generate K data asynchronous factors.

[0045] Specifically, the recognition space is a multidimensional space for data analysis and recognition, which contains a set of cleaning monitoring data and a set of recognition particles. In the recognition space, each particle corresponds to a cleaning monitoring data, indicating the position of the data point in the recognition space. Among them, the central particle is the most representative particle in the recognition space, usually the center or average value of the data. Iterative recognition particles are particles that move in the recognition space according to the scale and direction of fluctuation recognition, and are used to track data changes. Stage recognition particles are particles that have been iteratively updated and are used to represent the current recognition state. Target recognition particles refer to particles obtained after the last iteration, which are used to calculate the data asynchronous factor. Preset standard monitoring data are predefined standard data for comparison with cleaning monitoring data.

[0046] First, obtain K cleaning monitoring data sets, and construct a recognition space for each data set. Each space contains a set of recognition particles, and each particle in the set represents a cleaning monitoring data. Extract the central particles in the K recognition spaces. These central particles are usually the median or mean of the data, representing the average or representative state of each cleaning monitoring data set. Then, take these central particles as the starting point, move in the recognition space according to the determined fluctuation recognition scale and fluctuation recognition direction, and obtain K iterative recognition particles. This movement simulates the change of data in time or space, and vector operation methods can be used to move particles.

[0047] Afterwards, the particles are iteratively updated according to the iterative identification particles and the central particles to determine K stage identification particles. These particles reflect the state of the data after a certain number of iterations. A threshold of the number of iterations or a particle density difference is preset, that is, a preset number of iterations or a preset density difference, which is used as a criterion for determining the termination of the iteration. Starting from the stage identification particle, the iteration continues in the identification space until the preset number of iterations is met or the density difference between the stage identification particles obtained in two adjacent iterations is less than the preset density difference. At this point, the K stage identification particles obtained in the last iteration are used as target identification particles. Finally, the difference between the cleaning monitoring data corresponding to the target identification particle and the preset standard monitoring data is calculated, and the result is compared with the preset standard monitoring data to generate K data asynchronous factors. Through this process, the asynchronous fluctuation identification module 30 can perform complex iterative analysis and identification on the data, thereby more accurately determining the degree of asynchronous fluctuation of the data.

[0048] Optionally, the asynchronous fluctuation identification module 30 in the embodiment of the present application is further configured to perform the following steps:

[0049] Obtain K iteration densities of the K iteration identification particles and K center densities of the K center particles; determine whether the K iteration densities are greater than or equal to the K center densities, and if so, use the K iteration identification particles as K stage identification particles; if not, call a random number generator to generate a first random number and a second random number, and when the first random number is greater than or equal to the second random number, use the K center particles as K stage identification particles.

[0050] Optionally, the asynchronous fluctuation identification module 30 in the embodiment of the present application is further configured to perform the following steps:

[0051] When the first random number is less than or equal to the second random number, the K iterative identification particles are used as the K stage identification particles, and the moving direction at least in the next movement is the opposite direction of the K wave identification directions.

[0052] Specifically, the iteration density is the ratio of the number of particles in the region constructed with the iteration identification particle as the center and the corresponding fluctuation identification scale as the radius to the area of ​​the region, which indicates the concentration of data points around the particle. The center density is the density of particles in the region around the center particle, which indicates the concentration of data points around the center particle. A random number generator is a tool or algorithm for generating random numbers, such as the Rand function. The first random number and the second random number are two random numbers generated by a random number generator, which are used to determine which particle to select as the stage identification particle.

[0053] First, calculate the K iteration densities corresponding to the K iteration identification particles and the K center densities corresponding to the K center particles. The calculation method is to construct an area with the particle as the center and the fluctuation identification scale of the identification space where the particle is located as the radius, and count the number of particles in the area. The density corresponding to the particle is obtained by dividing the area of ​​the area by the iteration density of each iteration identification particle and the center density of the corresponding center particle. If the iteration density is greater than or equal to the center density, the iteration identification particle is used as the stage identification particle. If the iteration density is less than the center density, call the random number generator to generate two random numbers, namely the first random number and the second random number. Compare the first random number and the second random number. If the first random number is greater than or equal to the second random number, the center particle is used as the stage identification particle. If the first random number is less than or equal to the second random number, the iteration identification particle is used as the stage identification particle, and the direction of the next movement is the opposite direction of the fluctuation identification direction. This process improves the flexibility and accuracy of the identification process by combining iteration and random selection, so that the asynchronous fluctuation identification module 30 can more accurately determine the asynchronous fluctuation state of the data.

[0054] Furthermore, the cluster division module 40 in the embodiment of the present application is also used to perform the following steps:

[0055] The central layout position of the data processing center of the target building area is obtained; a distance analysis is performed according to the central layout position and the K unit layout positions to generate K position asynchronous factors; the K data acquisition units are clustered into the same type according to the K data asynchronous factors and the K position asynchronous factors to generate the M data acquisition clusters.

[0056] Specifically, use GIS tools or building plans to determine the central layout position of the data processing center in the target building area. This position is a fixed coordinate point used to receive and process data from all data acquisition units. According to the central layout position and the K unit layout positions, calculate the distance from each data acquisition unit to the data processing center. Divide the K distances by the sum of the K distances, and use the calculated ratios as K position asynchronous factors. The position asynchronous factor reflects the distance difference between each data acquisition unit and the data processing center. Based on the K data asynchronous factors and the K position asynchronous factors, perform cluster analysis on the K data acquisition units, select a suitable clustering algorithm, such as K-means clustering, and divide the units with similar data asynchronous factors and position asynchronous factors into the same cluster to generate M data acquisition clusters.

[0057] Furthermore, the data processing branch construction module 50 of the embodiment of the present application is also used to perform the following steps:

[0058] Constructing an asynchronous factor-extraction frequency mapping relationship; using the M data acquisition asynchronous factors as indexes, searching the asynchronous factor-extraction frequency mapping relationship to generate M extraction frequencies; and constructing the M data processing branches based on the M extraction frequencies.

[0059] Specifically, the asynchronous factor-extraction frequency mapping relationship is a data structure that represents the data extraction frequency corresponding to each data asynchronous factor. The extraction frequency refers to the frequency at which the system extracts data from the data acquisition unit. The higher the frequency, the greater the data fluctuation, and the more frequent the data extraction.

[0060] Establish a mapping relationship table, i.e., asynchronous factor-extraction frequency mapping relationship, to indicate the extraction frequency corresponding to each data asynchronous factor. This table can be established through historical data analysis or empirical data, which records the extraction frequency corresponding to each data asynchronous factor. According to the asynchronous factor of each data acquisition cluster, the corresponding extraction frequency is retrieved from the asynchronous factor-extraction frequency mapping relationship table. The retrieved extraction frequency indicates the frequency at which the system needs to extract data from the data acquisition unit. M extraction frequencies are generated according to M data acquisition asynchronous factors. According to the extraction frequency of each data acquisition cluster, the corresponding data processing branch is constructed. The processing branch includes data extraction, cleaning and processing steps, and the specific cleaning and processing methods are customized according to the data characteristics and extraction frequency.

[0061] Exemplarily, the asynchronous factor-extraction frequency mapping relationship is constructed as follows: {1: low; 2: medium; 3: high}. There are 5 data collection clusters, and the asynchronous factors are {1, 2, 3, 2, 1}. According to the asynchronous factor retrieval mapping relationship, the extraction frequency is generated as {low, medium, high, medium, low}. According to the extraction frequency, the data processing branch is constructed to handle data cleaning tasks of different frequencies:

[0062] Low-frequency processing branch: data is extracted every hour; the cleaning step only includes data calibration.

[0063] Medium frequency processing branch: extract data every 30 minutes; the cleaning steps include missing value processing, detailed outlier detection, and data smoothing.

[0064] High-frequency processing branch: extracts data every 10 minutes; the cleaning steps include missing value processing, multiple data smoothing and advanced filtering algorithms.

[0065] Through the above process, the data processing branch construction module 50 can construct appropriate data processing branches according to the data characteristics of different data acquisition clusters, thereby realizing efficient and customized data cleaning and processing for different data acquisition clusters, thereby improving the efficiency and accuracy of data cleaning.

[0066] In summary, the data cleaning and processing system for the entire construction process provided by the embodiment of the present application has the following technical effects:

[0067] The acquisition array acquisition module 10 acquires the data acquisition unit array of the target building area, wherein the data acquisition unit array includes K data acquisition units and K unit layout positions. This module determines the coverage and layout position of data acquisition, and provides a basis for subsequent data monitoring and processing. The monitoring data extraction module 20 traverses the K data acquisition units within the preset monitoring window to extract monitoring data and obtain K monitoring data sets. This module can obtain real-time data in the construction process by regularly extracting monitoring data, and provide the original data basis for subsequent data processing. The asynchronous fluctuation identification module 30 traverses the K monitoring data sets to identify data asynchronous fluctuations and determine K data asynchronous factors. This module can detect abnormal changes in the data by identifying asynchronous fluctuations, provide a basis for subsequent data clustering and cleaning, and improve the accuracy and reliability of the data. The clustering division module 40 clusters the K data acquisition units based on the K data asynchronous factors and the K unit layout positions to generate M data acquisition clusters, wherein the M data acquisition clusters include M data acquisition asynchronous factors. This module can classify similar acquisition units through clustering, thereby improving the efficiency and accuracy of data processing. The data processing branch construction module 50 constructs M data processing branches for cleaning the collected data of the M data acquisition clusters based on the M data acquisition asynchronous factors. This module constructs data processing branches, which can process the data of each data cluster in parallel, thereby improving the efficiency and processing capacity of data cleaning. The branch resource configuration module 60 performs resource configuration on the M data processing branches according to the data information entropy of the M data acquisition clusters, and generates M branch resource configuration information. Through resource configuration, this module can reasonably allocate computing resources according to data complexity and importance, thereby improving processing efficiency. The data cleaning processing module 70 constructs a data cleaning processing network layer based on the M branch resource configuration information and the M data processing branches, wherein the data cleaning processing network layer is used to perform data cleaning processing on the collected data of the data acquisition unit array of the target building area. This module achieves comprehensive and efficient data cleaning and improves data quality through the collaborative work of multiple branches.

[0068] Overall, the embodiment of the present application integrates the above-mentioned modules, analyzes the distribution and characteristics of the data of the entire construction process, effectively identifies and processes asynchronous fluctuations in the data, realizes the efficient collection, accurate processing and comprehensive cleaning of the data of the entire construction process, and improves the efficiency and accuracy of data cleaning and processing throughout the construction process, thereby significantly improving the reliability and quality of the data, and providing strong data support for the monitoring and management of construction.

[0069] Embodiment 2, as Figure 4 As shown, the embodiment of the present application provides a method for cleaning and processing data in the whole process of building construction, the method comprising:

[0070] Acquire a data acquisition unit array of a target building area, wherein the data acquisition unit array includes K data acquisition units and K unit layout positions; traverse the K data acquisition units within a preset monitoring window to extract monitoring data and obtain K monitoring data sets; traverse the K monitoring data sets to identify data asynchronous fluctuations and determine K data asynchronous factors; cluster the K data acquisition units based on the K data asynchronous factors and the K unit layout positions to generate M data acquisition clusters, wherein the M data acquisition clusters include M data acquisition asynchronous factors; construct M data processing branches for cleaning the collected data of the M data acquisition clusters based on the M data acquisition asynchronous factors; configure resources for the M data processing branches according to the data information entropy of the M data acquisition clusters to generate M branch resource configuration information; construct a data cleaning processing network layer based on the M branch resource configuration information and the M data processing branches, wherein the data cleaning processing network layer is used to perform data cleaning processing on the collected data of the data acquisition unit array of the target building area.

[0071] Furthermore, the embodiment of the present application traverses the K monitoring data sets to identify data asynchronous fluctuations, and further includes:

[0072] K monitoring data boxes are constructed based on the K monitoring data sets; K fluctuation identification scales and K fluctuation identification directions are determined according to the distances of the K medians of the K monitoring data boxes relative to the centers of the K monitoring data boxes; edge values ​​of the data exceeding the preset edges in the K monitoring data boxes are removed to generate K cleaning monitoring data sets; data asynchronous fluctuations are identified for the K cleaning monitoring data sets according to the K fluctuation identification scales and the K fluctuation identification directions to determine K data asynchronous factors.

[0073] Further, the embodiment of the present application performs data asynchronous fluctuation identification on the K cleaning monitoring data sets according to the K fluctuation identification scales and the K fluctuation identification directions, and determines K data asynchronous factors, and further includes:

[0074] K identification spaces are constructed based on the K cleaning monitoring data sets, wherein the K identification spaces include K identification particle sets, and each identification particle corresponds to one cleaning monitoring data; K central particles of the K identification spaces are extracted, and the K central particles are used as starting points to move in the K identification spaces according to the K fluctuation identification scales and the K fluctuation identification directions to obtain K iterative identification particles; particle iterative update is performed according to the K iterative identification particles and the K central particles to determine K stage identification particles; iterates in the K identification space starting from the K stage identification particles until a preset number of iterations is met or the density difference of the stage identification particles obtained from two adjacent iterations is less than a preset density difference, and the K stage identification particles obtained from the last iteration are used as K target identification particles; the difference between the K cleaning monitoring data corresponding to the K target identification particles and the K preset standard monitoring data is calculated, and the calculation results are respectively compared with the K preset standard monitoring data to generate K data asynchronous factors.

[0075] Furthermore, the embodiment of the present application performs particle iterative updating according to the K iteratively identified particles and the K central particles to determine K stage identified particles, and further includes:

[0076] Obtain K iteration densities of the K iteration identification particles and K center densities of the K center particles; determine whether the K iteration densities are greater than or equal to the K center densities, and if so, use the K iteration identification particles as K stage identification particles; if not, call a random number generator to generate a first random number and a second random number, and when the first random number is greater than or equal to the second random number, use the K center particles as K stage identification particles.

[0077] Furthermore, the method described in the embodiment of the present application also includes:

[0078] When the first random number is less than or equal to the second random number, the K iterative identification particles are used as the K stage identification particles, and the moving direction at least in the next movement is the opposite direction of the K wave identification directions.

[0079] Furthermore, the embodiment of the present application clusters the K data acquisition units based on the K data asynchronous factors and the K unit layout positions to generate M data acquisition clusters, and further includes:

[0080] The central layout position of the data processing center of the target building area is obtained; a distance analysis is performed according to the central layout position and the K unit layout positions to generate K position asynchronous factors; the K data acquisition units are clustered into the same type according to the K data asynchronous factors and the K position asynchronous factors to generate the M data acquisition clusters.

[0081] Furthermore, the embodiment of the present application constructs M data processing branches for cleaning the collected data of the M data collection clusters based on the M data collection asynchronous factors, and further includes:

[0082] Constructing an asynchronous factor-extraction frequency mapping relationship; using the M data acquisition asynchronous factors as indexes, searching the asynchronous factor-extraction frequency mapping relationship to generate M extraction frequencies; and constructing the M data processing branches based on the M extraction frequencies.

[0083] Through the above detailed description of a system for cleaning and processing data for the entire construction process, those skilled in the art can clearly understand a method for cleaning and processing data for the entire construction process in this embodiment. As for the method disclosed in the second embodiment, since it corresponds to the system disclosed in the first embodiment and has corresponding execution steps and beneficial effects, please refer to the system description for the relevant details.

[0084] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A data cleaning and processing system for the entire construction process, characterized in that: The system comprises: An acquisition array acquisition module, wherein the acquisition array acquisition module is used to acquire a data acquisition unit array of a target building area, wherein the data acquisition unit array includes K data acquisition units and K unit layout positions; A monitoring data extraction module, the monitoring data extraction module is used to traverse the K data acquisition units within a preset monitoring window to extract monitoring data and obtain K monitoring data sets; An asynchronous fluctuation identification module, the asynchronous fluctuation identification module is used to traverse the K monitoring data sets to identify data asynchronous fluctuations and determine K data asynchronous factors, the data asynchronous factors are indicators or parameters used to quantify and represent data asynchronous fluctuations, and reflect the degree and characteristics of data fluctuations; A clustering division module, the clustering division module is used to cluster the K data acquisition units based on the K data asynchronous factors and the K unit layout positions to generate M data acquisition clusters, wherein the M data acquisition clusters include the M data acquisition asynchronous factors; A data processing branch construction module, wherein the data processing branch construction module is used to construct M data processing branches for cleaning the collected data of the M data collection clusters based on the M data collection asynchronous factors; A branch resource configuration module, the branch resource configuration module is used to perform resource configuration on the M data processing branches according to the data information entropy of the M data acquisition clusters, and generate M branch resource configuration information; A data cleaning processing module, the data cleaning processing module is used to construct a data cleaning processing network layer based on the M branch resource configuration information and the M data processing branches, wherein the data cleaning processing network layer is used to perform data cleaning processing on the collected data of the data collection unit array of the target building area; The steps performed by the asynchronous fluctuation identification module further include: Constructing K monitoring data boxes based on the K monitoring data sets; Determine K fluctuation identification scales and K fluctuation identification directions according to the distances of the K medians of the K monitoring data boxes relative to the centers of the K monitoring data boxes; Eliminate edge values ​​of data exceeding a preset edge in the K monitoring data boxes to generate K cleaning monitoring data sets; Performing data asynchronous fluctuation identification on the K cleaning monitoring data sets according to the K fluctuation identification scales and the K fluctuation identification directions, and determining K data asynchronous factors; Constructing K identification spaces based on the K cleaning monitoring data sets, wherein the K identification spaces include K identification particle sets, and each identification particle corresponds to one cleaning monitoring data; Extracting K central particles of the K identification spaces, taking the K central particles as starting points, and moving in the K identification spaces according to the K fluctuation identification scales and the K fluctuation identification directions to obtain K iterative identification particles; Perform particle iterative updating according to the K iteratively identified particles and the K central particles to determine K stage identified particles; Iterating in the K recognition spaces with the K stage recognition particles as the starting point until a preset number of iterations is met or the density difference between the stage recognition particles obtained in two adjacent iterations is less than a preset density difference value, and taking the K stage recognition particles obtained in the last iteration as K target recognition particles; Calculate the difference between K cleaning monitoring data and K preset standard monitoring data corresponding to the K target identification particles, and compare the calculation results with the K preset standard monitoring data to generate K data asynchronous factors; The steps performed by the clustering module also include: Acquire the central layout position of the data processing center in the target building area; Performing distance analysis according to the central layout position and the K unit layout positions to generate K position asynchronous factors, including: calculating the distance from each data acquisition unit to the data processing center according to the central layout position and the K unit layout positions, and dividing the K distances by the sum of the K distances respectively, and using the calculated ratios as K position asynchronous factors, wherein the position asynchronous factors reflect the distance difference between each data acquisition unit and the data processing center; The K data acquisition units are divided into clusters of the same type according to the K data asynchronous factors and the K position asynchronous factors to generate the M data acquisition clusters.

2. A data cleaning and processing system for the entire construction process as claimed in claim 1, characterized in that: The steps performed by the asynchronous fluctuation identification module also include: Acquire K iteration densities of the K iteration identified particles and K center densities of the K center particles; Determine whether the K iteration densities are greater than or equal to the K center densities, and if so, use the K iteration identification particles as K stage identification particles; If not, a random number generator is called to generate a first random number and a second random number, and when the first random number is greater than or equal to the second random number, the K central particles are used as K stage identification particles.

3. A data cleaning and processing system for the entire construction process as claimed in claim 2, characterized in that: When the first random number is less than or equal to the second random number, the K iterative identification particles are used as the K stage identification particles, and the moving direction at least in the next movement is the opposite direction of the K wave identification directions.

4. A data cleaning and processing system for the entire construction process as claimed in claim 1, characterized in that: The steps performed by the data processing branch construction module also include: Construct asynchronous factor-extract frequency mapping relationship; Using the M data collection asynchronous factors as indexes, searching the asynchronous factor-extraction frequency mapping relationship to generate M extraction frequencies; The M data processing branches are constructed based on the M extraction frequencies.

5. A method for cleaning and processing data in the whole process of building construction, characterized in that: The method for cleaning and processing data during the whole construction process is performed by a system for cleaning and processing data during the whole construction process according to any one of claims 1 to 4, and the method comprises: Acquire a data acquisition unit array of a target building area, wherein the data acquisition unit array includes K data acquisition units and K unit layout positions; Traversing the K data acquisition units within a preset monitoring window to extract monitoring data and obtain K monitoring data sets; Traversing the K monitoring data sets to identify data asynchronous fluctuations, and determining K data asynchronous factors, where the data asynchronous factors are indicators or parameters used to quantify and represent data asynchronous fluctuations, and reflect the degree and characteristics of data fluctuations; Clustering the K data acquisition units based on the K data asynchronous factors and the K unit layout positions to generate M data acquisition clusters, wherein the M data acquisition clusters include the M data acquisition asynchronous factors; Based on the M data collection asynchronous factors, M data processing branches are constructed to clean the collected data of the M data collection clusters; Performing resource configuration on the M data processing branches according to the data information entropy of the M data acquisition clusters to generate M branch resource configuration information; A data cleaning processing network layer is constructed based on the M branch resource configuration information and the M data processing branches, wherein the data cleaning processing network layer is used to perform data cleaning processing on the collected data of the data collection unit array of the target building area.

Images