Data processing system for correcting abnormal data

Abstract

The invention relates to a data processing system for correcting abnormal data, the system comprises a processor and a memory in which a computer program is stored, and when the computer program is executed by the processor, the following steps are realized: obtaining target abnormal data, the target abnormal data comprises first abnormal data and second abnormal data, and when the target abnormal data is simple abnormal data, correcting the target abnormal data by adopting a first processing mode, and when the target abnormal data is complex abnormal data, correcting the target abnormal data by adopting a second processing mode. According to the method, the BIM model data can be corrected intelligently, and for complex problems, correction suggestions are provided and confirmed or corrected by a user, so that the accuracy and consistency of the BIM model data can be ensured by the real-time correction mechanism, resource waste is reduced, and the efficiency and quality of building construction are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular to a data processing system for correcting abnormal data. Background Art

[0002] With the informatization and intelligent development of the construction industry, the application of BIM models in the stages of building design, construction, operation and maintenance is becoming more and more extensive. Among them, the BIM model is an integrated digital technology solution that integrates all relevant information of a construction project into a three-dimensional model. However, due to the diversity of data sources, the complexity of data processing and the uncertainty of human operation, the data quality problem in the BIM model has become increasingly prominent, seriously affecting the application effect of BIM technology. The data quality problem needs to be solved urgently. Abnormal data does not affect the data quality of the BIM model and it is difficult to ensure the smooth progress of BIM application. Therefore, how to correct abnormal data has become a hot research direction. By accurately correcting the monitored abnormal data, it can promote the transformation of the construction industry to digitalization and intelligence, thereby improving the overall competitiveness of the construction industry. Summary of the invention

[0003] In view of the above technical problems, the technical solution adopted by the present invention is: a data processing system for correcting abnormal data, comprising: a processor and a memory storing a computer program, when the computer program is executed by the processor, the following steps are implemented: S1. Obtain target abnormal data, wherein the target abnormal data includes first abnormal data and second abnormal data, wherein the first abnormal data is abnormal data obtained by monitoring graphic data in first building data, and the second abnormal data is abnormal data obtained by monitoring text data and second building data in the first building data, the first building data is building data obtained by transmitting through an API interface of a BIM model system, and the second building data is building data corresponding to a target geographical location obtained by transmitting through a sensor.

[0004] S2, when the target abnormal data is simple abnormal data, the target abnormal data is corrected by a first processing method, wherein the simple abnormal data is data with format problems and data with missing value problems, wherein the format problems include date format inconsistency or non-numeric characters appearing in numeric fields.

[0005] S3, when the target abnormal data is complex abnormal data, the target abnormal data is corrected by the second processing method, wherein the complex abnormal data is data with problems other than the problems corresponding to the simple abnormal data, and the method of correcting the target abnormal data by the second processing method is to recommend a correction operation method to the user, and then the user reviews the recommended correction operation method and selects the final recommended correction operation method to correct the target abnormal data.

[0006] Compared with the prior art, the present invention has obvious beneficial effects. By means of the above technical solution, a data processing system for correcting abnormal data provided by the present invention can achieve considerable technical advancement and practicality, and has wide industrial utilization value, and has at least the following beneficial effects: The present invention is a data processing system for correcting abnormal data, the system comprising: a processor and a memory storing a computer program, when the computer program is executed by the processor, the following steps are implemented: obtaining target abnormal data, wherein the target abnormal data comprises first abnormal data and second abnormal data, when the target abnormal data is simple abnormal data, the target abnormal data is corrected by a first processing method, when the target abnormal data is complex abnormal data, the target abnormal data is corrected by a second processing method, wherein the method of correcting the target abnormal data by the second processing method is to correct the target abnormal data by recommending a correction operation method to a user and then the user reviews the recommended correction operation method and selects the final recommended correction operation method. The present invention adopts different correction methods for different types of abnormal data, and can intelligently correct simple errors, and provide correction suggestions for complex problems, and the user confirms or modifies them. This real-time correction mechanism can ensure the accuracy and consistency of BIM model data, reduce resource waste, improve the efficiency and quality of construction, and ensure the quality of the building.

[0007] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the following specifically cites a preferred embodiment and describes it in detail with the accompanying drawings as follows. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 A flowchart of a computer program executed by a data processing system for correcting abnormal data provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0009] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0010] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product, or server that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products, or devices.

[0011] The present invention is a data processing system for correcting abnormal data, the system comprising: a processor and a memory storing a computer program, when the computer program is executed by the processor, the following steps are implemented, such as Figure 1 As shown: S1. Obtain target abnormal data, wherein the target abnormal data includes first abnormal data and second abnormal data, wherein the first abnormal data is abnormal data obtained by monitoring graphic data in first building data, and the second abnormal data is abnormal data obtained by monitoring text data and second building data in the first building data, the first building data is building data obtained by transmitting through an API interface of a BIM model system, and the second building data is building data corresponding to a target geographical location obtained by transmitting through a sensor.

[0012] Specifically, the first building data includes building geometry data, building attribute data and building association data, wherein the building association data includes logical relationship data and spatial relationship data between building components, for example: building geometry data such as building size, location and shape, and building attribute data such as building material type, cost and manufacturer information.

[0013] Furthermore, the target geographical location is a geographical location corresponding to a construction site.

[0014] Furthermore, the second construction data includes construction progress data, building material usage data and target geographic location equipment data, including, for example: construction progress data such as daily engineering volume, building material usage data such as building material consumption rate and inventory quantity, and target geographic location equipment data such as on / off time of equipment used in the target geographic location and fault information.

[0015] Specifically, the first building data and the second building data are data that have been subjected to real-time stream processing and data compression.

[0016] Specifically, the method for monitoring data quality in this embodiment processes the first building data and the second building data under a real-time stream processing framework, for example, a real-time stream processing framework such as Apache Kafka or Apache Flink.

[0017] Specifically, S1 also includes the following steps: S100, based on the graphic data in the first building data, obtaining a plurality of key building elements, wherein the key building elements are building elements obtained by processing the graphic data in the first building data using a Canny edge detection algorithm and a gray level co-occurrence matrix.

[0018] Specifically, the graphic data is data that displays building structures and building elements in a graphical manner.

[0019] Specifically, the building elements are basic units and components constituting a building, such as walls, doors, windows and other building elements.

[0020] Specifically, S100 also includes the following steps: S101, obtaining a sample image list set, wherein the sample image list set includes a plurality of sample image lists, the sample image lists include a plurality of sample images, the sample images are acquired images including a plurality of specific architectural elements, the specific architectural elements corresponding to the sample images in each sample image list are of the same type, and the specific architectural elements corresponding to the sample images in different sample image lists are of different types.

[0021] S102, obtaining shape features and texture features corresponding to each sample image in the sample image list set, wherein the shape features corresponding to the sample image are the shape features obtained by performing a Canny edge detection algorithm on the sample image, and the texture features corresponding to the sample image are the texture features obtained by performing a grayscale co-occurrence matrix calculation on the sample image after performing a Canny edge detection algorithm.

[0022] S103, obtaining shape features and texture features corresponding to the candidate image to be matched, wherein the candidate image to be matched is any image obtained from the graphic data in the first building data, the shape features corresponding to the candidate image to be matched are the shape features obtained by performing a Canny edge detection algorithm on the candidate image to be matched, and the texture features corresponding to the candidate image to be matched are the texture features obtained by performing a grayscale co-occurrence matrix calculation on the candidate image to be matched after performing a Canny edge detection algorithm.

[0023] S104, obtaining a first similarity between the candidate image to be matched and each sample image, wherein the first similarity is a similarity obtained by using a normalized cross correlation (NCC) algorithm based on a shape feature corresponding to the candidate image to be matched and a shape feature corresponding to each sample image.

[0024] S105, obtaining a second similarity between the candidate image to be matched and each sample image, wherein the second similarity is the similarity between the texture features corresponding to the candidate image to be matched and the texture features corresponding to each sample image obtained by using a cosine similarity algorithm after normalizing the texture features corresponding to the candidate image to be matched and the texture features corresponding to each sample image respectively.

[0025] S106: When the sum of the first similarity and the second similarity between the candidate image to be matched and a sample image is greater than zero and is a maximum value, a specific architectural element in the sample image is obtained as a key architectural element corresponding to the candidate image to be matched.

[0026] S200, traversing all the key building elements among the plurality of key building elements, and obtaining the spatial relationship between the plurality of key building elements, wherein the spatial relationship is obtained by embedding a logic judgment statement generated based on a spatial relationship rule in a script, and identifying the spatial relationship between the key building elements according to the position relationship and size corresponding to the key building elements.

[0027] Specifically, those skilled in the art know that the spatial relationship rules can be set according to actual needs, all of which fall within the protection scope of the present invention and will not be described in detail here.

[0028] Specifically, the spatial relationship includes but is not limited to an adjacent relationship, a containing relationship, an intersecting relationship and a directional relationship.

[0029] S300, monitoring the graphic data in the first building data based on the spatial relationship to determine the first abnormal data, wherein when the spatial relationship corresponding to a certain graphic data in the first building data is missing or the spatial relationship matching is incorrect or the number of spatial relationships exceeds a preset threshold, the spatial relationship data corresponding to this graphic data is determined to be the first abnormal data.

[0030] Specifically, those skilled in the art know that the setting of the quantity threshold value can be preset according to the actual demand model, and all of them fall within the protection scope of the present invention.

[0031] S400: Acquire second abnormal data based on text data in the first building data and the second building data.

[0032] Specifically, S400 also includes the following steps: S401, performing data preprocessing on the second building data, wherein the preprocessing includes data cleaning, data deduplication and data standardization.

[0033] Specifically, the data cleaning is to use the data processing tool Pandas to identify and process missing values, error values ​​and outliers, the data deduplication is to use a hash function to calculate the hash value of each record in the second building data, and quickly identify duplicates by comparing the hash values, and the data standardization is to convert the data according to a unified format or standard.

[0034] S402, obtaining second abnormal data, wherein the second abnormal data is abnormal data obtained by using a K-means clustering algorithm and a DBSCAN algorithm to preprocess the second building data and the text data in the first building data.

[0035] S403, marking an abnormality category for the second abnormal data, wherein the abnormality category includes: missing, format error, and logical error.

[0036] From the above content, we can see that by utilizing the three-dimensional visualization characteristics of the BIM model and combining it with sensor technology, real-time and intelligent monitoring of BIM model data can be achieved. Intelligent data monitoring reduces the workload of manual inspection, reduces labor costs, and improves the efficiency of data monitoring. Different processing methods are used based on different data forms, so that the accuracy of the acquired abnormal data is higher, thereby improving the accuracy of data quality monitoring.

[0037] S2, when the target abnormal data is simple abnormal data, the target abnormal data is corrected by a first processing method, wherein the simple abnormal data is data with format problems and data with missing value problems, wherein the format problems include date format inconsistency or non-numeric characters appearing in numeric fields.

[0038] Specifically, S2 also includes the following steps: S21, when the target abnormal data is data with a format problem, calling a format conversion function corresponding to the format problem to perform format conversion on the target abnormal data.

[0039] S22, when the target abnormal data is data with a missing value problem and the target abnormal data is the first abnormal data, the target abnormal data is processed and supplemented. When the target abnormal data is building geometry data, the model's API interface is called to obtain size and location information for data supplement. When the target abnormal data is building attribute data, the data is supplemented by searching for the attribute supplement of adjacent elements of the same category of the element.

[0040] S23, when the target abnormal data is data with missing values ​​and the target abnormal data is the second abnormal data, the target abnormal data is processed and supplemented, wherein, when the target abnormal data is trendy and non-periodic data, it is supplemented by using a near-data linear interpolation estimation method, when the target abnormal data is trendless and non-periodic data, it is estimated and supplemented by using a statistical method, and when the target abnormal data is trendy and cyclical data, it is supplemented by using seasonal adjustment.

[0041] Specifically, the trend-free data are data that show a clear long-term growth or decline trend over time, but do not follow a cyclical fluctuation pattern, for example, trend-free data such as total construction progress and inventory.

[0042] Specifically, the trendless and non-periodic data are data that neither show a clear long-term growth or decline trend over time nor follow a periodic fluctuation pattern, for example: trendless and non-periodic data such as material consumption rate and daily workload.

[0043] Specifically, the trended and cyclical data are data that present a clear long-term growth or decline trend over time and follow a periodic fluctuation pattern, for example, trended and cyclical data such as equipment operating status.

[0044] Specifically, the corrected data will be quality-monitored again based on the method included in step S500 to verify the reliability of the corrected data. If the corrected data still has logical anomalies, it will be corrected again. If the corrected data is corrected more than three times, manual correction will be performed directly.

[0045] S3, when the target abnormal data is complex abnormal data, the target abnormal data is corrected by the second processing method, wherein the complex abnormal data is data with problems other than the problems corresponding to the simple abnormal data, and the method of correcting the target abnormal data by the second processing method is to recommend a correction operation method to the user, and then the user reviews the recommended correction operation method and selects the final recommended correction operation method to correct the target abnormal data.

[0046] Specifically, the complex abnormal data can be understood as relatively complex data that is not a simple format problem or data missing problem, for example: logical contradictions between data (such as mismatch between room area and door and window sizes), large-scale data anomalies, data spatial relationship errors, and other complex abnormal data.

[0047] Specifically, S3 also includes the following steps: S31, when the target abnormal data is building geometry data, obtaining a target priority, wherein the target priority is a difference between the target abnormal data and the real data.

[0048] S32, when the target priority is greater than a preset priority threshold, the real data corresponding to the building model is obtained through the API interface to correct the target abnormal data.

[0049] Specifically, those skilled in the art know that the preset priority threshold can be selected according to actual needs, all of which fall within the protection scope of the present invention and will not be described in detail here.

[0050] S33: When the target priority is not greater than a preset priority threshold, a geometric information modification model is recommended to the user to correct the abnormal data.

[0051] S34, when the target abnormal data is building attribute data, it is recommended to the user to replace the target abnormal data with the attribute data that appears most frequently in adjacent building elements of the same type.

[0052] S35, when the target abnormal data is building-related data, recommending the first law of geography to the user to correct the target abnormal data.

[0053] Specifically, for the case of missing spatial relationships, the nearest target type building element is searched based on the building element, and recommendations are given. For example, if the wall element lacks an adjacency relationship with other walls, other elements of the wall category are searched nearby, and correction recommendations are given: W08 (elementType: wall) may be adjacent to W09 (elementType: wall) and W10 (elementType: wall), and two adjacency relationships are missing; for the case of incorrect spatial relationships, it is recommended to modify the model, and it is prompted that the element should not have such spatial relationships; for the case of redundant spatial relationships, the farthest few related elements of this type are searched based on the element, and it is recommended to modify the model to delete the redundant spatial relationships.

[0054] S36, when the target abnormal data is time series data, the target abnormal data is corrected using a method consistent with the method of correcting the abnormal data in step S23.

[0055] Specifically, the method for data correction in this embodiment sets an RLHF learning feedback algorithm, and intelligently adjusts the parameter settings and algorithm selection of the correction strategy based on the correction results and user feedback. During the correction process, possible algorithms are spontaneously matched for correction, and the calculation results of multiple strategies are fed back by users. The system collects this feedback training reward model to strengthen the learning strategy, fine-tune the correction strategy, and improve the accuracy and efficiency of the correction.

[0056] From the above content, we can see that different correction methods are used for different types of abnormal data. Simple errors can be corrected intelligently. For complex problems, correction suggestions are provided and confirmed or modified by the user. This real-time correction mechanism can ensure the accuracy and consistency of BIM model data, reduce resource waste, improve the efficiency and quality of construction, and ensure the quality of the building.

[0057] The present invention is a data processing system for correcting abnormal data, the system comprising: a processor and a memory storing a computer program, when the computer program is executed by the processor, the following steps are implemented: obtaining target abnormal data, wherein the target abnormal data comprises first abnormal data and second abnormal data, when the target abnormal data is simple abnormal data, the target abnormal data is corrected by a first processing method, when the target abnormal data is complex abnormal data, the target abnormal data is corrected by a second processing method, wherein the method of correcting the target abnormal data by the second processing method is to correct the target abnormal data by recommending a correction operation method to a user and then the user reviews the recommended correction operation method and selects the final recommended correction operation method. The present invention adopts different correction methods for different types of abnormal data, and can intelligently correct simple errors, and provide correction suggestions for complex problems, and the user confirms or modifies them. This real-time correction mechanism can ensure the accuracy and consistency of BIM model data, reduce resource waste, improve the efficiency and quality of construction, and ensure the quality of the building.

[0058] Although some specific embodiments of the present invention have been described in detail by way of example, it will be appreciated by those skilled in the art that the above examples are for illustration only and are not intended to limit the scope of the present invention. It will also be appreciated by those skilled in the art that various modifications may be made to the embodiments without departing from the scope and spirit of the present invention. The scope of the present invention is defined by the appended claims.

Claims

1. A data processing system for correcting abnormal data, characterized in that: The system comprises: a processor and a memory storing a computer program, and when the computer program is executed by the processor, the following steps are implemented: S1, obtaining target abnormal data, wherein the target abnormal data includes first abnormal data and second abnormal data, wherein the first abnormal data is abnormal data obtained by monitoring graphic data in first building data, and the second abnormal data is abnormal data obtained by monitoring text data and second building data in the first building data, the first building data is building data obtained by transmitting through an API interface of a BIM model system, and the second building data is building data corresponding to a target geographical location obtained by transmitting through a sensor; S2, when the target abnormal data is simple abnormal data, the target abnormal data is corrected by a first processing method, wherein the simple abnormal data is data with format problems and data with missing value problems, wherein the format problems include date format mismatch or non-numeric characters appearing in numeric fields; S3, when the target abnormal data is complex abnormal data, the target abnormal data is corrected by the second processing method, wherein the complex abnormal data is data with problems other than the problems corresponding to the simple abnormal data, and the method of correcting the target abnormal data by the second processing method is to recommend a correction operation method to the user, and then the user reviews the recommended correction operation method and selects the final recommended correction operation method to correct the target abnormal data.

2. The data processing system for correcting abnormal data according to claim 1, characterized in that: The first building data includes building geometry data, building attribute data and building association data, wherein the building association data includes logical relationship data and spatial relationship data between building components.

3. The data processing system for correcting abnormal data according to claim 1, characterized in that: The second construction data includes construction progress data, construction material usage data and target geographic location equipment data.

4. The data processing system for correcting abnormal data according to claim 1, characterized in that: S2 also includes the following steps: S21, when the target abnormal data is data with a format problem, calling a format conversion function corresponding to the format problem to convert the format of the target abnormal data; S22, when the target abnormal data is data with a missing value problem and the target abnormal data is the first abnormal data, the target abnormal data is processed and supplemented, wherein, when the target abnormal data is building geometry data, the model API interface is called to obtain size and position information for data supplementation, and when the target abnormal data is building attribute data, the data is supplemented by searching for the attribute supplement of the adjacent elements of the same category of the element; S23, when the target abnormal data is data with missing values ​​and the target abnormal data is the second abnormal data, the target abnormal data is processed and supplemented, wherein, when the target abnormal data is trendy and non-periodic data, it is supplemented by using a near-data linear interpolation estimation method, when the target abnormal data is trendless and non-periodic data, it is estimated and supplemented by using a statistical method, and when the target abnormal data is trendy and cyclical data, it is supplemented by using seasonal adjustment.

5. The data processing system for correcting abnormal data according to claim 4, characterized in that: The trended non-cyclical data refers to data that presents a clear long-term growth or decline trend over time, but does not follow a cyclical fluctuation pattern.

6. The data processing system for correcting abnormal data according to claim 4, characterized in that: The non-trend and non-periodic data are data that neither show a clear long-term growth or decline trend over time nor follow a periodic fluctuation pattern.

7. The data processing system for correcting abnormal data according to claim 4, characterized in that: The trended and cyclical data refers to data that presents a clear long-term growth or decline trend over time and follows a cyclical fluctuation pattern.

8. The data processing system for correcting abnormal data according to claim 1, characterized in that: S3 also includes the following steps: S31, when the target abnormal data is building geometry data, obtaining a target priority, wherein the target priority is a difference between the target abnormal data and the real data; S32, when the target priority is greater than the preset priority threshold, the real data corresponding to the building model is obtained through the API interface to correct the target abnormal data; S33, when the target priority is not greater than a preset priority threshold, recommending a geometric information modification model to the user to correct the abnormal data; S34, when the target abnormal data is building attribute data, it is recommended to the user to replace the target abnormal data with the attribute data that appears most frequently in adjacent building elements of the same type; S35, when the target abnormal data is building-related data, recommending the first law of geography to the user to correct the target abnormal data; S36, when the target abnormal data is time series data, the target abnormal data is corrected using a method consistent with the method of correcting the abnormal data in step S23.

Images