Internet-based real estate data cleaning method

Abstract

The invention provides a real estate data cleaning method based on the Internet, and belongs to the technical field of data cleaning, and the method comprises the steps: collecting real estate data from a plurality of Internet data sources by using a web crawler technology, monitoring the collection process based on a monitoring tool to generate monitoring data, and setting a transmission set for each piece of real estate information under each data source; respectively counting the type distribution of each house type; performing anomaly identification on the real estate information, and when the anomaly type is related to transmission anomaly, analyzing the update type of the corresponding transmission set to perform anomaly update on the corresponding real estate information; when the exception type is irrelevant to the transmission exception, analyzing an input behavior corresponding to the internet data source, searching and identifying similar information by using an identification field corresponding to exception information, and performing error updating; based on distribution of all types and authority of each internet data source, all house types are arranged in sequence, and format conversion and classified storage are carried out. And the quality and availability of the real estate data are effectively improved.

Description

Technical Field

[0001] The present invention relates to the technical field of data cleaning, and particularly to an Internet-based real estate data cleaning method. Background Art

[0002] With the rapid development of Internet technology, the real estate industry has generated a vast amount of data. These data come from a wide range of sources, including real estate trading platforms, real estate agency websites, real estate management department websites, etc. However, due to the diversity of data sources and various factors in the data collection process, these real estate data often have problems such as data anomalies and inconsistent data formats. For example, different websites may use different units to represent the housing area, some in square meters and some in square feet; some data may be missing or incorrect due to network transmission problems. These problems seriously affect the quality and usability of real estate data, greatly reducing the accuracy and reliability of subsequent analysis, decision-making, and other work based on these data.

[0003] Therefore, the present invention proposes an Internet-based real estate data cleaning method. Summary of the Invention

[0004] The present invention provides an Internet-based real estate data cleaning method, which is used to obtain a transmission set by monitoring the collection process through monitoring work, and then ensure the reasonable correction of abnormal data by identifying anomalies in the information under different housing types, ensuring the reliability of the retained data. Further, by combining the type distribution and authority, format conversion is performed on the updated data to ensure format uniformity while meeting different format viewing requirements, effectively improving the quality and usability of real estate data.

[0005] The present invention provides an Internet-based real estate data cleaning method, including: Step 1: Use web crawler technology to collect real estate data from multiple Internet data sources. At the same time, monitor the collection process based on a monitoring tool to generate monitoring data, and set a transmission set for each piece of real estate information under each data source, where the transmission set is related to data quality and transmission constraints; Step 2: Count the housing types of each piece of real estate information in each Internet data source respectively to obtain the type distribution of the corresponding Internet data source; Step 3: Identify anomalies for each piece of real estate information under the corresponding housing type respectively. When the anomaly type is related to transmission anomalies, analyze the update type of the corresponding transmission set to perform anomaly updates on the corresponding real estate information; Step 4: When the anomaly type is not related to transmission anomalies, analyze the input behavior of the corresponding Internet data source, use the identification field of the corresponding anomaly information to search for and identify similar information, and perform error updates; Step 5: Based on all types of distributions and the authority of each Internet data source, arrange all housing types in sequence, and then perform format conversion and classified storage on each piece of updated information in turn.

[0006] Preferably, set a transmission set for the real estate data under each data source, including: Disassemble the monitoring data under each data source according to the transmission constraints, and input it into the corresponding constraint analysis model to obtain the judgment results of the corresponding transmission constraints at each transmission process time point, and obtain a judgment set, where the judgment results are to meet the corresponding transmission constraints and not meet the corresponding transmission constraints; Construct a judgment matrix by combining all judgment sets, and analyze the first variance of each column vector and the second variance of each row vector; Synchronously lock the first column with the first variance greater than the first threshold and the first row with the second variance greater than the second threshold to obtain cross elements, and count the first quantity of the cross elements appearing in each row vector and the second quantity of the cross elements appearing in each column vector; If the second quantity is consistent with the set quantity of the transmission constraint, set a low-quality label for the real estate data at the corresponding transmission process time point; If the second quantity is less than the set quantity of the transmission constraint and greater than half of the set quantity, and the first quantity of each row vector corresponding to the transmission constraint under the second quantity is greater than N / 2, set a low-quality label for the real estate data at the corresponding transmission process time point, where N represents the total number of transmission process time points; Otherwise, set the remaining quality labels for the real estate data at the corresponding transmission process time point, where the remaining quality labels include: medium-quality labels and high-quality labels; Based on the label setting results at each transmission process time point and combined with the judgment sets at each transmission process time point, obtain the transmission set.

[0007] Preferably, set the remaining quality labels for the corresponding real estate information at the corresponding transmission process time point, including: If the second quantity is less than the set quantity of the transmission constraint and greater than half of the set quantity, at this time, count the third quantity of the constraints where the first quantity of each row vector corresponding to the transmission constraint under the second quantity is not greater than N / 2; If , at this time, set a medium-quality label for the corresponding real estate information at the corresponding transmission process time point, where is the third quantity; is the set quantity; is the second quantity; is a quality constant, with a value of 0.05; If , set a high-quality label for the corresponding real estate information at the corresponding time point of the transmission process; If the second quantity is less than or equal to half of the set quantity, at this time, combine the fourth quantity of the elements that do not meet the corresponding transmission constraint in each row vector of the transmission constraint corresponding to the second quantity; If , at this time, set an intermediate-quality label for the corresponding real estate information at the corresponding time point of the transmission process, where represents the fourth quantity corresponding to the i1-th transmission constraint under the second quantity; is a quality constant with a value of 0.1; If , set a high-quality label for the corresponding real estate information at the corresponding time point of the transmission process.

[0008] Preferably, perform anomaly recognition on each piece of real estate information under the corresponding housing type, including: Detect the transmission duration and transmission integrity of each piece of real estate information under the corresponding housing type. When the detection result meets the transmission anomaly constraint, determine that the anomaly type is related to the transmission anomaly; Otherwise, determine that it is not related to the transmission anomaly.

[0009] Preferably, perform anomaly recognition on each piece of real estate information under the corresponding housing type, including: Lock the judgment information of each piece of real estate information related to the transmission anomaly under the same housing type from the transmission sets under each data source, where the judgment information includes: the information source of the real estate information under the transmission anomaly, the transmission set of the real estate information under the transmission anomaly; Perform cluster analysis on all pieces of real estate information related to the transmission anomaly under the same housing type according to the transmission set, and obtain several clusters; Analyze the distribution of the quality labels involved in each cluster, set an average quality label for each corresponding piece of real estate information, and at the same time, count the first total number of high-quality labels, the second total number of intermediate-quality labels, and the third total number of low-quality labels of the same piece of real estate information under all data sources to obtain an auxiliary quality label; According to the average quality label and the auxiliary quality label, determine the update type of the corresponding piece of real estate information, where the update type includes: deletion type, partial replacement type, and full replacement type; Perform anomaly update on the corresponding real estate information according to the update type.

[0010] Preferably, analyze the input behavior of the corresponding Internet data source, search and identify similar information using the identification field of the corresponding anomaly information, and perform error update, including: Input the input behavior into a behavior analysis model to obtain possible factors causing information errors; Rely on the possible factors to repair the abnormal information to obtain repaired information; Use the identification field corresponding to the abnormal information to search for and identify the identification fields of each correct information from all correct information to obtain the first information; If the first information exists, replace the corresponding abnormal information with the corresponding abnormal type according to the first information; If the first information does not exist, at this time, use the corresponding repaired information to sequentially match real estate matters from all correct information to obtain the second information, and realize the wrong update of the corresponding abnormal information.

[0011] Preferably, based on all type distributions and the authority of each Internet data source, arrange all housing types in sequence, including: Determine the authority according to the application popularity and application favorable comment coefficient of each Internet data source; Adjust the type distribution under the corresponding data source according to the authority of each Internet data source to obtain an adjusted distribution; Overlay all adjusted distributions to obtain the final distribution of all housing types; Arrange all housing types in sequence according to the quantity value of each housing type in the final distribution.

[0012] Preferably, determine the update type corresponding to a real estate information, including: When the priority of the average quality label is higher than the priority of the quality label of the corresponding real estate information at the corresponding time point, determine that the update type of the corresponding real estate information is the deletion type; When the priority of the average quality label is not higher than the average priority of the quality labels of the corresponding real estate information at the last time point under different Internet data sources, determine whether the priority of the auxiliary quality label and the average quality label is higher than the average priority of the quality labels of the corresponding real estate information at the last time point under different Internet data sources; If it is higher, determine that the update type of the corresponding real estate information is the partial replacement type; Otherwise, determine that the update type of the corresponding real estate information is the full replacement type.

[0013] Compared with the prior art, the beneficial effects of this application are as follows: Monitor the collection process based on monitoring work to obtain a transmission set, and then identify anomalies in the information under different housing types to ensure reasonable correction of abnormal data, ensuring the reliability of the retained data. Further, combine the type distribution and authority to perform format conversion on the updated data to meet different format viewing requirements while ensuring format uniformity, effectively improving the quality and usability of real estate data.

[0014] Other features and advantages of the present invention will be described in the following specification, and in part will be obvious from the specification, or can be understood by implementing the present invention. The objectives and other advantages of the present invention can be achieved and obtained by the structures specifically pointed out in the written specification and the drawings.

[0015] The technical solution of the present invention will be further described in detail below through the drawings and embodiments. Description of the Drawings

[0016] The drawings are used to provide a further understanding of the present invention, and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation to the present invention. In the drawings: Figure 1 It is a flowchart of a method for cleaning real estate data based on the Internet in an embodiment of the present invention. Detailed Embodiments

[0017] The following describes the preferred embodiments of the present invention with reference to the drawings. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present invention and are not used to limit the present invention.

[0018] The present invention provides a method for cleaning real estate data based on the Internet, as Figure 1 shown, including: Step 1: Use web crawler technology to collect real estate data from multiple Internet data sources. At the same time, monitor the collection process based on a monitoring tool to generate monitoring data, and set a transmission set for each piece of real estate information under each data source, where the transmission set is related to data quality and transmission constraints; Step 2: Count the housing type of each piece of real estate information in each Internet data source respectively to obtain the type distribution of the corresponding Internet data source; Step 3: Identify anomalies for each piece of real estate information under the corresponding housing type respectively. When the anomaly type is related to transmission anomalies, analyze the update type of the corresponding transmission set to perform anomaly updates on the corresponding real estate information; Step 4: When the anomaly type is not related to transmission anomalies, analyze the input behavior of the corresponding Internet data source, use the identification field of the corresponding anomaly information to search for and identify similar information, and perform error updates; Step 5: Based on all types of distributions and the authority of each Internet data source, arrange all housing types in sequence, and then perform format conversion and classified storage on each piece of updated information in turn.

[0019] In this embodiment, the web crawler technology uses a crawler based on the HTTP protocol, which belongs to the prior art. That is, real estate data is obtained by crawling different platforms (data sources). Among them, the real estate data includes several pieces of real estate information, and the real estate information includes specific building details, involving housing area, housing type, housing price, etc.

[0020] In this embodiment, the Internet data sources include, but are not limited to, real estate trading platforms, real estate agency websites, real estate management department websites, etc.

[0021] In this embodiment, Zabbix is used as the monitoring tool, that is, Zabbi monitors the network connection status between the data collection program and each data source and data collection matters. For example, it monitors indicators such as network latency, packet loss rate, security attacks, collection duration, collection rate, collection success rate, and data integrity of different housing matters (housing area, housing location, housing price, etc.), and regards them as monitoring data. And the data quality refers to the quality of the collected real estate data, and the transmission constraint refers to the set constraints for monitoring indicators. For example, the packet loss rate cannot be higher than 8%. And the specific collection process includes: First, according to the preset interface specification, the data collection program sends a data request containing specific query conditions (such as querying all residential real estate information in a certain area) to the government real estate management department database. After the database receives the request, it performs data retrieval and returns the qualified data to the collection program, that is, Zabbix will continuously play a role in this collection process.

[0022] In this embodiment, the collection process is monitored based on the monitoring tool, specifically for each piece of real estate information, that is, to monitor the possible situations encountered by the real estate information during the transmission process to ensure the quality of the information.

[0023] In this embodiment, the housing type is related to the housing type to which the real estate information belongs. For example, new housing type, second-hand housing type, foreclosed housing type, etc.

[0024] In this embodiment, the format conversion is the conversion of the housing area unit and the housing date format. Because different people have different viewing experiences for units and formats, it is necessary to perform conversion. For example, there are two types of area units: Area Unit 1, Area Unit 2; There are two types of housing date formats: Date Format 1, Date Format 2; At this time, each piece of update information needs to be converted and stored in four ways: area unit 1 - date format 1, area unit 2 - date format 1, area unit 1 - date format 2, and area unit 2 - date format 2, which facilitates the direct conversion of units and formats during the subsequent viewing process by the crowd and meets the viewing experience of users.

[0025] In this embodiment, anomaly recognition is to recognize the transmission duration and transmission integrity of real estate information. If the transmission duration is greater than the preset duration or the transmission integrity is less than the preset integrity, at this time, it is determined to be related to transmission anomalies.

[0026] In this embodiment, the update types of the transmission set are: partial replacement type, full replacement type, and deletion type, and then anomaly updates are implemented according to the replacement type.

[0027] In this embodiment, the input behavior is used as a basis for source analysis of whether the information is abnormal, and it is determined whether there is an abnormality itself by judging the input behavior.

[0028] In this embodiment, the identification field refers to the unique identifier of the abnormal information. For example, the house number, and each house has only one number. Because the same real estate information may appear on different platforms, so, searching based on the identification field can obtain relevant information to achieve the accuracy of error updates.

[0029] In this embodiment, the authority is realized based on the wide application degree and application favorable comment coefficient of the data source, and the value range is (0, 1).

[0030] In this embodiment, the sequence of house types is to sort the priority of data format conversion under different house types to ensure the orderly progress of data format conversion.

[0031] In this embodiment, classified storage is classified storage according to the format type.

[0032] The beneficial effects of the above technical solution are: based on the monitoring work to monitor the acquisition process to obtain the transmission set, and then by identifying anomalies in the information under different house types to ensure the reasonable correction of abnormal data, ensure the reliability of the retained data, and further combine the type distribution and authority to perform format conversion on the updated data to ensure format unity while meeting different format viewing requirements, effectively improving the quality and usability of real estate data.

[0033] The present invention provides an Internet-based real estate data cleaning method, which sets a transmission set for the real estate data under each data source, including: The monitoring data under each data source is disassembled according to the transmission constraints and input into the corresponding constraint analysis model to obtain the judgment results of the corresponding transmission constraints of each real estate information at each transmission process time point, and a judgment set is obtained. Among them, the judgment results are to meet the corresponding transmission constraints and not meet the corresponding transmission constraints; All judgment sets are constructed into a judgment matrix, and the first variance of each column vector and the second variance of each row vector are analyzed; Synchronously lock the first column with the first variance greater than the first threshold and the first row with the second variance greater than the second threshold to obtain cross elements, and count the first quantity of the cross elements appearing in each row vector and the second quantity of the cross elements appearing in each column vector; If the second quantity is consistent with the set quantity of the transmission constraint, set a low-quality label for the corresponding real estate information at the corresponding transmission process time point; If the second quantity is less than the set quantity of the transmission constraint and greater than half of the set quantity, and the first quantity of each row vector corresponding to each transmission constraint under the second quantity is greater than N / 2, set a low-quality label for the corresponding real estate information at the corresponding transmission process time point, where N represents the total number of transmission process time points; Otherwise, set the remaining quality labels for the corresponding real estate information at the corresponding transmission process time point, where the remaining quality labels include: medium-quality labels and high-quality labels; Based on the label setting results at each transmission process time point and combined with the judgment results of the corresponding transmission constraints at each transmission process time point, a transmission set of the corresponding real estate information is obtained.

[0034] In this embodiment, the transmission set of the corresponding real estate information = {the judgment results and label setting results of the corresponding transmission constraints at each transmission process time point}.

[0035] In this embodiment, the constraint analysis model is pre-trained and obtained by training a neural network model with the transmission constraints and whether the disassembled data meets the transmission constraints as samples. Therefore, the judgment results of the disassembled data under the corresponding transmission constraints at each transmission process time point can be directly obtained. It should be noted that the monitoring data (disassembled data) involved at each transmission time point involves relevant data for different transmission constraints. For example, the transmission rate at time 1 is within the preset transmission range. At this time, if the transmission constraint is met, the judgment result is regarded as 1, and if not, the judgment result is regarded as 0, and each real estate information is analyzed in turn.

[0036] In this embodiment, each row of the judgment matrix is the judgment result of the corresponding transmission constraint at different time points of the transmission process, and each column of the judgment matrix is the judgment result of different transmission constraints at the same time point of the transmission process. It should be noted that there are multiple rows and multiple columns in the judgment matrix and the number is at least greater than 8. At this time, for convenience, taking the existence of 3 time moments and 3 constraints as an example, at this time, the judgment matrix is as follows: , at this time, the row vectors {110}, {010}, {110} are the corresponding judgment sets respectively. Furthermore, based on this matrix, the variance of each column vector and each row vector is calculated.

[0037] In this embodiment, the first threshold is 0.1 and the second threshold is 0.08.

[0038] In this embodiment, the set quantity is the same as the quantity of the transmission constraints. If the first column in the judgment matrix refers to the 1st column and the 5th column in this matrix, and the first row refers to the 2nd row and the 3rd row in this matrix, at this time, the cross elements are: the elements in the 1st column of the 2nd row, the 5th column of the 2nd row, the 1st column of the 3rd row, and the 3rd column of the 3rd row. After determining the cross elements, the quantity of the cross elements in different rows and columns can be directly obtained. The quality labels include: low-quality label, high-quality label, and intermediate-quality label. Generally, when the corresponding variance threshold is not satisfied, the larger the value of the corresponding variance, the poorer the data quality of the corresponding row or column may be.

[0039] The beneficial effects of the above technical solution are: By obtaining the judgment set of the disassembled data at each time point of the transmission process to construct a matrix, the quality of the data at the corresponding time point is reasonably divided, providing an analysis basis for subsequent data update, and ensuring the reliability of the quality analysis of the corresponding real estate information.

[0040] The present invention provides an Internet-based real estate data cleaning method, and sets other quality labels for the corresponding real estate information at the corresponding time point of the transmission process, including: If the second quantity is less than the set quantity of the transmission constraints and greater than half of the set quantity, at this time, count the third quantity of the constraints where the first quantity of each transmission constraint corresponding to the row vector under the second quantity is not greater than N / 2; If , at this time, set an intermediate-quality label for the corresponding real estate information at the corresponding time point of the transmission process, where is the third quantity; is the set quantity; is the second quantity; is a quality constant, and the value is 0.05; If , set a high-quality label for the corresponding real estate information at the corresponding time point of the transmission process; If the second quantity is less than or equal to half of the set quantity, at this time, combine the fourth quantity of the elements that do not meet the corresponding transmission constraint in each row vector corresponding to the second quantity; If , at this time, set an intermediate quality label for the corresponding real estate information at the corresponding transmission process time point, where represents the fourth quantity corresponding to the i1-th transmission constraint under the second quantity; is a quality constant with a value of 0.1; If , set a high-quality label for the corresponding real estate information at the corresponding transmission process time point.

[0041] In this embodiment, if the second quantity is 5 and the set quantity is 8, at this time, 5 is between 4 and 8. Then, the first quantities in the row vector corresponding to the second quantity are 3, 6, 5, and 2 respectively, and N is 10. At this time, the third quantity of the first quantity in the corresponding row vector that has a constraint not greater than N / 2 is 2, that is, the quantities corresponding to the first quantity being 3 and 2.

[0042] In this embodiment, the fourth quantity of the elements that do not meet the corresponding transmission constraint refers to the number of elements with a value of 0 in the corresponding row vector.

[0043] The beneficial effect of the above technical solution is: By comparing the second quantity with half of the set quantity, and further combining the third quantity of the constraint and the fourth quantity of the element, the rationality of setting the quality label for the real estate information is realized.

[0044] The present invention provides an Internet-based real estate data cleaning method for anomaly recognition of each piece of real estate information under the corresponding housing type, including: Detect the transmission duration and transmission integrity of each piece of real estate information under the corresponding housing type. When the detection result meets the transmission anomaly constraint, determine that the anomaly type is related to the transmission anomaly; Otherwise, determine that it is not related to the transmission anomaly.

[0045] In this embodiment, the transmission anomaly constraint is: the transmission duration is greater than the preset duration, or the transmission integrity is less than the preset integrity. For example, when the transmission duration is 0.3s and the preset duration is 0.2s, at this time, it is determined as a transmission anomaly. It should be noted that the value of the preset integrity is 0.8, and the value of the transmission integrity is 1, that is: transmission integrity = sim (the information obtained after transmission, the original information before transmission), and sim represents the similarity function, which belongs to the prior art.

[0046] The beneficial effects of the above technical solution are as follows: By detecting the transmission duration and transmission integrity of real estate information, it can be effectively determined whether it is related to abnormal transmission, providing a basis for subsequent analysis.

[0047] The present invention provides an Internet-based real estate data cleaning method for identifying abnormalities in each piece of real estate information under the corresponding housing type, including: Lock the judgment information of each piece of real estate information related to transmission abnormality under the same housing type from the transmission sets under each data source, where the judgment information includes: the information source of the real estate information under transmission abnormality, and the transmission set of the real estate information under transmission abnormality; Perform clustering analysis on all pieces of real estate information related to transmission abnormality under the same housing type according to the transmission set, and obtain several clustering groups; Analyze the distribution of quality labels involved in each clustering group, set an average quality label for each corresponding piece of real estate information, and at the same time, count the first total number of high-quality labels, the second total number of medium-quality labels, and the third total number of low-quality labels of the same piece of real estate information under all data sources to obtain an auxiliary quality label; According to the average quality label and the auxiliary quality label, determine the update type of the corresponding piece of real estate information, where the update type includes: deletion type, partial replacement type, and full replacement type; Perform abnormal update on the corresponding real estate information according to the update type.

[0048] In this embodiment, the information source refers to an Internet data source.

[0049] In this embodiment, the clustering analysis is implemented based on the K-means clustering algorithm, which belongs to the prior art. According to the judgment matrix composed of the transmission set, and based on the new matrix obtained by adding the quality label vector to the first row of the judgment matrix, clustering analysis is performed on all pieces of real estate information related to transmission abnormality under the same housing type, and several clustering results can be directly obtained, and each clustering result is regarded as a clustering group.

[0050] In this embodiment, the quality labels of each real estate information at different transmission process time points have been determined before recognition. At this time, by combining the label results of each real estate information involved in the corresponding cluster group, the distribution can be obtained. Specifically: all quality labels of each real estate information involved in the cluster group are included. The average quality label refers to the statistics of the number of high-quality labels, the number of medium-quality labels, and the number of low-quality labels involved in each remaining real estate information in the cluster group except the corresponding real estate information. At this time, the coefficient of the high-quality label is 1, the coefficient of the medium-quality label is 0.2, and the coefficient of the low-quality label is -1. At this time, (1×the number of high-quality labels + 0.2×the number of medium-quality labels + (-1)×the number of low-quality labels) / (the number of high-quality labels + the number of medium-quality labels + the number of low-quality labels), and the result obtained is the coefficient assigned to the average quality label.

[0051] In this embodiment, each real estate information has N quality labels under an Internet data source, which is consistent with the number of transmission process time points. Therefore, the first total number of high-quality labels, the second total number of medium-quality labels, and the third total number of low-quality labels belonging to the same real estate information under all data sources are statistically calculated. The first total number is Z1, the second total number is Z2, and the third total number is Z3. At this time, the result obtained by (1×Z1 + 0.2×Z2 + (-1)×Z3) / (Z1 + Z2 + Z3) is the coefficient assigned to the auxiliary quality label.

[0052] The beneficial effects of the above technical solution are: starting from the transmission set to perform cluster analysis to obtain the distribution and get the average quality label, and then obtaining the auxiliary quality label by statistically calculating the total number of quality labels, effectively determining the update type, ensuring the reliability of the information, and providing a basis for the evaluation of real estate, etc.

[0053] The present invention provides an Internet-based real estate data cleaning method, which analyzes the input behavior of the corresponding Internet data source, searches and identifies similar information by using the identification field of the corresponding abnormal information, and performs error update, including: Input the input behavior into the behavior analysis model to obtain possible factors causing information errors; Repair the abnormal information depending on the possible factors to obtain repaired information; Use the identification field of the corresponding abnormal information to search and identify the identification field of each correct information from all correct information to obtain the first information; If the first information exists, replace the corresponding abnormal information with the corresponding abnormal type according to the first information; If the first information does not exist, at this time, use the corresponding repair information to sequentially match the real estate matters from all correct information to obtain the second information, and realize the incorrect update of the corresponding abnormal information.

[0054] In this embodiment, the behavior analysis model is obtained by training a neural network model with different input behaviors and input errors caused by the behaviors as samples, and the number of training samples is greater than 10,000. Therefore, it is possible to directly obtain the possible factors associated with the input behaviors, which are generally related to human operation errors or system failures. For example, "the input personnel have poor operation habits and are prone to decimal point misalignment", which is a factor that may cause information errors. Decimal point misalignment will cause errors in price, area, etc.

[0055] In this embodiment, the possible repair information means that the originally input area is 30.2 square meters, and after repair, it may become 302 square meters. It should be noted that since each real estate information includes various parameters such as area, location, pattern, price, etc., the repair information obtained after modifying the area still includes various parameters, but only the value of one parameter has changed.

[0056] In this embodiment, the identification field is the house number, that is, the real estate registration number. During the transaction process of real estate, there will be a transaction number, and the real estate information recorded in the contract under its number can be regarded as correct information. If the data is not abnormal under different data sources, it can also be regarded as normal information at this time, and then search to obtain the first information.

[0057] In this embodiment, each real estate matter in the repair information is respectively matched with each real estate matter of each correct information to determine the information with the highest matching degree as the second information, and the replacement principle of the second information and the first information for the corresponding abnormal information is the same.

[0058] The beneficial effects of the above technical solution are: by analyzing the input behavior to obtain possible factors, and then repairing the abnormal information, the abnormal information can be reasonably updated from the identification field of the abnormal information and the real estate matters of the repair information.

[0059] The present invention provides an Internet-based real estate data cleaning method. Based on all type distributions and the authority of each Internet data source, all housing types are arranged successively, including: Determine the authority according to the application breadth and application favorable comment coefficient of each Internet data source; Adjust the type distribution under the corresponding data source according to the authority of each Internet data source to obtain an adjusted distribution; Overlay all the adjusted distributions to obtain the final distribution of all housing types; Arrange all housing types in sequence according to the numerical values of each housing type in the final distribution.

[0060] In this embodiment, authority = breadth of application × coefficient of favorable reviews for application, and breadth of application = number of users / set number, and coefficient of favorable reviews for application = number of five-star favorable reviews / number of all reviews.

[0061] Among them, the set number is 10 million.

[0062] In this embodiment, the type distribution ratio is, for example: 100:30:29:70. At this time, multiply each value by the authority respectively, 100 × authority u1: 30 × authority u1: 29 × authority u1: 70 × authority u1. At this time, there is also 190 × authority u2: 38 × authority u2: 30 × authority u2: 48 × authority u2. Then the obtained final distribution is: 100 × authority u1 + 190 × authority u2: 30 × authority u1 + 38 × authority u2: 29 × authority u1 + 30 × authority u2: 70 × authority u1 + 48 × authority u2.

[0063] In this embodiment, just sort by the numerical results.

[0064] The beneficial effect of the above technical solution is: Adjust the distribution based on the authority, and then through superposition, the final distribution can be obtained to ensure the orderliness of subsequent classification.

[0065] The present invention provides an Internet-based real estate data cleaning method, which determines the update type of corresponding real estate information, including: When the priority of the average quality label is higher than the average priority of the quality labels of the corresponding real estate information at the last moment under different Internet data sources, determine that the update type of the corresponding real estate information is the deletion type; When the priority of the average quality label is not higher than the average priority of the quality labels of the corresponding real estate information at the last moment under different Internet data sources, determine whether the priority of the auxiliary quality label and the average quality label is higher than the average priority of the quality labels of the corresponding real estate information at the last moment under different Internet data sources; If it is higher, determine that the update type of the corresponding real estate information is the partial replacement type; Otherwise, determine that the update type of the corresponding real estate information is the full replacement type.

[0066] In this embodiment, the priorities of the average quality label and the auxiliary quality label are determined numerical values, which are coefficients assigned to the labels.

[0067] In this embodiment, the average priority of the quality labels of the corresponding real estate information at the last moment under different Internet data sources: The quality labels of the corresponding real estate information at the last moment under different data sources are: high quality, low quality. At this time, the corresponding average priority is: 1 - 1 = 0.

[0068] In this embodiment, if the priority of the average quality label is 0.3 and the average priority is 0, then this piece of real estate information will be deleted.

[0069] In this embodiment, the local replacement type can be to replace the value corresponding to a certain real estate matter.

[0070] In this embodiment, the full replacement type refers to fully replacing the real estate information based on the obtained correct information.

[0071] The beneficial effects of the above technical solution are: By comparing the magnitudes of the priorities, information in different situations can be reasonably replaced effectively, and while ensuring the correctness of the replacement, the space occupied by the data generated during the replacement process can be reduced.

[0072] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalent technologies, the present invention also intends to include these changes and modifications.

Claims

1. A real estate data cleaning method based on the Internet, characterized in that: include: Step 1: Use web crawler technology to collect real estate data from multiple Internet data sources. At the same time, monitor the collection process based on monitoring tools to generate monitoring data, and set a transmission set for each real estate information under each data source, wherein the transmission set is related to data quality and transmission constraints; Step 2: Count the house type of each piece of real estate information in each Internet data source respectively to obtain the type distribution of the corresponding Internet data source; Step 3: Perform abnormal identification on each piece of real estate information under the corresponding house type. When the abnormal type is related to the transmission abnormality, analyze the update type of the corresponding transmission set and perform abnormal update on the corresponding real estate information; Step 4: When the exception type is not related to the transmission exception, the input behavior of the corresponding Internet data source is analyzed, and similar information is searched and identified using the identification field of the corresponding exception information to perform error updates; Step 5: Based on the distribution of all types and the authority of each Internet data source, all house types are arranged in order, and each updated information is formatted and stored in a classified manner.

2. The Internet-based real estate data cleaning method according to claim 1, characterized in that: Set up a transfer set for each piece of property information under each data source, including: The monitoring data under each data source is disassembled according to the transmission constraint, and input into the corresponding constraint analysis model to obtain the judgment result of the corresponding transmission constraint of each real estate information at each transmission process time point, and obtain a judgment set, wherein the judgment result is whether the corresponding transmission constraint is satisfied or not satisfied; All judgment sets are matrix-constructed to obtain a judgment matrix, and the first variance of each column vector and the second variance of each row vector are analyzed; Synchronously lock the first column whose first variance is greater than the first threshold and the first row whose second variance is greater than the second threshold to obtain cross elements, and count the first number of cross elements appearing in each row vector and the second number of cross elements appearing in each column vector; If the second number is consistent with the set number of the transmission constraint, setting a low-quality label to the corresponding piece of real estate information at the corresponding transmission process time point; If the second number is less than the set number of transmission constraints and greater than half of the set number, and the first number of row vectors corresponding to each transmission constraint under the second number is greater than N / 2, a low quality label is set for the corresponding piece of real estate information at the corresponding transmission process time point, where N represents the total number of transmission process time points; Otherwise, setting other quality labels for the corresponding piece of real estate information at the corresponding transmission process time point, wherein the other quality labels include: an intermediate quality label and a high quality label; Based on the label setting result at each transmission process time point and combined with the judgment result of the corresponding transmission constraint at each transmission process time point, a transmission set of the corresponding piece of real estate information is obtained.

3. The Internet-based real estate data cleaning method according to claim 2, characterized in that: Other quality tags are set for the corresponding piece of real estate information at the corresponding transmission process time point, including: If the second number is less than the set number of transmission constraints and greater than half of the set number, at this time, a third number of constraints not greater than N / 2 is counted for each transmission constraint under the second number corresponding to the first number of row vectors; like At this time, an intermediate quality label is set for the corresponding piece of real estate information at the corresponding transmission process time point, where is the third quantity; To set the quantity; is the second quantity; is the mass constant, with a value of 0.05; like , setting a high-quality label to the corresponding piece of real estate information at the corresponding transmission process time point; If the second number is less than or equal to half of the set number, then, combining each transmission constraint under the second number with a fourth number of elements in the row vector that do not satisfy the corresponding transmission constraint; like At this time, an intermediate quality label is set for the corresponding piece of real estate information at the corresponding transmission process time point, where represents a fourth quantity corresponding to the i1th transmission constraint under the second quantity; is the mass constant, with a value of 0.1; like , set a high-quality label to the corresponding piece of real estate information at the corresponding transmission process time point.

4. The Internet-based real estate data cleaning method according to claim 1, characterized in that: Perform anomaly identification on each piece of real estate information under the corresponding housing type, including: The transmission duration and transmission integrity of each piece of real estate information under the corresponding house type are detected. When the detection result meets the transmission anomaly constraint, it is determined that the anomaly type is related to the transmission anomaly. Otherwise, the determination is not related to the transmission anomaly.

5. The Internet-based real estate data cleaning method according to claim 1, characterized in that: Perform anomaly identification on each piece of real estate information under the corresponding housing type, including: From the transmission set under each data source, respectively lock the judgment information of each piece of real estate information related to the transmission anomaly under the same house type, wherein the judgment information includes: the information source of the real estate information under the transmission anomaly, and the transmission set of the real estate information under the transmission anomaly; Performing cluster analysis on all pieces of real estate information related to transmission anomalies under the same house type according to the transmission set, and obtaining a number of cluster groups; Analyze the distribution of quality labels involved in each cluster group, set an average quality label for each piece of real estate information, and at the same time, count the first total number of high-quality labels, the second total number of medium-quality labels, and the third total number of low-quality labels for the same piece of real estate information under all data sources to obtain auxiliary quality labels; Determine the update type of the corresponding piece of real estate information according to the average quality label and the auxiliary quality label, wherein the update type includes: a deletion type, a partial replacement type, and a full replacement type; The corresponding real estate information is abnormally updated according to the update type.

6. The Internet-based real estate data cleaning method according to claim 1, characterized in that: Analyze the input behavior of the corresponding Internet data source, use the identification field of the corresponding abnormal information to search and identify similar information, and perform error updates, including: Inputting the input behavior into a behavior analysis model to obtain possible factors causing information errors; Repairing the abnormal information based on the possible factors to obtain repair information; The first information is obtained by searching all correct information using the identification field corresponding to the abnormal information and identifying the identification field of each correct information; If the first information exists, replace the corresponding exception information with the corresponding exception type according to the first information; If the first information does not exist, at this time, the corresponding repair information is used to match the real estate matters in sequence from all the correct information, obtain the second information, and implement the error update of the corresponding abnormal information.

7. The Internet-based real estate data cleaning method according to claim 1, characterized in that: Based on the distribution of all types and the authority of each Internet data source, all housing types are ranked in order, including: Determine the authority of each Internet data source based on its widespread application and application praise coefficient; According to the authority of each Internet data source, the type distribution under the corresponding data source is adjusted to obtain an adjusted distribution; Superimpose all adjusted distributions to obtain the final distribution of all housing types; All house types are arranged in order according to the quantity value of each house type in the final distribution.

8. The Internet-based real estate data cleaning method according to claim 5, characterized in that: Determine the update type of the corresponding real estate information, including: When the priority of the average quality label is higher than the average priority of the quality labels of the corresponding piece of real estate information at the last moment under different Internet data sources, determining that the update type of the corresponding piece of real estate information is a deletion type; When the priority of the average quality label is not higher than the average priority of the quality label of the corresponding piece of real estate information at the last moment under different Internet data sources, determine whether the priority of the auxiliary quality label and the average quality label is higher than the average priority of the quality label of the corresponding piece of real estate information at the last moment under different Internet data sources; If it is higher, it is determined that the update type of the corresponding piece of real estate information is a partial replacement type; Otherwise, it is determined that the update type of the corresponding piece of real estate information is a complete replacement type.

Images