Real Estate Data Cleaning and Quality Assessment Methods Based on Internet Big Data

Abstract

This invention provides a method for cleaning and quality assessment of real estate data based on internet big data, belonging to the field of data processing technology. The method includes: collecting N pieces of real estate-related data from different data sources through parallel operation of multiple crawler nodes; performing multi-dimensional index analysis on each piece of real estate-related data to establish a multi-dimensional vector, and performing a first correction by analyzing the multi-dimensional features of the multi-dimensional matrix and the confidence level of the same data source; identifying abnormal real estate data, extracting output arrays from the normal dataset that match the abnormal indicators of the corresponding abnormal real estate data, and constructing individual correction models for the abnormal indicators; fusing all individual correction models under each piece of abnormal real estate data according to the mutual influence relationship of the abnormal indicators; and performing a second correction on the corresponding piece of abnormal real estate data when the quality assessment of the fused model is passed, to obtain the cleaned data. This method effectively solves the problems of messy, numerous outliers, and low reliability of internet real estate data, providing accurate data support for real estate analysis and decision-making.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a method for cleaning and quality assessment of real estate data based on Internet big data. Background Technology

[0002] With the development of internet big data technology, a large amount of real estate-related data has been accumulated, including basic property information, transaction data, and data on surrounding facilities. However, this data comes from a wide range of sources and is in various formats, containing a large amount of dirty data (such as erroneous, duplicate, and incomplete data), which seriously affects data quality and the accuracy of subsequent analysis. Currently, existing real estate data processing methods have many shortcomings in data cleaning and quality assessment. For example, traditional data cleaning methods often use simple rule matching, which cannot effectively handle complex data errors and inconsistencies; in terms of quality assessment, there is a lack of a comprehensive and scientific evaluation system, making it difficult to accurately determine the quality level of the data, thus failing to provide reliable data support for decision-making in the real estate industry.

[0003] Therefore, this invention proposes a method for cleaning and quality assessment of real estate data based on Internet big data. Summary of the Invention

[0004] This invention provides a method for cleaning and quality assessment of real estate data based on Internet big data, which solves the problems of dirty data and difficulty in accurately assessing data quality in existing technologies, improves the quality of real estate data, and provides reliable data support for decision-making in the real estate industry.

[0005] This invention provides a method for cleaning and quality assessment of real estate data based on Internet big data, comprising:

[0006] Step 1: Collect N real estate-related data from different data sources by having multiple crawler nodes work in parallel;

[0007] Step 2: Perform multi-dimensional indicator analysis on each piece of real estate-related data to establish a multi-dimensional vector, and establish a multi-dimensional matrix based on the same data source. By analyzing the multi-dimensional features of the multi-dimensional matrix and the confidence level of the same data source, the corresponding multi-dimensional vector is corrected once.

[0008] Step 3: Perform quality assessment on all correction vectors under the same type of data source to identify real estate anomaly data, and select normal datasets similar to each real estate anomaly data from N real estate related data. Extract the output array that matches the anomaly index of the corresponding real estate anomaly data from the normal dataset, and construct a separate correction model for the anomaly index.

[0009] Step 4: For each piece of abnormal real estate data, all individual correction models are merged according to the mutual influence relationship of abnormal indicators. When the quality verification of the merged model is passed, the corresponding piece of abnormal real estate data is corrected a second time to obtain cleaned data.

[0010] Preferably, N real estate-related data points are collected from different data sources based on the parallel operation of multiple crawler nodes, including:

[0011] The first collection task queue of each crawler node is determined, the real estate data source information to be collected is added to the first collection task queue, and the collection results corresponding to the tasks in the first collection task queue are written into the result storage segment corresponding to the first collection task queue in the data temporary storage area. Each crawler node is also configured with a task status synchronization tag with the same trigger time range as each of the other nodes. N real estate related data are collected, where each crawler node acts as a data sending and receiving side and is connected to the data temporary storage area, and N is a positive integer.

[0012] The determination of the trigger time range includes:

[0013] The temporary storage address of the latest released real estate data in the data temporary storage area is obtained as the first sequence number in the data temporary storage area, the temporary storage address of the latest pushed real estate data in the data temporary storage area is obtained as the second sequence number in the data temporary storage area, and all addresses belonging to the data input port and in an idle state in the address list configured for each crawler node based on the data temporary storage area are in the third sequence number set in the data temporary storage area.

[0014] The address release order of the data temporary storage area is determined according to the first sequence number, and the address temporary storage order of the data temporary storage area is determined according to the second sequence number. In combination with the working characteristics of the data temporary storage area, the release address block and the temporary storage address block are predicted.

[0015] Based on the release address block, temporary address block, and the third sequence number set of the corresponding crawler node, and combined with the standard communication feedback duration between the corresponding crawler node and the first real estate data source in the consistent first collection task queue, the trigger time range for each crawler node to start collecting according to the first collection task queue is set.

[0016] Preferably, a correction is made to the corresponding multidimensional vector, including:

[0017] Based on the dimension indicators set according to the data source type, the corresponding real estate-related data is decomposed to obtain indicator values. Combined with the set weight of each dimension indicator, a multi-dimensional vector Wi = {(Yj1,Qj1),j1 = 1,2,3,...,ni} is constructed, where ni represents the number of dimension indicators under the i-th data source type; Yj1 represents the indicator value of the corresponding real estate-related data under the j1-th dimension indicator; and Qj1 represents the set weight of the j1-th dimension indicator under the i-th data source type.

[0018] All multidimensional vectors of the same data source type are sorted in ascending order by collection timestamp to form an initial matrix, and redundant dimensions are removed to form a multidimensional matrix;

[0019] The core features of the multidimensional matrix are obtained, and the determinant value and eigenvalue dispersion of the multidimensional matrix are combined as auxiliary features to form a multidimensional feature set.

[0020] The confidence level of the same data source is determined by the duration of data source access, historical data correction rate, and cross-validation pass rate with authoritative third-party databases.

[0021] When principal components in a multidimensional feature set deviate from the preset confidence interval or the determinant value fluctuates abnormally, stratified correction is performed based on the confidence level.

[0022] Preferably, obtaining the core features of the multidimensional matrix includes:

[0023] Obtain the overall eigenvector of the multidimensional matrix;

[0024] The multidimensional matrix is ​​randomly divided into two columns, excluding the first and last columns, to obtain a first vertical submatrix and a second vertical submatrix. The first subvector and the second subvector are then obtained and concatenated to obtain the vertical feature vector.

[0025] The multidimensional matrix is ​​randomly divided into two sub-matrices except for the first and last rows to obtain the first and second horizontal sub-matrices. The third and fourth sub-vectors are then calculated and averaged to obtain the horizontal feature vectors.

[0026] The overall feature vector, vertical feature vector, and horizontal feature vector are sequentially input into the vector analysis model to obtain the overall feature representation sequence, vertical feature representation sequence, and horizontal feature representation sequence.

[0027] The overall feature representation sequence, vertical feature representation sequence, and horizontal feature representation sequence are sequentially subjected to deep mining to determine the enhancement variables of the same sub-representation sequence. The enhancement variables are the cumulative sum of the weight changes of the dimensionality indicators mapped to the sub-features of the corresponding sub-representation sequence.

[0028] The corresponding feature coefficients in the overall feature vector are enhanced according to the enhancement variables;

[0029] The overall feature representation sequence, vertical feature representation sequence, and horizontal feature representation sequence, along with their corresponding data source types, are input into the combinatorial analysis model to obtain the derived sequence of each sub-representation sequence. Based on all derived sequences and the proportion coefficient of each derived sequence based on the corresponding enhancement variable, new features are derived from the feature vector.

[0030] Extract the sub-representation sequence that matches the derived new features and regard it as the first sequence. At the same time, extract the second sequence from all sub-representation sequences of the overall feature vector, which has an importance greater than a preset importance.

[0031] The features matched by the first sequence and the second sequence are used as the core features of the multidimensional matrix.

[0032] Preferably, a quality assessment is performed on all single-correction vectors from the same type of data source to identify real estate outliers, including:

[0033] A three-dimensional quality assessment system is constructed, which includes completeness, logical consistency, and numerical rationality. Each correction vector is evaluated and a quality score is calculated. Real estate-related data corresponding to vectors with quality scores lower than a preset threshold are regarded as abnormal real estate data.

[0034] Preferably, an output array matching the abnormal indicators of the corresponding real estate anomaly data is extracted from the normal dataset, and a separate correction model for the abnormal indicators is constructed, including:

[0035] Calculate the correlation between abnormal indicators of real estate abnormal data and other indicators in normal data, extract the associated indicators to form an associated feature set, and sort the numerical sequences of each indicator in the associated feature set by timestamp to form an output array that matches the abnormal indicators. The array length is the number of similar data samples M days before and after the collection time of the corresponding abnormal related data.

[0036] A three-level network architecture consisting of a feature extraction layer, a mapping and transformation layer, and an output correction layer is constructed as the basic model framework.

[0037] Construct a loss function based on the output array;

[0038] A separate correction model is constructed based on the basic model framework and the three-dimensional loss function. The output of the separate correction model includes the correction value and the correction confidence interval.

[0039] Preferably, the loss function is constructed based on the product array, including:

[0040] The absolute error loss is determined based on the prediction results of the output correction layer and the actual results of the output array. At the same time, the failure operation of random neuron combination is performed on the output neuron sets of the feature extraction layer, the mapping transformation layer and the output correction layer to determine the three-layer failure loss.

[0041] A loss function is constructed based on the absolute error loss and the three-layer failure loss.

[0042] Preferably, the dimensions for quality assessment of the fusion model include: accuracy, robustness, and timeliness.

[0043] Compared with the prior art, the beneficial effects of this application are:

[0044] By improving data coverage through parallel acquisition across multiple nodes, and achieving initial correction by combining multidimensional matrix analysis with data source confidence, and then completing precise correction through anomaly locking, individual model construction and fusion optimization, high-quality real estate data is finally output. This effectively solves the problems of messy, outlier and low reliability of Internet real estate data, and provides accurate data support for real estate analysis and decision-making.

[0045] Other features and advantages of the invention will be set forth in the following description, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention may be realized and obtained by means of the structures particularly pointed out in the written description and the accompanying drawings.

[0046] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0047] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0048] Figure 1 This is a flowchart of a real estate data cleaning and quality assessment method based on Internet big data in an embodiment of the present invention. Detailed Implementation

[0049] The preferred embodiments of the present invention will be described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.

[0050] This invention provides a method for cleaning and quality assessment of real estate data based on Internet big data, such as... Figure 1 As shown, it includes:

[0051] Step 1: Collect N real estate-related data from different data sources by having multiple crawler nodes work in parallel;

[0052] Step 2: Perform multi-dimensional indicator analysis on each piece of real estate-related data to establish a multi-dimensional vector, and establish a multi-dimensional matrix based on the same data source. By analyzing the multi-dimensional features of the multi-dimensional matrix and the confidence level of the same data source, the corresponding multi-dimensional vector is corrected once.

[0053] Step 3: Perform quality assessment on all correction vectors under the same type of data source to identify real estate anomaly data, and select normal datasets similar to each real estate anomaly data from N real estate related data. Extract the output array that matches the anomaly index of the corresponding real estate anomaly data from the normal dataset, and construct a separate correction model for the anomaly index.

[0054] Step 4: For each piece of abnormal real estate data, all individual correction models are merged according to the mutual influence relationship of abnormal indicators. When the quality verification of the merged model is passed, the corresponding piece of abnormal real estate data is corrected a second time to obtain cleaned data.

[0055] In this embodiment, multiple crawler nodes refer to multiple data acquisition program instances deployed on different servers or virtual machines, which can execute data acquisition tasks independently and in parallel.

[0056] Different data sources are various channels or platforms that provide real estate-related data. Common ones include real estate transaction platforms such as Lianjia and Beike, official websites of real estate management departments, official websites of real estate developers, and real estate-related discussion sections on social media.

[0057] In this embodiment, real estate-related data refers to various types of information related to real estate, including basic property information, transaction information, property rights information, and surrounding amenities information. For example, when N=10000, 10000 of the above-mentioned real estate-related data items are collected.

[0058] In this embodiment, multidimensional indicator analysis decomposes and quantifies a single piece of real estate data from multiple dimensions (such as price, area, location, etc.) to comprehensively describe the data characteristics. For example, for a piece of second-hand housing data, it is decomposed from three dimensions: basic attributes (area, unit type, floor), transaction attributes (listing price, unit price, listing duration), and location attributes (school district, distance from subway) to obtain 10 specific indicators.

[0059] In this embodiment, the multidimensional vector is a vector formed by arranging the multidimensional index values ​​of a single data point in a preset order and combining the weights of each index, and is used to quantify data features.

[0060] A multidimensional matrix from the same data source is a matrix formed by arranging the multidimensional vectors of all data collected from the same data source in ascending order of collection timestamps. Rows represent data samples, and columns represent indicator dimensions. The multidimensional features of a multidimensional matrix are a set of features reflecting the overall pattern of the matrix, including core features (such as principal components and eigenvectors) and auxiliary features (such as determinant values ​​and eigenvalue dispersion). The confidence level of a data source is an indicator of its reliability, calculated based on the data source's historical performance and authority. Data sources of the same type are sets of data sources with consistent data types or source attributes, such as both being second-hand housing transaction platforms or both being government land data platforms.

[0061] In this embodiment, the first correction is based on the multidimensional matrix features and the confidence level of the data source, and makes preliminary adjustments to the indicators in the multidimensional vector that deviate from the normal pattern, such as correcting obviously abnormal prices and filling in missing areas.

[0062] Quality assessment scores a correction vector based on three dimensions: data completeness, logical consistency, and numerical reasonableness, to determine data quality. Real estate abnormal data are real estate data whose quality assessment score is lower than a preset threshold of 60 points, indicating obvious errors or deviations from normal patterns. Similar normal datasets are sets of normal data selected from all data that are similar to abnormal data in core attributes and have passed the quality assessment. Abnormal indicators are specific indicators in abnormal data that lead to a low quality score, such as missing unit types or unreasonable unit prices.

[0063] The output array is a sequence of indicator values ​​extracted from the normal dataset and associated with the abnormal indicators, sorted by time. It is used to train the correction model. The individual correction model is a machine learning model built for a single abnormal indicator to predict the reasonable value of the indicator and the corrected confidence interval. The mutual influence relationship of abnormal indicators is the correlation between different abnormal indicators in the same abnormal data. For example, an abnormal area may affect the reasonableness of the unit price, and missing unit type may lead to deviation in area calculation.

[0064] Model fusion combines multiple separate correction models (corresponding to different anomaly indicators) for the same outlier data into a single fusion model based on the influence relationships between the indicators, thereby improving the reliability of the correction. For example:

[0065] Related indicator group: {Building area, Listing price per unit} (correlation strength ≥ 0.7 points): Core feature: There is a direct constraint relationship between the two at the mathematical formula level. When making corrections, they must be adjusted synchronously to ensure logical consistency. For example, after the area is corrected from 200 square meters to 110 square meters, the price per unit needs to be corrected from 30,000 yuan / square meter to 55,000 yuan / square meter in order for the total price of 6.05 million yuan to conform to the regional market.

[0066] The related indicator group: {age of the building, listing price per square meter, building area} (correlation strength 0.4-0.6 points): core characteristics: there is an indirect impact on business logic. When making corrections, the interaction needs to be considered. For example, if the building age is corrected from 5 years to 23 years, the price per square meter should be appropriately reduced to 52,000 yuan / square meter, rather than maintained at 55,000 yuan / square meter, because the depreciation of the old building will lower the price.

[0067] Weakly correlated indicator group: {Greening rate of the community, developer name} (correlation strength ≤ 0.3 points): Core characteristics: The impact on core indicators (unit price, area) is weak. They can be handled independently when making corrections. For example, if the greening rate is corrected from 80% to 35%, or the developer name is changed from XX Real Estate to YY Real Estate, only individual adjustments are needed, without the need to modify the unit price or area in conjunction with the change.

[0068] Finally, combining the three sets of results gives the output of the fusion model.

[0069] The quality assessment of the fusion model evaluates its suitability from three dimensions: accuracy (deviation between the corrected value and the true value), robustness (stability to noisy data), and timeliness (correction speed). If any one of these dimensions is less than the corresponding set value, the fusion model is deemed unqualified and requires further correction. Only when the results in each dimension are greater than or equal to the corresponding set value is the model considered qualified.

[0070] Secondary correction is based on a quality-verified fusion model, which makes final adjustments to all abnormal indicators of the anomalous data. The correction result is closer to the actual situation. The cleaned data is real estate data that has met the quality standards (completeness, consistency, and rationality) after secondary correction and can be directly used for subsequent analysis. For example, if the area of ​​a property is incorrectly recorded as 500 square meters (actually 100 square meters), resulting in an inflated total price, secondary correction first changes the area to 100 square meters, and then recalculates the total price according to the formula Total Price = Area × Unit Price, avoiding a series of errors from one step.

[0071] The beneficial effects of the above technical solution are: improving data coverage through multi-node parallel acquisition, achieving preliminary correction by combining multi-dimensional matrix analysis and data source confidence, and then completing accurate correction through anomaly locking, individual model construction and fusion optimization, ultimately outputting high-quality real estate data, effectively solving the problems of messy, outlier and low reliability of Internet real estate data, and providing accurate data support for real estate analysis and decision-making.

[0072] This invention provides a method for cleaning and quality assessment of real estate data based on internet big data. It involves collecting N pieces of real estate-related data from different data sources through multiple crawler nodes working in parallel, including:

[0073] The first collection task queue of each crawler node is determined, the real estate data source information to be collected is added to the first collection task queue, and the collection results corresponding to the tasks in the first collection task queue are written into the result storage segment corresponding to the first collection task queue in the data temporary storage area. Each crawler node is also configured with a task status synchronization tag with the same trigger time range as each of the other nodes. N real estate related data are collected, where each crawler node acts as a data sending and receiving side and is connected to the data temporary storage area, and N is a positive integer.

[0074] The determination of the trigger time range includes:

[0075] The temporary storage address of the latest released real estate data in the data temporary storage area is obtained as the first sequence number in the data temporary storage area, the temporary storage address of the latest pushed real estate data in the data temporary storage area is obtained as the second sequence number in the data temporary storage area, and all addresses belonging to the data input port and in an idle state in the address list configured for each crawler node based on the data temporary storage area are in the third sequence number set in the data temporary storage area.

[0076] The address release order of the data temporary storage area is determined according to the first sequence number, and the address temporary storage order of the data temporary storage area is determined according to the second sequence number. In combination with the working characteristics of the data temporary storage area, the release address block and the temporary storage address block are predicted.

[0077] Based on the release address block, temporary address block, and the third sequence number set of the corresponding crawler node, and combined with the standard communication feedback duration between the corresponding crawler node and the first real estate data source in the consistent first collection task queue, the trigger time range for each crawler node to start collecting according to the first collection task queue is set.

[0078] In this embodiment, the first collection task queue is an ordered list of collection tasks assigned to each crawler node, which includes information such as the address of the data source to be collected and the content to be collected (such as new house prices, land transfer information). For example, the first collection task queue is: {Lianjia Beijing Chaoyang District new house list page, Lianjia Beijing Haidian District new house details page, Anjuke Beijing second-hand house transaction record page}.

[0079] In this embodiment, the real estate data source information to be collected includes the platform address, data type, and access rules (such as whether login is required and request frequency limits) of the real estate-related data to be collected. In this embodiment, the data buffer is an intermediate storage space for temporarily storing the crawler's collected results, used to avoid conflicts when data is directly written to the database. The result storage segment is a separate storage area within the data buffer for each crawler node's first collection task queue, used to store the collection results of that queue separately, avoiding data confusion between different nodes. For example, the data buffer is divided into three storage segments according to nodes: storage segment A corresponds to Node A's queue, storage segment B corresponds to Node B's queue, and storage segment C corresponds to Node C's queue.

[0080] In this embodiment, the task status synchronization tag is the identification information used by each crawler node to synchronize the task progress. It includes the task completion status, current collection location, temporary storage area data address, etc., and all nodes have the same tag format and update frequency. For example, the tag format is Node ID: number of tasks completed / total number of tasks; latest temporary storage address: XXX; trigger time range: [T1, T2]; the tag of NodeA is NodeA: 2 / 5; latest temporary storage address: 500; trigger time range: [10:00:00, 10:00:05].

[0081] In this embodiment, the data transceiver side is the role of the crawler node as both the data receiver (receiving tasks from the task queue) and the sender (sending the collected results to the data temporary storage area).

[0082] In this embodiment, the trigger time range is the time interval during which the crawler node starts executing the first task in the first collection task queue, such as 10:00:00 to 10:00:03, in order to avoid conflicts caused by multiple nodes accessing the same data source or writing to the temporary storage area at the same time.

[0083] In this embodiment, the first sequence number is the temporary storage address of the latest released data in the data temporary storage area (the temporary storage area addresses are numbered sequentially, and release means that the address becomes free after the data is transferred from the temporary storage area to the formal database). For example, the data temporary storage area addresses are numbered 1, 2, 3, ..., 100000, and the latest data transferred from the temporary storage area to the formal database is stored at address 500, so the first sequence number is 500.

[0084] In this embodiment, the second sequence number is the number of the temporary storage address of the latest pushed real estate data in the data temporary storage area (pushing refers to the crawler node writing the collected results into the temporary storage area).

[0085] In this embodiment, the third sequence number set is a set of address numbers configured for each crawler node that belong to the data input port in the data temporary storage area and are in an idle state (the data input port refers to the address where the node can write data, and idle means that it is not occupied). For example, the data input port address range configured for NodeA is 1-2000, of which the currently idle addresses are 501, 502, 601, 602. Therefore, the third sequence number set of NodeA is {501, 502, 601, 602}.

[0086] In this embodiment, the address release order is the order in which addresses in the data temporary storage area are from occupied to free (i.e., the order in which data is transferred from the temporary storage area to the formal database). For example, if the first sequence number is 500, and the previously released addresses are 499, 498, and 497, it means that the address release order is from the largest to the smallest number (500→499→498→487).

[0087] In this embodiment, the address temporary storage order is the order in which addresses in the data temporary storage area are written to data (i.e., the address order in which crawler nodes push data). For example, if the second sequence number is 600, and the addresses written previously are 599, 598, and 597, it means that the address temporary storage order is in ascending order of the numbers (597→598→599→600).

[0088] In this embodiment, the working characteristics of the data buffer area are the read / write rules, address allocation mechanism, capacity limit, etc., such as the first address written is released first, and a maximum of 100 consecutive addresses can be written at a time.

[0089] In this embodiment, the release address block is a continuous address range that is about to be released from the temporary storage area (becomes free) based on the address release order and the working characteristics of the data temporary storage area. For example, if the address release order is known to be 500→499, and the temporary storage area forms a block every 10 consecutive addresses released, then the next release address block is predicted to be 490-499 (a total of 10 addresses).

[0090] In this embodiment, the temporary address block is a continuous address range that is about to be written to, which is predicted based on the address temporary storage order and the working characteristics of the data temporary storage area. For example, if the address temporary storage order is known to be 600→601, and the temporary storage area forms a block every 20 consecutive addresses written, then the next temporary address block is predicted to be 601-620 (a total of 20 addresses).

[0091] In this embodiment, the standard communication feedback time is the average time from sending a request to receiving a response when the crawler node communicates with the first real estate data source in the first data collection task queue (to avoid congestion caused by nodes accessing the data source at the same time).

[0092] In this embodiment, setting the trigger time range is the core technical means to solve the dual conflict when distributed crawlers collect data in parallel. The trigger time range is set to [Ts, Te].

[0093]

[0094] Do=max(0,min(Rend,Wend)-max(Rstart,Wstart)+1)

[0095] Where Tc is the current system time (in seconds); S1 is the first sequence number; S2 is the second sequence number; |S3| is the number of free addresses based on the third sequence number set S3; Rstart and Rend are the start and end sequence numbers of the predicted release address block; Rs = Rend - Rstart + 1 is the size of the release block; Wstart and Wend are the start and end sequence numbers of the predicted temporary storage address block; Ws = Wend - Wstart + 1 is the size of the temporary storage block; Tstd is the standard communication feedback time between the crawler node and the first data source (in seconds); Atotal is the total number of addresses in the data temporary storage area; Nnode is the total number of crawler nodes participating in parallel collection; k1, k2, k3, and k4 are weight coefficients, ranging from 0.8 to 1.2; ∈ is the minimum value, with a value of 10. -6 To avoid a denominator of 0; Do represents the degree of overlap; Address conflict risk factor; This is the free address sufficiency factor; The data source access pressure factor; Wspan represents the address sequence deviation; Wspan represents the address range span related to free address statistics.

[0096] By constructing a time-series model that dynamically adapts to the interaction characteristics of distributed crawlers and data storage areas, the technical shortcomings of traditional fixed-delay triggering methods in dealing with dynamic address changes and data source load fluctuations are solved, significantly improving the stability and efficiency of parallel data collection.

[0097] The beneficial effects of the above technical solution are as follows: by allocating independent task queues and temporary storage segments to crawler nodes, analyzing and predicting the release and temporary address blocks based on address sequence numbers, and setting the trigger time range based on communication feedback duration, the orderly parallel collection of multiple nodes is achieved, effectively avoiding data source access conflicts and temporary address occupation conflicts, and improving data collection efficiency and temporary space utilization.

[0098] This invention provides a method for cleaning and quality assessment of real estate data based on Internet big data, which involves a correction of the corresponding multi-dimensional vector, including:

[0099] Based on the dimension indicators set according to the data source type, the corresponding real estate-related data is decomposed to obtain indicator values. Combined with the set weight of each dimension indicator, a multi-dimensional vector Wi = {(Yj1,Qj1),j1 = 1,2,3,...,ni} is constructed, where ni represents the number of dimension indicators under the i-th data source type; Yj1 represents the indicator value of the corresponding real estate-related data under the j1-th dimension indicator; and Qj1 represents the set weight of the j1-th dimension indicator under the i-th data source type.

[0100] All multidimensional vectors of the same data source type are sorted in ascending order by collection timestamp to form an initial matrix, and redundant dimensions are removed to form a multidimensional matrix;

[0101] The core features of the multidimensional matrix are obtained, and the determinant value and eigenvalue dispersion of the multidimensional matrix are combined as auxiliary features to form a multidimensional feature set.

[0102] The confidence level of the same data source is determined by the duration of data source access, historical data correction rate, and cross-validation pass rate with authoritative third-party databases.

[0103] When principal components in a multidimensional feature set deviate from the preset confidence interval or the determinant value fluctuates abnormally, stratified correction is performed based on the confidence level.

[0104] In this embodiment, the indicator value is the data value corresponding to the dimension indicator. The property area is 100 square meters, the listing price is 5 million yuan, and the property age is 10 years. It is extracted by parsing the data source (web crawling, API call). For example, the property area value can be parsed from the Lianjia page using a Python crawler.

[0105] Weights are coefficients that reflect the importance of dimensional indicators within the data source type. For example, the listing price on the transaction platform influences user decisions, with a weight of 0.4; the age of the property has a weight of 0.2. The weights are determined based on business needs, expert experience, or statistical methods. The transaction platform calculates these weights using regression analysis based on user browsing behavior data.

[0106] The data collection timestamp is an identifier that records the time when real estate data was collected, accurate to the second.

[0107] The initial matrix is ​​a matrix composed of multidimensional vectors arranged in ascending order of collection timestamps, under the same data source type.

[0108] Redundant dimensions are indicators that are not valuable for analysis and are highly correlated with other dimensions. The building area of ​​the transaction platform is highly correlated with the internal area plus the shared area. The internal area plus the shared area is a redundant dimension, so it can be removed.

[0109] A multidimensional matrix is ​​the initial matrix after removing redundant dimensions.

[0110] The core feature is the key feature that reflects the distribution and correlation patterns of multidimensional matrix data.

[0111] The determinant is a scalar value for a square matrix (when the number of rows equals the number of columns in a multidimensional matrix), reflecting the linear dependence of the matrix and the degree of data dispersion. A non-invertible matrix (determinant 0) indicates linear dependence in dimensions; a large determinant value indicates data dispersion.

[0112] Eigenvalue dispersion refers to the degree of dispersion of eigenvalues, measured by standard deviation and coefficient of variation. A large coefficient of variation indicates large differences in eigenvalues ​​and an unbalanced data distribution. In trading platform data, a large standard deviation of PCA eigenvalues ​​indicates significant differences in the explanatory power of the principal components.

[0113] Auxiliary features include determinant value and eigenvalue dispersion, which assist the core features in describing the matrix. Determinant value reflects dimensional correlation, and eigenvalue dispersion reflects principal component differences. Together with the core features, they comprehensively depict the data patterns.

[0114] A multidimensional feature set is a collection of core features and auxiliary features. The feature set of trading platform data includes price-area principal components (core), matrix determinant values ​​(auxiliary), and eigenvalue variation coefficients (auxiliary).

[0115] In this embodiment, the data source access duration is the length of time the data source has been connected to the system, which is the number of days / months / years from the access date to the present, and the access duration is 2 years (24 months).

[0116] Historical data correction rate is the percentage of historical data that needs correction. For example, if a trading platform has 1,000 historical data entries and 100 entries need correction, the correction rate is 10%.

[0117] Cross-validation pass rate is the percentage of data from the data source that matches the data in a third-party authoritative database, such as an industry-recognized database. In a transaction platform with 500 data entries, 450 matches the industry-recognized database, resulting in a 90% pass rate. This is achieved through API integration and data matching, such as field association (unique property code) comparison.

[0118] The confidence level of the data source is obtained by combining the access duration, correction rate, and pass rate. The access duration of the trading platform is 2 years, with a maximum duration of 5 years. The correction rate is 10%, and the pass rate is 90%. At this time, the confidence level = 0.3 × (2 / 5) + 0.3 × 0.9 + 0.4 × 0.9 = 0.81, and 0.3, 0.3, and 0.4 correspond to the weights of access duration, correction rate, and pass rate, respectively, and the sum of the weights is 1.

[0119] Principal component deviation from the pre-set confidence interval means that the principal component values ​​extracted by PCA exceed the reasonable range of pre-statistics, such as the 95% confidence interval.

[0120] Abnormal fluctuations in the determinant value occur when the determinant value changes over time with a fluctuation range exceeding the normal range, such as a difference from the mean greater than twice the standard deviation. For example, the mean of the determinant value in a trading platform matrix is ​​5, but it drops to 15 at a certain point in time, indicating abnormal fluctuation. The mean and standard deviation can be calculated using Python NumPy to monitor the fluctuations.

[0121] Layered correction employs different strategies based on the confidence level of the data source. For high-confidence data sources (e.g., >0.8), statistical regularity correction is used directly (principal components are used to pull the data back into the confidence interval); for low-confidence data sources (e.g., <0.6), correction is performed using cross-validation across multiple data sources. For example, if the trading platform has a confidence level of 0.81 (high), and the principal components deviate, historical principal component mean correction is used; for low-confidence data sources, third-party data is used for correction.

[0122] The beneficial effects of the above technical solution are as follows: by accurately decomposing the data source to construct multi-dimensional vectors and matrices, it deeply mines core and auxiliary features to characterize data patterns, scientifically quantifies the confidence level of the data source, and achieves tiered correction of abnormal data. This approach not only ensures the rationality of data dimensions by aligning with the characteristics of different data sources, but also efficiently identifies and corrects data anomalies through feature analysis and confidence level tiering, laying a solid foundation for real estate data cleaning.

[0123] This invention provides a method for cleaning and quality assessment of real estate data based on Internet big data, which obtains the core features of the multidimensional matrix, including:

[0124] Obtain the overall eigenvector of the multidimensional matrix;

[0125] The multidimensional matrix is ​​randomly divided into two columns, excluding the first and last columns, to obtain a first vertical submatrix and a second vertical submatrix. The first subvector and the second subvector are then obtained and concatenated to obtain the vertical feature vector.

[0126] The multidimensional matrix is ​​randomly divided into two sub-matrices except for the first and last rows to obtain the first and second horizontal sub-matrices. The third and fourth sub-vectors are then calculated and averaged to obtain the horizontal feature vectors.

[0127] The overall feature vector, vertical feature vector, and horizontal feature vector are sequentially input into the vector analysis model to obtain the overall feature representation sequence, vertical feature representation sequence, and horizontal feature representation sequence.

[0128] The overall feature representation sequence, vertical feature representation sequence, and horizontal feature representation sequence are sequentially subjected to deep mining to determine the enhancement variables of the same sub-representation sequence. The enhancement variables are the cumulative sum of the weight changes of the dimensionality indicators mapped to the sub-features of the corresponding sub-representation sequence.

[0129] The corresponding feature coefficients in the overall feature vector are enhanced according to the enhancement variables;

[0130] The overall feature representation sequence, vertical feature representation sequence, and horizontal feature representation sequence, along with their corresponding data source types, are input into the combinatorial analysis model to obtain the derived sequence of each sub-representation sequence. Based on all derived sequences and the proportion coefficient of each derived sequence based on the corresponding enhancement variable, new features are derived from the feature vector.

[0131] Extract the sub-representation sequence that matches the derived new features and regard it as the first sequence. At the same time, extract the second sequence from all sub-representation sequences of the overall feature vector, which has an importance greater than a preset importance.

[0132] The features matched by the first sequence and the second sequence are used as the core features of the multidimensional matrix.

[0133] In this embodiment, the overall feature vector is the core vector that reflects the overall pattern of the matrix extracted from the multidimensional matrix by a matrix decomposition algorithm (such as principal component analysis PCA). For example, the overall feature vector V_overall = {0.6, 0.3, 0.1} is obtained by using principal component analysis PCA to extract the multidimensional matrix M0. This means that the area (weight 0.6) is the most important feature affecting the overall distribution of the data, followed by the unit price (0.3).

[0134] In this embodiment, the random column cutting, excluding the first and last columns, involves excluding the first and last dimensions from the columns (dimension indicators) of the multidimensional matrix, randomly selecting a column as the cutting point, and dividing the matrix vertically into two parts (vertical sub-matrices). The first vertical sub-matrice and the second vertical sub-matrice are the two sub-matrices obtained after column cutting, which respectively contain the columns to the left and right of the cutting point.

[0135] In this embodiment, the first sub-vector and the second sub-vector are feature vectors extracted from the two vertical sub-matrices respectively, reflecting the core rules of each sub-matrix. For example, the sub-vector V1 = {0.9} (single-dimensional feature of area) is extracted from the first vertical sub-matrix of M0, and the sub-vector V2 = {0.7, 0.3} (unit price weight 0.7, listing duration 0.3) is extracted from the second vertical sub-matrix of M0.

[0136] The vertical eigenvector is formed by concatenating the first sub-vector and the second sub-vector in the original column order. It reflects the local features of the matrix in the vertical direction (dimension index), namely: {0.9, 0.7, 0.3}.

[0137] In this embodiment, the random row cutting, excluding the first and last rows, involves excluding the first and last samples from the rows of the multidimensional matrix and randomly selecting a row as the cutting point to divide the matrix horizontally into two parts (horizontal submatrices). The first and second horizontal submatrices are the two submatrices obtained after row cutting, containing the rows above and below the cutting point, respectively. The third and fourth subvectors are feature vectors extracted from the two horizontal submatrices, reflecting the core patterns of each submatric, such as the sample features at different time periods. For example, for the first horizontal submatric of M0, V3 = {0.5, 0.4, 0.1} (area and unit price have a significant impact in early samples); for the second horizontal submatric of M0, V4 = {0.4, 0.5, 0.1} (unit price has a more significant impact in later samples). In this case, the average of V3 and V4 is superimposed to obtain {0.45, 0.45, 0.1}.

[0138] Vector analysis models are used to transform feature vectors into a more refined sequence form in machine learning models. LSTM0 (Long Short-Term Memory Network) is used as the vector analysis model, and the output is a sequence containing time-step features after inputting a vector.

[0139] The overall feature representation sequence, vertical feature representation sequence, and horizontal feature representation sequence are sequence data obtained after vectors are processed by the analysis model. Features are refined in a multi-step manner (such as sub-features split by time or dimension). For example, after the overall feature vector {0.6,0.3,0.1} is input into LSTM0, the overall feature representation sequence {(0.6,0.3),(0.5,0.4),...,(0.6,0.3)} is obtained (10 time steps, each step contains 2 sub-features); similarly, the vertical refinement sequence and the horizontal refinement sequence are obtained.

[0140] Deep mining identifies key sub-features (parts that significantly affect the overall pattern) from the feature representation sequence through attention mechanisms or statistical analysis (such as analysis of variance). For example, by applying attention mechanisms to the overall feature representation sequence, we can find that the sub-features in time steps 3-5 are strongly correlated with abnormal fluctuations in unit price and mark them as key sub-features.

[0141] A sub-representation sequence is a local sequence segment in the feature representation sequence that corresponds to a key sub-feature. For example, the sequence segment at time steps 3-5 in the overall feature representation sequence: {(0.5,0.4),(0.55,0.35),(0.5,0.45)} is a sub-representation sequence.

[0142] The augmentation variable is the sum of the weight changes of the dimensional indicators (such as area and unit price) of the sub-representation sequence mapping. It is used to quantify the importance of the sub-features. For example, if the sub-representation sequence mapping area (weight change +0.1) and unit price (weight change +0.2) are augmentation variables, then the augmentation variable is: 0.1 + 0.2 = 0.3.

[0143] The corresponding feature coefficients in the overall feature vector are the dimension coefficients (such as the coefficients corresponding to area and unit price) associated with the augmenting variable. Augmentation is performed by amplifying the corresponding feature coefficients using the augmenting variable to highlight the influence of key features. For example, if the augmenting variable is 0.3, the augmented coefficients will be 0.6 × (1 + 0.3) = 0.78 (area) and 0.3 × (1 + 0.3) = 0.39 (unit price), resulting in an augmented overall feature vector of {0.78, 0.39, 0.1}.

[0144] Combinatorial analysis models are models that fuse feature representation sequences with data source types (such as attention fusion networks) to generate derived features that combine data source characteristics. They are pre-trained, with training samples consisting of sequence and type combinations and derived features for those combinations.

[0145] A derived sequence is a new sequence output by the combinatorial analysis model that includes data source characteristics (adding data source-related features to the original features). For example, after model processing, the derived sequence {0.9, 0.7, 0.2} is obtained, which represents a new sub-feature with a platform credibility of 0.2.

[0146] The proportion coefficient is the contribution ratio of each derived sequence to the new feature (calculated based on the enhancement variables). If the enhancement variable of a certain derived sequence is 0.3 and the total enhancement variables are 0.9, then the proportion coefficient is 0.3 / 0.9.

[0147] The new feature derivation is a new feature generated based on the derived sequence and the proportion coefficient, which integrates the original feature and the characteristics of the data source. For example, the proportion of 0.33 and other derived sequences generate a new feature: the rationality index of unit price of commercial platform.

[0148] The first sequence is the sub-representation sequence matching the derived new features (i.e., the original sequence segment corresponding to the new feature). The preset importance is a manually set feature importance threshold (e.g., 0.7 based on business needs) used to filter key features. The second sequence is the sequence segment in the sub-representation sequence of the overall feature vector whose importance exceeds the preset threshold. The core feature is the feature shared by the first and second sequences, combining the newly derived features and highly important features, and can most accurately reflect the core rules of the multidimensional matrix.

[0149] The beneficial effects of the above technical solution are: by extracting local features through horizontal and vertical cutting of multidimensional matrices, refining feature sequences by combining vector analysis models, highlighting key features by enhancing variables, and deriving new features by integrating data source types, the core features that take into account both the original patterns and derived value are finally selected, which greatly improves the accuracy of capturing the overall patterns of real estate data and effectively reduces the correction errors caused by incomplete feature extraction.

[0150] This invention provides a method for cleaning and quality assessment of real estate data based on internet big data. It involves performing quality assessments on all primary correction vectors from the same type of data source to identify abnormal real estate data, including:

[0151] A three-dimensional quality assessment system is constructed, which includes completeness, logical consistency, and numerical rationality. Each correction vector is evaluated and a quality score is calculated. Real estate-related data corresponding to vectors with quality scores lower than a preset threshold are regarded as abnormal real estate data.

[0152] Preferably, an output array matching the abnormal indicators of the corresponding real estate anomaly data is extracted from the normal dataset, and a separate correction model for the abnormal indicators is constructed, including:

[0153] Calculate the correlation between abnormal indicators of real estate abnormal data and other indicators in normal data, extract the associated indicators to form an associated feature set, and sort the numerical sequences of each indicator in the associated feature set by timestamp to form an output array that matches the abnormal indicators. The array length is the number of similar data samples M days before and after the collection time of the corresponding abnormal related data.

[0154] A three-level network architecture consisting of a feature extraction layer, a mapping and transformation layer, and an output correction layer is constructed as the basic model framework.

[0155] Construct a loss function based on the output array;

[0156] A separate correction model is constructed based on the basic model framework and the three-dimensional loss function. The output of the separate correction model includes the correction value and the correction confidence interval.

[0157] Preferably, the loss function is constructed based on the product array, including:

[0158] The absolute error loss is determined based on the prediction results of the output correction layer and the actual results of the output array. At the same time, the failure operation of random neuron combination is performed on the output neuron sets of the feature extraction layer, the mapping transformation layer and the output correction layer to determine the three-layer failure loss.

[0159] A loss function is constructed based on the absolute error loss and the three-layer failure loss.

[0160] In this embodiment, completeness measures whether the data includes key indicators (such as core attributes of the property and transaction information) and is free of omissions or deficiencies. Logical consistency measures whether the logical relationships between different indicators in the data are valid (such as mathematical relationships or business rules). Numerical reasonableness measures whether the values ​​of the indicators in the data are within a reasonable range (based on the distribution of similar data or common business sense).

[0161] In this embodiment, the quality score is based on the scoring of a correction vector by a three-dimensional evaluation system (e.g., a maximum score of 100 points). The higher the score, the better the data quality. For example, if a certain data has no missing indicators (completeness 80 points), meets all logical rules (logical consistency 100 points), and the unit price is within a reasonable range (numerical reasonableness 90 points), the quality score can be obtained by weighting and calculating according to the weights (completeness 30%, logic 30%, reasonableness 40%).

[0162] In this embodiment, the preset threshold is a critical score for judging whether the data is abnormal (set according to business needs). Data below this value is considered abnormal. Based on the quality of historical data, the preset threshold is set to 60 points, that is, real estate data corresponding to vectors with a quality score < 60 points are judged as abnormal.

[0163] In this embodiment, the abnormal indicators are the specific indicators that cause real estate data to be judged as abnormal (i.e., indicators with low scores in the three-dimensional assessment). For example, the abnormal indicators of the above-mentioned abnormal data are area (missing) and unit price (too high).

[0164] In this embodiment, the correlation degree is the degree of correlation between the abnormal indicator and other indicators (such as area, floor, and building age) in the normal dataset. It is used to determine which indicators can help correct the abnormal indicator and is represented by the Pearson correlation coefficient. The correlation feature set is a combination of indicators (features that help correct the abnormal indicator) extracted from the normal dataset that have a high correlation degree with the abnormal indicator. For example, for the abnormal indicator unit price, the indicators with a correlation degree ≥ 0.5 are area, distance from the subway, and building age. Then the correlation feature set = {area, distance from the subway, building age}.

[0165] In this embodiment, the output array is a sequence formed by arranging the values ​​of each indicator in the associated feature set in ascending order according to the collection timestamp. The length is the number of similar data samples M days before and after the abnormal data collection time. It is used to train and correct the model. For example, M=30. The unit price values ​​corresponding to area, distance from subway, and building age are extracted from the normal dataset and sorted by time to obtain the output array {5.2,5.3,5.1,...,5.4} (a total of 200 values, corresponding to samples within 30 days).

[0166] In this embodiment, the feature extraction layer is the first layer of the network, used to extract key features (such as time trends and indicator correlations) from the associated feature set (such as the output array). For example, an LSTM (Long Short-Term Memory) network is used as the feature extraction layer, taking into input a 30-day sequence of area, distance from the subway, and building age, and outputting a feature vector containing time dependencies (such as the trend feature that the unit price decreases by 0.3 million yuan per square meter for every 10 square meters increase in area). The mapping and transformation layer is the second layer of the network, converting the high-dimensional features output by the feature extraction layer into correction parameters related to abnormal indicators (such as the unit price adjustment coefficient). A two-layer fully connected neural network is used as the mapping and transformation layer, taking into input the 64-dimensional features extracted by the LSTM, and converting them into 10-dimensional positive parameters (such as regional coefficients, time coefficients, area coefficients, etc.) through an activation function (such as ReLU). The output correction layer is the third layer of the network. Based on the correction parameters of the mapping and transformation layer, it outputs reasonable correction values ​​and correction confidence intervals for abnormal indicators (reflecting the reliability of the correction results). For example, if the input is a positive parameter of 10 maintenance, the output unit price correction value and confidence interval = {52,54} ten thousand / square meter (meaning there is a 95% probability that it is within this interval).

[0167] In this embodiment, the absolute error loss measures the deviation between the prediction result of the output correction layer and the actual result of the output array, and is expressed as the mean of the absolute differences between the two, calculated using the MAE (mean absolute error) formula. The failure operation of random neuron combinations simulates the random failure scenario of some neurons in the neural network, such as neurons in the feature extraction layer and the mapping transformation layer, to enhance the robustness of the model. For example, 10% (7) of the 64 neurons in the feature extraction layer are randomly selected and their failure outputs are set to 0, and the change in the model's prediction error is observed; similarly, this operation is performed on the mapping transformation layer and the output correction layer.

[0168] In this embodiment, the three-layer failure loss is the deviation between the model's prediction result and the actual result of the output array after the neuron fails (calculated separately for the feature extraction layer, mapping transformation layer, and output correction layer). For example, if the prediction error increases from 0.2 to 0.4 after the neuron in the feature extraction layer fails, then the failure loss of the feature extraction layer is 0.4; if the error increases to 0.3 after the mapping transformation layer fails, then its failure loss is 0.3; if the error increases to 0.5 after the output correction layer fails, then its failure loss is 0.5.

[0169] In this embodiment, the three-dimensional loss function is a comprehensive loss function that integrates the absolute error loss, the feature extraction layer failure loss, the mapping transformation layer failure loss, and the output correction layer failure loss, and is used to guide model training. For example, the three-dimensional loss function = 0.5 × absolute error loss + 0.2 × feature extraction layer failure loss + 0.2 × mapping transformation layer failure loss + 0.1 × output correction layer failure loss (the weights are set according to their importance).

[0170] In this embodiment, the individual correction model is a model trained based on a three-level network architecture and a three-dimensional loss function, which can output correction values ​​and confidence intervals for a single abnormal indicator.

[0171] The beneficial effects of the above technical solution are: by accurately identifying abnormal real estate data through a three-dimensional quality assessment system, constructing a time-series output array by combining the correlation indicators in the normal dataset, and then using a separate correction model with a three-level network architecture and a three-dimensional loss function, the accurate correction of abnormal indicators is achieved, providing effective technical support for high-quality cleaning of real estate data.

[0172] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A real estate data cleaning and quality verification method based on Internet big data, characterized in that, The method comprises the following steps: Step 1: Collecting N pieces of real estate related data from different data sources based on multiple crawler nodes working in parallel; Step 2: Performing multi-dimensional index analysis on each piece of real estate related data to establish a multi-dimensional vector, and establishing a multi-dimensional matrix based on the same data source, and correcting the corresponding multi-dimensional vector by analyzing the multi-dimensional characteristics of the multi-dimensional matrix and the confidence of the same data source; Step 3: Locking real estate abnormal data by performing quality assessment on all once-corrected vectors under the same type of data source, and screening normal data sets similar to each piece of real estate abnormal data from the N pieces of real estate related data, extracting an output array matching the abnormal index of the corresponding piece of real estate abnormal data from the normal data set, and constructing an individual correction model of the abnormal index; Wherein, the construction of the individual correction model of the abnormal index comprises: Calculating the correlation degree of the abnormal index of the real estate abnormal data and other indexes in the normal data set, extracting the correlation index combination to form a correlation feature set, and sorting the numerical sequence of each index in the correlation feature set according to the time stamp to form an output array matching the abnormal index, wherein the array length is the sample amount of the same type of data before and after the collection time of the corresponding piece of abnormal related data M days; Constructing a three-level network architecture including a feature extraction layer, a mapping conversion layer and an output correction layer as a basic model framework; Constructing a loss function based on the output array; Based on the basic model framework and the three-dimensional loss function, an individual correction model is constructed, and the individual correction model outputs a correction value and a correction confidence interval; Wherein, the construction of the loss function based on the output array comprises: Determine the absolute error loss based on the prediction result of the output correction layer and the actual result of the output array, and simultaneously perform a failure operation of random neuron combination on the output neuron set of the feature extraction layer, the mapping conversion layer and the output correction layer to determine the three-layer failure loss; Based on the absolute error loss and the three-layer failure loss, a loss function is constructed; Step 4: Fusing all individual correction models under each piece of real estate abnormal data according to the mutual influence relationship of the abnormal index, and performing secondary correction on the corresponding piece of real estate abnormal data when the quality of the fusion model is qualified, to obtain the cleaned data. 2.The real estate data cleaning and quality checking method based on Internet big data according to claim 1, wherein, Collecting N pieces of real estate related data from different data sources based on multiple crawler nodes working in parallel, comprising: Determine the first collection task queue of each crawler node, add the real estate data source information to be collected to the first collection task queue, and write the collection results corresponding to the tasks in the first collection task queue to the result storage section corresponding to the first collection task queue in the data temporary storage area, and each crawler node is also configured with a task state synchronization tag with the same trigger time range as each remaining node, and N pieces of real estate related data are collected, wherein each crawler node is connected with the data temporary storage area as a data transceiver side, and N is a positive integer; Wherein, the determination of the trigger time range comprises: Obtaining the first sequence number of the data staging area of the staging address of the latest released real estate data, the second sequence number of the data staging area of the latest pushed real estate data, and the third sequence number set of all addresses belonging to the data input port and in the idle state in the address list configured to each crawler node based on the data staging area in the data staging area; Determine the address release order of the data staging area according to the first sequence number and the address staging order of the data staging area according to the second sequence number, and predict the release address block and the staging address block in combination with the working characteristics of the data staging area; Depending on the release address block, the staging address block, and the third sequence number set of the corresponding crawler node, and in combination with the standard communication feedback time length of the first real estate data source in the consistent first collection task queue of the corresponding crawler node, set the trigger time range for each crawler node to start collecting according to the first collection task queue. 3.The real estate data cleaning and quality checking method based on Internet big data according to claim 1, wherein, Once the corresponding multi-dimensional vector is modified, including: The dimension indicators set according to the data source types are used to decompose corresponding piece of real estate related data to obtain index values, and a multi-dimensional vector is constructed by combining the set weights of each dimension indicator wherein, represents the number of dimension indicators under the ith data source type; represents the index value of the corresponding piece of real estate related data under the j1th dimension indicator; represents the set weight of the j1th dimension indicator under the ith data source type; Arranging all multi-dimensional vectors under the same data source type in ascending order of collection timestamp to form an initial matrix, and eliminating redundant dimensions to form a multi-dimensional matrix; Obtaining the core features of the multi-dimensional matrix, and combining the determinant value and the eigenvalue dispersion of the multi-dimensional matrix as auxiliary features to form a multi-dimensional feature set; Determine the confidence of the same data source through the data source access time, the historical data correction rate, and the cross-validation pass rate with the third-party authoritative database; When the principal component in the multi-dimensional feature set deviates from the pre-set confidence interval or the determinant value abnormally fluctuates, perform hierarchical correction according to the confidence. 4.The real estate data cleaning and quality checking method based on Internet big data according to claim 3, wherein, Obtaining the core features of the multi-dimensional matrix, including: Obtaining the overall feature vector of the multi-dimensional matrix; Cutting a random column of the multi-dimensional matrix except the first and last columns to obtain a first vertical sub-matrix and a second vertical sub-matrix, and respectively calculating a first sub-vector and a second sub-vector to obtain a vertical feature vector; Cutting a random row of the multi-dimensional matrix except the first and last rows to obtain a first horizontal sub-matrix and a second horizontal sub-matrix, and respectively calculating a third sub-vector and a fourth sub-vector to obtain a horizontal feature vector; Inputting the overall feature vector, the vertical feature vector, and the horizontal feature vector into the vector analysis model in turn to obtain an overall feature representation sequence, a vertical feature representation sequence, and a horizontal feature representation sequence; Performing deep mining on the overall feature representation sequence, the vertical feature representation sequence, and the horizontal feature representation sequence in turn to determine the enhancement variable of the same sub-representation sequence, wherein the enhancement variable is the weight change accumulation of the dimension index mapped to the sub-feature of the corresponding sub-representation sequence; Enhancing the corresponding feature coefficients in the overall feature vector according to the enhancement variable; Inputting the combination of the overall feature representation sequence, the vertical feature representation sequence, and the horizontal feature representation sequence and the corresponding data source type into the combination analysis model to obtain the derivative sequence of each sub-representation sequence, and deriving new features from the feature vector based on all derivative sequences and the proportion coefficient of each derivative sequence based on the corresponding enhancement variable. extracting sub-representation sequences matched with the new features of derivation and considering them as the first sequences, and meanwhile, extracting second sequences with importance greater than a preset importance from all sub-representation sequences of the overall feature vector; matching the features of the first sequences and the second sequences as core features of the multi-dimensional matrix. 5.The real estate data cleaning and quality checking method based on Internet big data according to claim 1, wherein, Locking the property anomaly data, including: Building a three-dimensional quality evaluation system including integrity, logical consistency and numerical rationality, calculating the quality score of each primary correction vector, and regarding the related data of the vector corresponding to the land property as the land property anomaly data when the quality score is lower than the preset threshold.

6. The method of claim 1, wherein the method further comprises: The dimensions of the quality qualification of the fusion model include: accuracy dimension, robustness dimension and timeliness dimension.

Images