A data processing method and apparatus

Abstract

The application discloses a data processing method and device, and relates to the technical field of big data processing. A specific implementation of the method comprises the following steps: acquiring training sample data; extracting a plurality of features from the cell data and the current confidence data through a preset dimension, performing data correlation analysis on the plurality of features, and taking the features meeting a preset condition as input features; determining the data corresponding to the input features in the cell data and the current confidence data, combining the classification labels of a plurality of cells, training a weak learner, and obtaining a fitting confidence model; predicting the classification label of to-be-processed cell data based on the fitting confidence model, using a fitting confidence calculation method corresponding to the obtained classification label, obtaining a fitting confidence, and taking the fitting confidence as the latest confidence of an evaluation model. The implementation builds the fitting confidence model to evaluate the confidence of different evaluation models under the same standard, and the confidence is more objective and fair compared with the original confidence.

Description

Technical Field

[0001] This invention relates to the field of big data processing technology, and in particular to a data processing method and apparatus. Background Technology

[0002] An automated valuation model (AVM) is a valuation model that uses mathematical statistical methods to combine property attribute data with transaction data to calculate the value of a property. The core method of most automated valuation models is to compare the values ​​of similar properties at the same point in time to derive the appraised value of the target property.

[0003] Confidence level is a common method for evaluating model performance. After a model is built, a confidence level is usually assigned to the model results to determine their reliability; it can be understood as a self-evaluation of the model results. When banks obtain results from multiple valuation models simultaneously during property valuation, they need to evaluate the accuracy of each model's results. However, because each model relies on different data during its construction, the standards and rules for confidence levels differ, making it impossible to objectively evaluate the model's confidence level on a single standard. Summary of the Invention

[0004] In view of this, embodiments of the present invention provide a data processing method and apparatus that solves the problem of evaluating the confidence of different models on the same standard by establishing a fitting confidence level.

[0005] To achieve the above objectives, according to one aspect of the present invention, a data processing method is provided, comprising:

[0006] Obtain training sample data; wherein, the training sample data includes cell data and classification labels of multiple cells, as well as the current confidence data of the estimation model;

[0007] Multiple features are extracted from the cell data and current confidence data through preset dimensions, and data correlation analysis is performed on the multiple features to select features that meet preset conditions as input features.

[0008] The data corresponding to the input parameter features in the cell data and current confidence data are determined, and the weak learner is trained by combining the classification labels of the multiple cells to obtain the fitted confidence model.

[0009] Based on the fitted confidence model, the classification labels of the cell data to be processed are predicted. The fitted confidence is obtained by using the fitted confidence calculation method corresponding to the obtained classification labels. The fitted confidence is then used as the latest confidence of the estimation model.

[0010] Optionally, obtaining training sample data includes:

[0011] The system receives information from multiple housing developments and uses a housing development fusion model to fuse the information to obtain the common communities of the multiple housing developments. The housing development fusion model includes a mapping relationship between the common names and aliases of the communities.

[0012] The third-party value of each shared community is queried. The target communities that exist in the query results are classified and labeled. The community data and classification labels of the target communities are used as training sample data.

[0013] Optionally, the classification and labeling of target cells whose query results indicate existence includes:

[0014] The cell data of a single target cell is input into multiple valuation models to obtain the cell value estimated by each valuation model for the single target cell;

[0015] For a single cell value, determine the difference between the single cell value and the third-party value of the single target cell, and calculate the ratio of the difference to the single cell value;

[0016] Determine the range of the ratio, and use the value of the label category corresponding to the range as the classification label of the individual target cell in the corresponding estimation model.

[0017] Optionally, the step of extracting multiple features from the cell data and current confidence data through preset dimensions, performing data correlation analysis on the multiple features, and using features that meet preset conditions as input features includes:

[0018] Multiple features are extracted from the cell data through cell feature dimension and statistical feature dimension, respectively; and multiple features are extracted from the current confidence data through the estimated confidence dimension.

[0019] Calculate the correlation matrix of all features, plot the correlation matrix as a heatmap, and identify feature pairs in the heatmap whose correlation is greater than a preset correlation threshold;

[0020] Determine the ease of acquisition of each feature in the feature pair in order to remove target features with relatively low ease of acquisition from the feature pair.

[0021] Optional, also includes:

[0022] If the target feature exists in the preset special feature table, the target feature is retained.

[0023] Optionally, the data corresponding to the input parameter features in the determined cell data and current confidence data, combined with the classification labels of the multiple cells, are used to train the weak learner to obtain a fitted confidence model, including:

[0024] The gradient calculation method is called to calculate the gradient of the data for each input parameter feature. The input parameter features are arranged in descending order of the absolute value of the gradient to obtain the input parameter feature sequence.

[0025] From the input parameter feature sequence, the input parameter features with the first preset sampling rate that are ranked first are extracted to generate the first sample set;

[0026] From the remaining input parameter feature sequence, input parameter features with a second preset sampling rate are randomly extracted to generate a second sample set, and the data of each input parameter feature in the set is multiplied by a preset weight coefficient; wherein, the preset weight coefficient is the difference between the preset value and the first preset sampling rate divided by the second preset sampling rate;

[0027] The first sample set and the second sample set are merged, and the weak learner is trained using the merged sample set to obtain a new weak learner.

[0028] Repeat the above operation of randomly extracting input parameters at the second preset sampling rate to update the second sample set for iterative training until the preset number of iterations is reached or the preset convergence value of the loss function is reached. Use the weak learner obtained from the last training as the fitted confidence model.

[0029] Optionally, the cell data to be processed does not include third-party value.

[0030] Optionally, the confidence score for fitting can be calculated as the sum of the base score corresponding to the classification label, the product of the classification label probability score and the preset probability, and the score of similar transactions.

[0031] Among them, the similarity score is obtained by calculating the difference between the value of the community and the most recent transaction price, obtaining the preset difference corresponding to the obtained classification label, and determining it according to the relationship between the obtained difference and the preset difference.

[0032] To achieve the above objectives, according to another aspect of the present invention, a data processing apparatus is provided, comprising:

[0033] The training sample module is used to acquire training sample data; wherein, the training sample data includes cell data and classification labels of multiple cells, as well as the current confidence data of the estimation model;

[0034] The feature extraction module is used to extract multiple features from the cell data and the current confidence data through preset dimensions, perform data correlation analysis on the multiple features, and use the features that meet the preset conditions as input features.

[0035] The model training module is used to determine the data corresponding to the input parameter features in the cell data and the current confidence data, and to train the weak learner by combining the classification labels of the multiple cells to obtain the fitted confidence model.

[0036] The model prediction module is used to predict the classification labels of the cell data to be processed based on the fitted confidence model, use the fitted confidence calculation method corresponding to the obtained classification labels to obtain the fitted confidence, and use the fitted confidence as the latest confidence of the estimation model.

[0037] Optionally, the training sample module is used for:

[0038] The system receives information from multiple housing developments and uses a housing development fusion model to fuse the information to obtain the common communities of the multiple housing developments. The housing development fusion model includes a mapping relationship between the common names and aliases of the communities.

[0039] The third-party value of each shared community is queried. The target communities that exist in the query results are classified and labeled. The community data and classification labels of the target communities are used as training sample data.

[0040] Optionally, the training sample module is used for:

[0041] The cell data of a single target cell is input into multiple valuation models to obtain the cell value estimated by each valuation model for the single target cell;

[0042] For a single cell value, determine the difference between the single cell value and the third-party value of the single target cell, and calculate the ratio of the difference to the single cell value;

[0043] Determine the range of the ratio, and use the value of the label category corresponding to the range as the classification label of the individual target cell in the corresponding estimation model.

[0044] Optionally, the feature extraction module is used for:

[0045] Multiple features are extracted from the cell data through cell feature dimension and statistical feature dimension, respectively; and multiple features are extracted from the current confidence data through the estimated confidence dimension.

[0046] Calculate the correlation matrix of all features, plot the correlation matrix as a heatmap, and identify feature pairs in the heatmap whose correlation is greater than a preset correlation threshold;

[0047] Determine the ease of acquisition of each feature in the feature pair in order to remove target features with relatively low ease of acquisition from the feature pair.

[0048] Optionally, the feature extraction module is further configured to:

[0049] If the target feature exists in the preset special feature table, the target feature is retained.

[0050] Optionally, the model training module is used for:

[0051] The gradient calculation method is called to calculate the gradient of the data for each input parameter feature. The input parameter features are arranged in descending order of the absolute value of the gradient to obtain the input parameter feature sequence.

[0052] From the input parameter feature sequence, the input parameter features with the first preset sampling rate that are ranked first are extracted to generate the first sample set;

[0053] From the remaining input parameter feature sequence, input parameter features with a second preset sampling rate are randomly extracted to generate a second sample set, and the data of each input parameter feature in the set is multiplied by a preset weight coefficient; wherein, the preset weight coefficient is the difference between the preset value and the first preset sampling rate divided by the second preset sampling rate;

[0054] The first sample set and the second sample set are merged, and the weak learner is trained using the merged sample set to obtain a new weak learner.

[0055] Repeat the above operation of randomly extracting input parameters at the second preset sampling rate to update the second sample set for iterative training until the preset number of iterations is reached or the preset convergence value of the loss function is reached. Use the weak learner obtained from the last training as the fitted confidence model.

[0056] Optionally, the cell data to be processed does not include third-party value.

[0057] Optionally, the confidence score for fitting can be calculated as the sum of the base score corresponding to the classification label, the product of the classification label probability score and the preset probability, and the score of similar transactions.

[0058] Among them, the similarity score is obtained by calculating the difference between the value of the community and the most recent transaction price, obtaining the preset difference corresponding to the obtained classification label, and determining it according to the relationship between the obtained difference and the preset difference.

[0059] To achieve the above objectives, according to another aspect of the present invention, a data processing electronic device is provided.

[0060] An electronic device according to an embodiment of the present invention includes: one or more processors; and a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement any of the data processing methods described above.

[0061] To achieve the above objectives, according to another aspect of the present invention, a computer-readable medium is provided having a computer program stored thereon, which, when executed by a processor, implements any of the data processing methods described above.

[0062] To achieve the above objectives, according to another aspect of the present invention, a computing program product is provided. One such computing program product includes a computer program that, when executed by a processor, implements the data processing method provided in the present invention.

[0063] According to the solution provided by the present invention, one embodiment of the above invention has the following advantages or beneficial effects: Based on machine learning algorithms, multi-dimensional feature extraction is performed on the training sample data, combined with the classification labels of the community, to construct a fitted confidence model, thereby realizing the evaluation of the confidence of different automatic housing price valuation models on the same standard. Compared with the original confidence results, it is more objective and fair, thereby updating the confidence of each automatic housing price valuation model so as to compare and apply different automatic housing price valuation models in the future.

[0064] The further effects of the aforementioned unconventional alternative methods will be explained below in conjunction with specific implementation methods. Attached Figure Description

[0065] The accompanying drawings are provided to better understand the invention and are not intended to unduly limit the scope of the invention. Wherein:

[0066] Figure 1 This is a schematic diagram of the main flow of a data processing method according to an embodiment of the present invention;

[0067] Figure 2 This is a flowchart illustrating an optional data processing method according to an embodiment of the present invention;

[0068] Figure 3 This is a flowchart illustrating another optional data processing method according to an embodiment of the present invention;

[0069] Figure 4 Here is an example of a feature correlation heatmap;

[0070] Figure 5 This is a flowchart illustrating another optional data processing method according to an embodiment of the present invention;

[0071] Figure 6 This is a schematic diagram of the main modules of a data processing device according to an embodiment of the present invention;

[0072] Figure 7 This is an exemplary system architecture diagram in which embodiments of the present invention can be applied;

[0073] Figure 8 This is a schematic diagram of the structure of a computer system suitable for implementing the embodiments of the present invention, such as a mobile device or server. Detailed Implementation

[0074] The following description, in conjunction with the accompanying drawings, illustrates exemplary embodiments of the present invention, including various details to aid understanding. These details should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the invention. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0075] It should be noted that, unless otherwise specified, the embodiments and features described in the embodiments of this invention can be combined with each other. The collection, analysis, use, transmission, and storage of data (such as user personal information) in the technical solutions of this application all comply with relevant laws and regulations, are used for legitimate and reasonable purposes, and are not shared, disclosed, or sold outside of these legitimate uses, and are subject to supervision and management by regulatory authorities. Necessary measures should be taken to prevent unauthorized access to such personal information data, ensure that personnel authorized to access personal information data comply with relevant laws and regulations, and ensure the security of user personal information. Once such user personal information data is no longer needed, the risk should be minimized by restricting or even prohibiting data collection and / or deleting the data. When applicable, including in certain relevant applications, user privacy may be protected by de-identifying data, such as by removing specific identifiers (e.g., date of birth), controlling the amount or specificity of stored data, controlling how data is stored, and / or other de-identification methods.

[0076] Confidence level, also known as reliability, confidence level, or confidence coefficient, refers to the uncertainty of the conclusions reached when estimating population parameters through sampling due to the randomness of the sample. The span of a confidence interval is a positive function of the confidence level; that is, the greater the required degree of certainty, the wider the confidence interval will be, which correspondingly reduces the accuracy of the estimate.

[0077] With the development of big data technology, automated property valuation models have been widely applied in various scenarios of real estate finance, greatly improving business efficiency and reducing the operating costs of financial institutions. When valuing properties, banks hope to assess the accuracy of multiple valuation models relatively objectively and fairly under the same standard. However, due to the limitations of the confidence levels given by the models themselves, the original confidence levels of the valuation models cannot be directly used.

[0078] To address this issue, this solution involves establishing a fit confidence score. The fit confidence score is a further assessment of the original model's confidence score; essentially, it's a re-evaluation of the accuracy of the original confidence score. This evaluation requires collecting specific sample data and training a particular model to establish the calculation model for the fit confidence score.

[0079] See Figure 1 The diagram shows the main flowchart of a data processing method provided by an embodiment of the present invention, which includes the following steps:

[0080] S101: Obtain training sample data; wherein, the training sample data includes cell data and classification labels of multiple cells, as well as the current confidence data of the estimation model;

[0081] S102: Extract multiple features from the cell data and current confidence data through preset dimensions, perform data correlation analysis on the multiple features, and use the features that meet the preset conditions as input features;

[0082] S103: Determine the data corresponding to the input parameter features in the cell data and the current confidence data, and train the weak learner by combining the classification labels of the multiple cells to obtain the fitted confidence model;

[0083] S104: Based on the fitted confidence model, predict the classification label of the cell data to be processed, use the fitted confidence calculation method corresponding to the obtained classification label to obtain the fitted confidence, and use the fitted confidence as the latest confidence of the estimation model.

[0084] In the above implementation, for step S101, this solution obtains information on multiple properties in advance from multiple channels (such as home purchase channels and rental channels). A property refers to a house designed, constructed, and sold on the market by a developer with development qualifications in a certain location.

[0085] Different property suppliers may use different names for the same residential complex, such as supplier A - complex name X, supplier B - complex name Y. However, both complex name X and complex name Y are actually different aliases for the same complex. This is similar to pharmaceuticals, which have a generic name and multiple aliases. The generic name is the name that is used worldwide, while the aliases are determined by the pharmaceutical manufacturer. The property fusion model stores the correspondence between supplier-complex aliases, the generic name of the complex, and the aliases of the complex. Therefore, the property fusion model can be used to merge the information of these properties to obtain the common complexes among them.

[0086] Community data includes, but is not limited to: total building area, residential building area, commercial building area, basement building area, chargeable area, green area, number of buildings, number of floors, total number of households, number of residential households, number of commercial households, total number of households sold, number of residential households sold, number of residential households sold (increase / decrease), number of commercial households sold, number of commercial households sold (increase / decrease), number of households delivered, number of households occupied, number of newly occupied households, number of households paying fees, number of households paying fees (increase / decrease), number of underground parking spaces, and number of elevators.

[0087] Some cell data may include third-party value, some of which can be obtained from other online channels, while others may not be available. This solution's training sample data only includes target cells for which third-party value can be obtained. Based on the cell value of these target cells, they are classified and labeled, such as labeled 0, 1, 2, 3. These labeled cell data are the training samples for fitting the confidence model.

[0088] In addition, this solution also considers the original confidence level data for each valuation model. Confidence level here includes, but is not limited to, confidence scores (e.g., Model 1 - 85% confidence level, Model 2 - 83% confidence level). It may also include other data besides confidence scores; see subsequent sections for details. Figure 3 As shown in the description.

[0089] For step S102, after obtaining the training sample data for the fitted confidence model, to simplify the training process, features can be extracted from the training sample data in three dimensions: cell feature dimension, statistical feature dimension, and estimated confidence dimension, resulting in multiple features, such as the last 20 features. Data correlation analysis is then performed on these last 20 features, and finally, a subset of features are initially selected as input parameters for the fitted confidence model. See subsequent sections for details. Figure 3 The description shown will not be repeated here.

[0090] For step S103, after obtaining the input parameter features of the fitted confidence model, the input parameter features include the data, classification label, number of iterations d, sampling rate a for large gradient data, sampling rate b for small gradient data, loss function, and type of weak learner (generally a decision tree). A grid search is used to find the optimal values ​​of the model's important parameters. The model is then trained using LightGBM to generate a well-trained strong learner, which serves as the fitted confidence model. The model's advantages are as follows:

[0091] 1) This model is based on the Histogram-based decision tree algorithm;

[0092] 2) Gradient-based One-Side Sampling (GOSS): This can reduce a large number of data instances with only small gradients. When calculating information gain, only the remaining data with high gradients can be used, which saves time and space compared to other models.

[0093] 3) Exclusive Feature Bundling (EFB): This can bind many mutually exclusive features into one feature to achieve dimensionality reduction. It is a leaf-wise leaf growth strategy with depth constraints.

[0094] 4) Directly supports categorical features without the need for normalization, and supports efficient parallel processing.

[0095] In step S104, the third-party values ​​for the cells in the training sample data can be queried. However, these third-party values ​​are mainly used to determine the classification label of the cell, which serves as the training input for the fitted confidence model. Therefore, the fitted confidence model trains the mapping relationship between the input feature data and the classification label. Consequently, when using the fitted confidence model to predict the classification label of the cell later, the cell data may or may not contain third-party values; preferably, it does not.

[0096] After predicting the classification label of a cell, the confidence level of the fit can be calculated based on its classification label. Assuming there are 5 levels of classification labels, the method for calculating the confidence level of the fit is as follows:

[0097] 1) Tag 0: Base score 0.8 + probability score of category tag: p*0.1 + similar transaction score: similar +0.1, dissimilar -0.1

[0098] 2) Tag 1: Base score 0.7 + probability score of category tag: p*0.1 + similar transaction score: similar +0.1, dissimilar -0.1

[0099] 3) Tag 2: Base score 0.6 + probability score of the category tag: p*0.1 + similar transaction score: similar +0.1, dissimilar -0.1

[0100] 4) Tag 3: Base score 0.5 + probability score of the category tag: p*0.1 + similar transaction score: similar +0.1, dissimilar -0.1

[0101] 5) Tag 4: Base score 0.5 - Probability score of the category tag: p*0.1 + Similar transaction score: Similar +0.1, Dissimilar -0.1

[0102] The logic for judging whether transactions are similar is as follows: Calculate the transaction PPE between the benchmark price of the neighborhood and the most recent transaction price. If the predicted category label is 0, then the transaction PPE < 10 is considered similar; if the predicted category label is 1 or 2, then the transaction PPE < 15 is considered similar; if the predicted category label is 3 or 4, then the transaction PPE < 20 is considered similar.

[0103] When the fitted confidence model outputs classification labels, it only outputs the label with the highest probability. For example, when calculating the classification labels of a certain community: label 0-80%, label 1-10%, label 2-5%, label 3-3%, label 4-2%, label 0 is finally used as the final classification label. When calculating the fitted confidence, p=80%.

[0104] After calculating the fit confidence level, this fit confidence level is used as the latest confidence level of the estimation model. For example, if the original confidence level of Model 1 is 80%, the result obtained by using the fit confidence level model and the fit confidence level calculation method is 78%, so the confidence level of Model 1 is changed from 80% to 78%.

[0105] The method provided in the above embodiments improves the accuracy of the model by training a fitted confidence model through multi-dimensional feature extraction. The fitted confidence model can predict the classification labels of other communities without third-party average price benchmark data, and then update the confidence of the valuation model based on the classification labels. The entire process updates the confidence under the same standard, which facilitates banks to quickly, accurately, and objectively obtain property valuations when conducting property valuations, reducing valuation costs and time, and reducing the reliance on human subjectivity in valuation.

[0106] See Figure 2 The diagram illustrates an optional data processing method according to an embodiment of the present invention, including the following steps:

[0107] S201: Input the cell data of a single target cell into multiple valuation models to obtain the cell value estimated by each valuation model for the single target cell;

[0108] S202: For a single cell value, determine the difference between the single cell value and the third-party value of the single target cell, and calculate the ratio of the difference to the single cell value;

[0109] S203: Determine the range of the ratio, and use the value of the label category corresponding to the range as the classification label of the individual target cell in the corresponding estimation model.

[0110] In the above implementation, for steps S201 to S202, for a single target cell, its cell data is input into multiple valuation models to obtain the value calculated by each valuation model for the target cell, such as model 1-1.5 million, model 2-1.6 million. The value here is usually the benchmark price or average price of houses in the cell. There are usually multiple houses in the cell.

[0111] The next step is to analyze the value estimated by each individual model, calculate the difference between the value of the residential community and the value of its third party, and then calculate the ratio of this difference to the value of the residential community to obtain the percentage difference (PPE) between the residential community value and the value of its third party. For example, if the value of the residential community is 2 million and the value of its third party is 2.1 million, the PPE is calculated as |200 - 210| / 200 * magnification factor k = 0.05k. Assuming the magnification factor k is 200, the resulting PPE is 10.

[0112] For step S203, after calculating the difference between the cell value and its third-party value (ppe), the cells can be classified according to the ppe value. For example, cells with ppe <= 15 are classified into one category, and cells with ppe > 15 are classified into another category. The former category is labeled as Label 0, and the latter category is labeled as Label 1.

[0113] Furthermore, analysis can be performed using a pylon plot, which combines the features of a box plot and a kernel density estimation plot. It characterizes and compares the distribution of continuous variable data under two categorical variables, and is an effective method for observing the distribution of multiple data. It can observe outliers, medians, upper and lower quantiles, and ranges of feature data, thereby guiding the selection of features and data processing.

[0114] However, in practice, it was found that the above classification results had a large granularity when training the confidence model. To solve this problem, this solution further restricts the classification level to divide the labeled classification labels into multiple levels, preferably five categories.

[0115] Table 1

[0116]

[0117] The range of each difference is determined, and the value corresponding to each difference range is used as the classification label value of the target cell in the corresponding valuation model. For example, in Model 1: Cell A: Base price label 0, Cell B: Base price label 1, Cell C: Base price label 2, Cell D: Base price label 3, Cell E: Base price label 4, while in Model 2: Cell A: Base price label 2, Cell B: Base price label 3, Cell C: Base price label 1, Cell D: Base price label 1, Cell E: Base price label 0.

[0118] It should be noted that the aforementioned PPE is obtained by multiplying the difference between the cell value and the third-party value by a certain factor, and then determining the range of the multiplied value. Alternatively, the multiplication can be omitted, and the difference between the cell value and the third-party value can be used directly for judgment, but the critical values ​​of the range, such as 5, 10, and 15, need to be adaptively reduced by a certain factor.

[0119] The method provided in the above embodiments may yield different estimation results for the same target cell by different estimation models. To address this, a method is provided to classify the target cell based on the estimation results, thereby obtaining the classification labels of the target cell under different estimation models and realizing the input of the training results of the fitted confidence model.

[0120] See Figure 3 The diagram illustrates another optional data processing method according to an embodiment of the present invention, including the following steps:

[0121] S301: Extract multiple features from cell data using cell feature dimension and statistical feature dimension respectively;

[0122] S302: Extract multiple features from the current confidence data through the valuation confidence dimension;

[0123] S303: Calculate the correlation matrix of all features, plot the correlation matrix as a heatmap, and determine the feature pairs in the heatmap whose correlation is greater than a preset correlation threshold;

[0124] S304: Determine the ease of acquisition of each feature in the feature pair, so as to remove the target features with relatively low ease of acquisition in the feature pair.

[0125] In the above implementation, the fit confidence score is used to re-evaluate the accuracy of the original valuation model's valuation results. The selection of input parameter features is mainly considered from the following two aspects:

[0126] 1) Input parameters all have a certain impact on the valuation results of the valuation model. In addition to considering the original valuation results and confidence level of the valuation model, it is also necessary to measure whether the original confidence level of the valuation model is reasonable. Therefore, the input variables of the original confidence level of the valuation model need to be added to the input parameters of the fitted confidence level model.

[0127] 2) The selected input features need to have a certain degree of difference. If the correlation between two features is too strong, it will interfere with the training of the confidence model, similar to a certain feature being weighted.

[0128] For steps S301 to S302, this scheme extracts features from the training sample data in three dimensions: cell feature dimension, statistical feature dimension, and estimated confidence dimension. The resulting features and their descriptions are as follows:

[0129] Table 2 Examples of Community Characteristics

[0130]

[0131] Table 3 Examples of Statistical Characteristics

[0132]

[0133] Table 4 Examples of Valuation Confidence Characteristics

[0134]

[0135] For steps S303 to S304, the correlation matrix of all features is calculated and plotted as a heatmap for analysis. See [link to relevant documentation]. Figure 4 As shown (only a partial example of feature correlation), it can be seen that the correlation between features is not very high. Furthermore, the correlations in the heatmap that exceed the preset correlation threshold can be highlighted.

[0136] When two features have a very high correlation, such as greater than 0.7, one of the easier features to obtain can be selected and retained, while the other difficult feature can be removed. An ease-of-obtaining table can be pre-set, recording the ease of obtaining each feature. The ease of obtaining is determined manually by staff. For example, if feature 1 has an ease of obtaining of 50% and feature 2 has an ease of obtaining of 70%, it means that feature 2 is easier to obtain than feature 1. Therefore, only feature 2 should be retained, and feature 1 should be removed.

[0137] It should be noted that some deleted features need to be retained due to their specific characteristics. Deleting them would lead to unrealistic fitting in subsequent data. For example, sample dispersion and price calculation methods may have data distortion caused by sample dispersion, so they are temporarily retained. In this case, a special feature table can be set up to store features that cannot be deleted. Therefore, even if a feature is highly correlated with other features, it cannot be deleted.

[0138] Example of filtered features:

[0139] Table 5 Examples of features after processing

[0140]

[0141] The method provided in the above embodiments extracts features from the training sample data in three dimensions: cell feature dimension, statistical feature dimension, and estimated confidence dimension, to obtain the input parameter features of the fitted confidence model. This reduces the amount of training data input, achieves targeted data input effects, avoids interference from irrelevant data, and improves the accuracy of model training.

[0142] See Figure 5 The diagram illustrates another optional data processing method according to an embodiment of the present invention, comprising the following steps:

[0143] S501: Call the gradient calculation method to calculate the gradient of the data of each input parameter feature, and arrange the input parameter features in descending order of the absolute value of the gradient to obtain the input parameter feature sequence.

[0144] S502: Extract the input parameter features with the first preset sampling rate that are ranked first from the input parameter feature sequence to generate a first sample set;

[0145] S503: Randomly extract input parameters at the second preset sampling rate from the remaining input parameter feature sequence to generate a second sample set, and multiply the data of each input parameter feature in the set by a preset weight coefficient; wherein, the preset weight coefficient is the ratio of the difference between the preset value and the first preset sampling rate to the second preset sampling rate;

[0146] S504: Merge the first sample set and the second sample set, and use the merged sample set to train the weak learner to obtain a new weak learner;

[0147] S505: Repeat the above operation of randomly extracting input parameters at the second preset sampling rate to update the second sample set for iterative training until the preset number of iterations is reached or the preset convergence value of the loss function is reached. Use the weak learner obtained from the last training as the fitted confidence model.

[0148] In the above implementation, for steps S501 to S505, the inputs to the model training process are: training data, number of iterations d, sampling rate a for large gradient data, sampling rate b for small gradient data, loss function, and type of weak learner (generally a decision tree).

[0149] 1. First, the gradient calculation function is invoked to calculate the gradient of each input parameter feature. This gradient calculation method is existing and will not be elaborated here. The input parameter features are then sorted in descending order according to the absolute value of the gradient, resulting in the input parameter feature sequence.

[0150] 2. Select the top a*100% samples from the input parameter feature sequence to generate a set of large gradient sample points (i.e., the first sample set);

[0151] 3. For the remaining input parameter feature sequence, including (1-a)*100% of the input parameter features, randomly select b*(1-a)*100% of samples to generate a set of small gradient samples (i.e., the second sample set);

[0152] 4. Merge the large gradient sample set and the small gradient sample set, and multiply the data in the small gradient sample set by a weight coefficient frac{1-a}{b}; frac represents a fraction, for example: frac{1}{x} means 1 / x;

[0153] 5. Use the merged sample set to train the input weak learner to obtain a new weak learner;

[0154] 6. Throughout the process, the sampling rate 'a' for large gradient data and the sampling rate 'b' for small gradient data remain unchanged, therefore the first sample set will not change. Considering that the second sample set is obtained through random selection, if 'd' iterations are required, only the second sample set can be updated. Therefore, during iteration, it is only necessary to repeat step 3 until the number of iterations 'd' or the preset convergence value of the loss function is reached. If the result converges to around 0.001, the weak learner obtained from the last training is the desired fitted confidence model.

[0155] The method provided in the above embodiments sets different sampling rates and selection methods for different gradient sample sets, and the sample set obtained in each iteration is different, thereby achieving the purpose of cyclic iteration.

[0156] See Figure 6 The diagram shows a schematic representation of the main modules of a data processing apparatus 600 provided in an embodiment of the present invention, including:

[0157] The training sample module 601 is used to acquire training sample data; wherein, the training sample data includes cell data and classification labels of multiple cells, as well as the current confidence data of the estimation model;

[0158] The feature extraction module 602 is used to extract multiple features from the cell data and the current confidence data through a preset dimension, perform data correlation analysis on the multiple features, and use the features that meet the preset conditions as input features.

[0159] The model training module 603 is used to determine the data corresponding to the input parameter features in the cell data and the current confidence data, and to train the weak learner by combining the classification labels of the multiple cells to obtain the fitted confidence model.

[0160] The model prediction module 604 is used to predict the classification label of the cell data to be processed based on the fitted confidence model, use the fitted confidence calculation method corresponding to the obtained classification label to obtain the fitted confidence, and use the fitted confidence as the latest confidence of the estimation model.

[0161] In the apparatus of this invention, the training sample module 601 is used for:

[0162] The system receives information from multiple housing developments and uses a housing development fusion model to fuse the information to obtain the common communities of the multiple housing developments. The housing development fusion model includes a mapping relationship between the common names and aliases of the communities.

[0163] The third-party value of each shared community is queried. The target communities that exist in the query results are classified and labeled. The community data and classification labels of the target communities are used as training sample data.

[0164] In the apparatus of this invention, the training sample module 601 is used for:

[0165] The cell data of a single target cell is input into multiple valuation models to obtain the cell value estimated by each valuation model for the single target cell;

[0166] For a single cell value, determine the difference between the single cell value and the third-party value of the single target cell, and calculate the ratio of the difference to the single cell value;

[0167] Determine the range of the ratio, and use the value of the label category corresponding to the range as the classification label of the individual target cell in the corresponding estimation model.

[0168] In the apparatus of this invention, the feature extraction module 602 is used for:

[0169] Multiple features are extracted from the cell data through cell feature dimension and statistical feature dimension, respectively; and multiple features are extracted from the current confidence data through the estimated confidence dimension.

[0170] Calculate the correlation matrix of all features, plot the correlation matrix as a heatmap, and identify feature pairs in the heatmap whose correlation is greater than a preset correlation threshold;

[0171] Determine the ease of acquisition of each feature in the feature pair in order to remove target features with relatively low ease of acquisition from the feature pair.

[0172] In the apparatus of this invention, the model training module 603 is used for:

[0173] The gradient calculation method is called to calculate the gradient of the data for each input parameter feature. The input parameter features are arranged in descending order of the absolute value of the gradient to obtain the input parameter feature sequence.

[0174] From the input parameter feature sequence, the input parameter features with the first preset sampling rate that are ranked first are extracted to generate the first sample set;

[0175] From the remaining input parameter feature sequence, input parameter features with a second preset sampling rate are randomly extracted to generate a second sample set, and the data of each input parameter feature in the set is multiplied by a preset weight coefficient; wherein, the preset weight coefficient is the difference between the preset value and the first preset sampling rate divided by the second preset sampling rate;

[0176] The first sample set and the second sample set are merged, and the weak learner is trained using the merged sample set to obtain a new weak learner.

[0177] Repeat the above operation of randomly extracting input parameters at the second preset sampling rate to update the second sample set for iterative training until the preset number of iterations is reached or the preset convergence value of the loss function is reached. Use the weak learner obtained from the last training as the fitted confidence model.

[0178] In the implementation device of this invention, the fitting confidence calculation method is the sum of three factors: the basic score corresponding to the classification label, the product of the classification label probability score and the preset probability, and the transaction similarity score.

[0179] Among them, the similarity score is obtained by calculating the difference between the value of the community and the most recent transaction price, obtaining the preset difference corresponding to the obtained classification label, and determining it according to the relationship between the obtained difference and the preset difference.

[0180] Furthermore, the specific implementation details of the device described in the embodiments of the present invention have been described in detail in the above-described method, so the details will not be repeated here.

[0181] Figure 7 An exemplary system architecture 700 to which embodiments of the present invention can be applied is shown, including terminal devices 701, 702, 703, network 704, and server 705 (this is merely an example).

[0182] Terminal devices 701, 702, and 703 can be various electronic devices with displays and support for web browsing, and have various communication client applications installed. Users can use terminal devices 701, 702, and 703 to interact with server 705 through network 704 to receive or send messages, etc.

[0183] Network 704 is a medium used to provide a communication link between terminal devices 701, 702, 703 and server 705. Network 704 may include various connection types, such as wired or wireless communication links or fiber optic cables, etc.

[0184] Server 705 can be a server that provides various services. It should be noted that the methods provided in the embodiments of the present invention are generally executed by server 705, and correspondingly, the devices are generally set in server 705.

[0185] It should be understood that Figure 7 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0186] The following is for reference. Figure 8 It shows a schematic diagram of the structure of a computer system 800 suitable for implementing a terminal device of the present invention. Figure 8 The terminal device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of the present invention.

[0187] like Figure 8 As shown, the computer system 800 includes a central processing unit (CPU) 801, which can perform various appropriate actions and processes based on programs stored in read-only memory (ROM) 802 or programs loaded from storage section 808 into random access memory (RAM) 803. The RAM 803 also stores various programs and data required for the operation of the system 800. The CPU 801, ROM 802, and RAM 803 are interconnected via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0188] The following components are connected to I / O interface 805: an input section 806 including a keyboard, mouse, etc.; an output section 807 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 808 including a hard disk, etc.; and a communication section 809 including a network interface card such as a LAN card, modem, etc. The communication section 809 performs communication processing via a network such as the Internet. A drive 810 is also connected to I / O interface 805 as needed. A removable medium 811, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on drive 810 as needed so that computer programs read from it can be installed into storage section 808 as needed.

[0189] In particular, according to the embodiments disclosed in this invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this invention include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 809, and / or installed from removable medium 811. When the computer program is executed by central processing unit (CPU) 801, it performs the functions defined above in the system of this invention.

[0190] It should be noted that the computer-readable medium shown in this invention can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this invention, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0191] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0192] The modules described in the embodiments of the present invention can be implemented in software or hardware. The described modules can also be housed in a processor; for example, a processor can be described as including a training sample module, a feature extraction module, a model training module, and a model prediction module. The names of these modules do not necessarily limit the module itself; for example, the model prediction module can also be described as a "model confidence update module".

[0193] In another aspect, the present invention also provides a computer-readable medium, which may be included in the device described in the above embodiments; or it may exist independently and not assembled into the device. The computer-readable medium carries one or more programs that, when executed by the device, cause the device to perform any of the data processing methods described above.

[0194] The computer program product of the present invention includes a computer program that, when executed by a processor, implements the data processing method in the embodiments of the present invention.

[0195] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can occur depending on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A data processing method, characterized in that, include: Acquire training sample data; wherein the training sample data includes community data and classification labels of multiple communities, as well as the current confidence data of the valuation model; the acquisition of training sample data includes: receiving information from multiple properties, using a property fusion model to fuse the information from the multiple properties to obtain the common communities of the multiple properties; wherein the property fusion model includes the mapping relationship between the common names and aliases of the communities; querying the third-party value of each common community, and classifying and labeling the target communities that exist in the query results; Multiple features are extracted from the cell data and current confidence data through preset dimensions. Data correlation analysis is performed on the multiple features to select features that meet preset conditions as input features. This includes: extracting multiple features from the cell data through cell feature dimensions and statistical feature dimensions respectively; and extracting multiple features from the current confidence data through the estimated confidence dimension; calculating the correlation matrix of all features; plotting the correlation matrix into a heatmap; identifying feature pairs in the heatmap with a correlation greater than a preset correlation threshold; and determining the ease of acquisition of each feature in the feature pair to remove target features with relatively low ease of acquisition from the feature pair. The process involves identifying data corresponding to the input parameter features in the cell data and current confidence data, combining this data with the classification labels of the multiple cells, and training a weak learner to obtain a fitted confidence model. This includes: calculating the gradient of each input parameter feature using a gradient calculation method; arranging the input parameter features in descending order of absolute gradient value to obtain an input parameter feature sequence; extracting the top-ranked input parameter features with a first preset sampling rate from the input parameter feature sequence to generate a first sample set; and randomly extracting input parameter features with a second preset sampling rate from the remaining input parameter feature sequences to generate a second sample set. The first sample set and the second sample set are combined, and the data of each input parameter feature in the set are multiplied by a preset weight coefficient. The preset weight coefficient is the difference between the preset value and the first preset sampling rate divided by the second preset sampling rate. The first sample set and the second sample set are merged, and the weak learner is trained using the merged sample set to obtain a new weak learner. The above operation of randomly extracting input parameter features at the second preset sampling rate is repeated to update the second sample set for iterative training until the preset number of iterations is reached or the preset convergence value of the loss function is reached. The weak learner obtained from the last training is used as the fitted confidence model. Based on the fitted confidence model, the classification labels of the cell data to be processed are predicted. The fitted confidence is obtained by using the fitted confidence calculation method corresponding to the obtained classification labels. The fitted confidence is then used as the latest confidence of the estimation model.

2. The method according to claim 1, characterized in that, The process of classifying and labeling target cells that exist in the query results includes: The cell data of a single target cell is input into multiple valuation models to obtain the cell value estimated by each valuation model for the single target cell; For a single cell value, determine the difference between the single cell value and the third-party value of the single target cell, and calculate the ratio of the difference to the single cell value; Determine the range of the ratio, and use the value of the label category corresponding to the range as the classification label of the individual target cell in the corresponding estimation model.

3. The method according to claim 1, characterized in that, Also includes: If the target feature exists in the preset special feature table, the target feature is retained.

4. The method according to claim 1, characterized in that, The cell data to be processed does not include third-party value.

5. The method according to claim 1, characterized in that, The confidence score for fitting is calculated as the sum of the base score corresponding to the classification label, the product of the classification label probability score and the preset probability, and the score of similar transactions. Among them, the similarity score is obtained by calculating the difference between the value of the community and the most recent transaction price, obtaining a preset difference corresponding to the obtained classification label, and determining the score based on the relationship between the obtained difference and the preset difference.

6. A data processing apparatus, characterized in that, include: The training sample module is used to acquire training sample data; wherein, the training sample data includes community data and classification labels of multiple communities, as well as the current confidence data of the valuation model; the acquisition of training sample data includes: receiving information from multiple properties, using a property fusion model to fuse the information from the multiple properties to obtain the common communities of the multiple properties; wherein, the property fusion model includes the mapping relationship between the common names and aliases of the communities; querying the third-party value of each common community, and classifying and labeling the target communities that exist in the query results; The feature extraction module is used to extract multiple features from the cell data and current confidence data through preset dimensions, perform data correlation analysis on the multiple features, and use features that meet preset conditions as input features. This includes: extracting multiple features from the cell data through cell feature dimensions and statistical feature dimensions respectively; and extracting multiple features from the current confidence data through the estimated confidence dimension; calculating the correlation matrix of all features, plotting the correlation matrix as a heatmap, identifying feature pairs in the heatmap with a correlation greater than a preset correlation threshold; and determining the ease of acquisition of each feature in the feature pair to remove target features with relatively low ease of acquisition from the feature pair. The model training module is used to determine the data corresponding to the input parameter features in the cell data and current confidence data, and train the weak learner by combining the classification labels of the multiple cells to obtain a fitted confidence model. This includes: calculating the gradient of each input parameter feature using a gradient calculation method; arranging the input parameter features in descending order of absolute gradient value to obtain an input parameter feature sequence; extracting the top-ranked input parameter features with a first preset sampling rate from the input parameter feature sequence to generate a first sample set; and randomly extracting input parameter features with a second preset sampling rate from the remaining input parameter feature sequences to generate a second sample set. The second sample set is used, and the data of each input feature in the set is multiplied by a preset weight coefficient. The preset weight coefficient is the difference between the preset value and the first preset sampling rate divided by the second preset sampling rate. The first sample set and the second sample set are merged, and the weak learner is trained using the merged sample set to obtain a new weak learner. The above operation of randomly extracting input features at the second preset sampling rate is repeated to update the second sample set for iterative training until the preset number of iterations is reached or the preset convergence value of the loss function is reached. The weak learner obtained from the last training is used as the fitted confidence model. The model prediction module is used to predict the classification labels of the cell data to be processed based on the fitted confidence model, use the fitted confidence calculation method corresponding to the obtained classification labels to obtain the fitted confidence, and use the fitted confidence as the latest confidence of the estimation model.

7. The apparatus according to claim 6, characterized in that, The training sample module is used for: The cell data of a single target cell is input into multiple valuation models to obtain the cell value estimated by each valuation model for the single target cell; For a single cell value, determine the difference between the single cell value and the third-party value of the single target cell, and calculate the ratio of the difference to the single cell value; Determine the range of the ratio, and use the value of the label category corresponding to the range as the classification label of the individual target cell in the corresponding estimation model.

8. The apparatus according to claim 6, characterized in that, The confidence score for fitting is calculated as the sum of the base score corresponding to the classification label, the product of the classification label probability score and the preset probability, and the score of similar transactions. Among them, the similarity score is obtained by calculating the difference between the value of the community and the most recent transaction price, obtaining a preset difference corresponding to the obtained classification label, and determining the score based on the relationship between the obtained difference and the preset difference.

9. An electronic device, characterized in that, include: One or more processors; Storage device for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-5.

10. A computer-readable medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1-5.

11. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-5.

Images