House resource search comprehensive sorting feature weight calculation method based on big data technology

Abstract

The invention discloses a house resource search comprehensive sorting feature weight calculation method based on a big data technology, and the method comprises the following steps: S1, data source integration and preprocessing: carrying out the cleaning, duplicate removal and standardized preprocessing operation of massive house resource search log data, and guaranteeing the data quality; s2, feature selection and extraction: screening out key features which have great influence on a sorting result according to service requirements and user feedback; extracting values or statistics of the features from the original data by using a data mining technology; s3, weight calculation and optimization: learning the weight of each feature from historical data; s4, performing Spark distributed calculation: deploying the preprocessed data and a weight calculation method into a Spark cluster, and performing rapid processing on mass data by utilizing the parallel calculation capability of Spark; and S5, performing a feature weight AB test. According to the method, scientific and objective distribution of the feature weights is realized, and the accuracy and credibility of the sorting result are improved.

Description

Technical Field

[0001] The present invention relates to the field of data analysis application technology, and in particular to a method for calculating feature weights of comprehensive ranking of house source searches based on big data technology. Background Art

[0002] With the booming rental market and the rapid development of Internet technology, online housing search platforms have become an important channel for users to find housing. However, existing housing search sorting algorithms often have problems with strong subjectivity and poor flexibility in the allocation of feature weights, making it difficult to accurately reflect users' actual needs and preferences for housing. At the same time, in the face of massive data, traditional computing methods cannot meet the timeliness requirements in terms of processing speed and efficiency.

[0003] Specifically, the problems and shortcomings of the prior art include: Weight allocation is highly subjective: The allocation of feature weights often relies on the developer’s experience or intuition and lacks scientific and objective basis.

[0004] Poor flexibility: It is difficult to adjust weights in a timely manner based on user feedback and market changes, resulting in the sorting results being out of touch with actual needs.

[0005] Low processing efficiency: When processing large-scale data, traditional computing methods face challenges such as insufficient computing resources and long processing time. Summary of the invention

[0006] The purpose of the present invention is to provide a method for calculating the weight of comprehensive ranking features of house source search based on big data technology to solve the problems in the prior art.

[0007] To achieve the above object, the present invention provides the following technical solution: a method for calculating the weight of comprehensive ranking features of house source search based on big data technology, comprising the following steps: S1. Data source integration and preprocessing: Clean, remove duplicates, and standardize massive amounts of property search log data to ensure data quality; S2. Feature selection and extraction: According to business needs and user feedback, select key features that have a greater impact on the sorting results; use data mining technology to extract the values ​​or statistics of these features from the original data; S3. Weight calculation and optimization: Design a weight calculation method based on big data, learn the weight of each feature from historical data; find the optimal weight combination through cross-validation and grid search techniques; S4, Spark distributed computing: deploy the preprocessed data and weight calculation method to the Spark cluster, use Spark's parallel computing capabilities to quickly process massive data and calculate the optimal feature weight combination; S5. Feature weight AB test: Apply the calculated weight value to the platform's online property search ranking, and conduct AB tests on users to verify the weight effect online.

[0008] Preferably, the massive amount of house search log data in S1 is data from the company's APP site.

[0009] Preferably, the data cleaning is to remove invalid data, the data deduplication is to remove duplicate data and fill in missing data, and the data standardization is to unify the data format.

[0010] Preferably, the key features screened in S2 include price, community, business district, subway station, room type, rental type, commuting location, check-in date, video, and VR; the statistics are used to calculate the distance between the house and the key features.

[0011] Preferably, S3 specifically includes: Use correlation analysis or variance analysis to assess the relationship between different features and listing popularity; Build multiple supervised learning models to predict the popularity of listings and thus obtain the importance of features; By fusing the results of multiple models, more stable feature weights are obtained.

[0012] Preferably, the weights in S3 also include user behavior analysis when calculating, specifically including analyzing the features that users care about based on the user's house clicking behavior during the house search process.

[0013] Preferably, the house search ranking in S5 needs to adjust feature weights based on real-time user behavior data.

[0014] Compared with the prior art, the present invention has the following beneficial effects: Scientific and objective weight allocation: Through big data technology, scientific and objective allocation of feature weights is achieved, improving the accuracy and credibility of sorting results.

[0015] Strong flexibility: Ability to adjust weights in a timely manner based on user feedback and market changes to maintain the timeliness and pertinence of the sorting results.

[0016] High processing efficiency: Distributed computing frameworks such as Spark are used to quickly process massive amounts of data and meet timeliness requirements.

[0017] Improve user experience: Improve user satisfaction and platform competitiveness by providing more accurate and personalized housing search results. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings: Figure 1 It is a flow chart of the present invention. DETAILED DESCRIPTION

[0019] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention.

[0020] See also Figure 1 In an embodiment of the present invention, a method for calculating the weight of comprehensive ranking features of house source search based on big data technology includes the following steps: S1. Data source integration and preprocessing: Clean, remove duplicates, and standardize massive amounts of property search log data to ensure data quality; S2. Feature selection and extraction: According to business needs and user feedback, select key features that have a greater impact on the sorting results; use data mining technology to extract the values ​​or statistics of these features from the original data; S3. Weight calculation and optimization: Design a weight calculation method based on big data, learn the weight of each feature from historical data; find the optimal weight combination through cross-validation and grid search techniques; S4, Spark distributed computing: deploy the preprocessed data and weight calculation method to the Spark cluster, use Spark's parallel computing capabilities to quickly process massive data and calculate the optimal feature weight combination; S5. Feature weight AB test: Apply the calculated weight value to the platform's online property search ranking, and conduct AB tests on users to verify the weight effect online.

[0021] Preferably, the massive amount of house search log data in S1 is data from the company's APP site.

[0022] Preferably, the data cleaning is to remove invalid data, the data deduplication is to remove duplicate data and fill in missing data, and the data standardization is to unify the data format.

[0023] Preferably, the key features screened in S2 include price, community, business district, subway station, room type, rental type, commuting location, check-in date, video, and VR; the statistics are used to calculate the distance between the house and the key features.

[0024] Preferably, S3 specifically includes: Use correlation analysis or ANOVA to assess the relationship between different features and listing popularity.

[0025] Preferably, the weights in S3 also include user behavior analysis when calculating, specifically including analyzing the features that users care about based on the user's house clicking behavior during the house search process.

[0026] Preferably, the house search ranking in S5 needs to adjust feature weights based on real-time user behavior data.

[0027] The working principle of the present invention is as follows: pre-processing operations such as cleaning, de-duplication and standardization are performed on massive house search log data to ensure data quality; key features with a greater impact on sorting results are screened out according to business needs and user feedback; values ​​or statistics of these features are extracted from raw data using data mining technology; a weight calculation method based on big data is designed to learn the weights of each feature from historical data; the optimal weight combination is found through cross-validation and grid search technology; the pre-processed data and weight calculation method are deployed in a Spark cluster, and the parallel computing capability of Spark is used to quickly process massive data and calculate the optimal feature weight combination; the calculated weight value is applied to the online house search sorting of the platform, and an AB test is conducted on users to verify the weight effect online.

[0028] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art can still modify the technical solutions described in the aforementioned embodiments or replace some of the technical features therein by equivalents. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for calculating the weight of comprehensive ranking features of house source search based on big data technology, characterized in that: The following steps are involved: S1. Data source integration and preprocessing: Clean, remove duplicates, and standardize massive amounts of property search log data to ensure data quality; S2. Feature selection and extraction: According to business needs and user feedback, select key features that have a greater impact on the sorting results; use data mining technology to extract the values ​​or statistics of these features from the original data; S3. Weight calculation and optimization: Design a weight calculation method based on big data, learn the weight of each feature from historical data; find the optimal weight combination through cross-validation and grid search techniques; S4, Spark distributed computing: deploy the preprocessed data and weight calculation method to the Spark cluster, use Spark's parallel computing capabilities to quickly process massive data and calculate the optimal feature weight combination; S5. Feature weight AB test: Apply the calculated weight value to the platform's online property search ranking, and conduct AB tests on users to verify the weight effect online.

2. According to the method for calculating the weight of comprehensive ranking features of house source search based on big data technology in claim 1, it is characterized in that: The massive amount of property search log data in S1 is the data within the company’s APP.

3. A method for calculating the weight of comprehensive ranking features of house source search based on big data technology according to claim 1 or 2, characterized in that: The data cleaning is to remove invalid data, the data deduplication is to remove duplicate data and fill in missing data, and the data standardization is to unify the data format.

4. According to the method for calculating the weight of comprehensive ranking features of house source search based on big data technology in claim 1, it is characterized in that: The key features screened in S2 include price, community, business district, subway station, room type, rental type, commuting location, check-in date, video, and VR; the statistics are used to calculate the distance between the house and the key features.

5. According to the method for calculating the weight of comprehensive ranking features of house source search based on big data technology in claim 1, it is characterized in that: The S3 specifically includes: Use correlation analysis or ANOVA to assess the relationship between different features and listing popularity.

6. The method for calculating the weight of comprehensive ranking features of house source search based on big data technology according to claim 1 or 5, characterized in that: The weights in S3 are also calculated by analyzing user behavior, specifically analyzing the features that users care about based on their house clicking behavior during the house search process.

7. The method for calculating the weight of comprehensive ranking features of house source search based on big data technology according to claim 1 is characterized in that: The house search ranking in S5 needs to adjust the feature weights according to the real-time user behavior data.

Images