A user classification method based on a pre-aggregated storage table

Abstract

The application discloses a user classification method based on a pre-aggregation storage table, relates to the technical field of data processing, and comprises the following steps: S1, data collection; S2, data preprocessing: comprehensively cleaning customer data collected in the step S1, and removing noise data and abnormal values by adopting a 3sigma principle based on a statistical method; S3, feature extraction: deeply extracting features of the customer data preprocessed in the step S2; S4, constructing a pre-aggregation storage table; and S5, data analysis. In the data collection stage, the network crawler technology, the log analysis tool and the third-party data interface are comprehensively used to comprehensively collect customer behavior and basic information data, and the laws, regulations and website protocols are strictly followed to ensure that the data is legal and compliant. Meanwhile, the 3sigma principle, various missing value processing methods, data smoothing and normalization technology are adopted in the data preprocessing to effectively remove noise, fill in missing values, improve data quality and make the classification result truly reflect customer features.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, and in particular to a user classification method based on a pre-aggregation storage table. BACKGROUND

[0002] With the rapid development of the Internet and information technology, various industries have accumulated massive amounts of user data. Enterprises expect to use this data to gain a deep understanding of user behavior, consumption habits, etc., so as to achieve precision marketing, personalized services and product optimization, and improve their competitiveness; in the field of data processing, the explosive growth of data volume has prompted the widespread application of big data analysis technology, while web crawlers, log analysis, third-party data interfaces, etc. have become important means of obtaining data, and clustering analysis, association rule mining, etc. algorithms are also constantly developing, providing technical support for user classification and behavior analysis. In terms of storage, distributed storage technology and various index structures are used to efficiently manage data. Against this background, how to integrate these technologies and build a complete and efficient user classification method has become a research hotspot.

[0003] The prior art has many deficiencies in user classification. In the data collection stage, some data collection methods may violate laws and regulations or website agreements, making the legality of data acquisition questionable, and the data obtained may not be comprehensive and accurate. The data preprocessing link is not fine enough for noise data and outliers, and the missing value processing method is single, which is difficult to adapt to complex data scenarios and affects data quality. There is a lack of effective feature selection algorithm in feature extraction, and the extracted features have high redundancy and cannot accurately reflect the key information of users. In terms of data storage and analysis, the storage structure design is unreasonable, the index technology is inefficient, resulting in slow data query and processing speed. The selection and application of data analysis algorithms are not flexible enough, the clustering results are not accurate, the association rule mining depth is insufficient, and it is difficult to deeply understand user behavior, ultimately resulting in poor user classification effect, which cannot meet the needs of precision marketing and personalized services of enterprises. A user classification method based on a pre-aggregation storage table is needed to solve the above problems. SUMMARY

[0004] The purpose of the present application is to solve the problems in the prior art and provide a user classification method based on a pre-aggregation storage table.

[0005] To achieve the above purpose, the present application is implemented by the following technical solution: a user classification method based on a pre-aggregation storage table, comprising the following classification method:

[0006] S1, data collection: first, a variety of technical means and channels are used, and based on historical data and browsing, customer behavior data and basic information data are comprehensively collected, and through web crawler technology, according to the set rules and strategies, the customer's browsing records are accurately captured from the specified web pages, e-commerce platforms and social media network resources, and detailed records of the customer's access page URL, access time, browsing order and other information are recorded; then, using log analysis tools, the website server log, APP operation log and other information are deeply analyzed to obtain the customer's operation track in the application, including search records, click behavior, dwell time and the like; secondly, through the interface with the third party data, the customer's basic information data is obtained, and through the population statistics data platform, the customer's age, gender, region and other information are obtained, and through the industry data service provider, the customer's occupation information is obtained; finally, the collected customer behavior data includes but is not limited to customer browsing records, purchase records, search records and dwell time, and the customer's basic information data is also collected;

[0007] S2, data preprocessing: the customer data collected in step S1 is comprehensively cleaned, and the 3σ principle based on statistical method is used to identify and remove noise data and outliers, the 3σ principle is based on the normal distribution assumption of data, if the data point is more than 3 times the standard deviation from the mean, it is determined as an outlier, for missing values, different methods are used according to the characteristics of the data, for numerical data, if the data distribution is relatively uniform, the mean filling method is used, if the data has a linear relationship, the regression prediction filling method is used, and for complex data scenarios, multiple filling methods are used to generate multiple reasonable filling values; at the same time, data smoothing technology is used to eliminate small fluctuations in the data, and the data is processed regularly; normalization technology is used to improve data quality, and the data is processed to have zero mean and unit variance;

[0008] S3, feature extraction: the customer data preprocessed in step S2 is deeply extracted, the customer behavior data after preprocessing is deeply extracted, the customer's activity and use time are obtained by statistical analysis of the customer's login times and operation frequency data within a period of time; the purchase frequency is obtained by calculating the number of purchases of the customer within a certain period; the purchase amount is obtained by accumulating the purchase amount of the customer each time; the preferred category is determined by analyzing the category of the goods purchased by the customer; and a feature selection algorithm is used to select features with representative and discriminant degree from the extracted features;

[0009] The feature selection algorithm includes information gain method and chi-square test method, the information gain method calculates the information gain of each feature to the classification target, and selects the feature with high information gain; the chi-square test method is based on the independence assumption between the feature and the classification target, and measures the importance of the feature;

[0010] S4, constructing a pre-aggregation storage table: constructing a pre-aggregation storage table according to the extracted features of the customer data preprocessed in the S3 step, and performing aggregation storage and classification aggregation on the data according to the time dimension and the attribute dimension, while performing aggregation storage on the corresponding data according to the behavior type dimension, using a distributed storage technology to store the pre-aggregation storage table, designing an efficient data index structure for different feature dimensions, using a B-tree index method for range query of the time dimension, and using a hash index method for range query of the attribute and behavior dimensions;

[0011] S5, data analysis: using big data analysis technology to perform in-depth analysis on the customer behavior data based on the pre-aggregation storage table constructed in the S4 step, specifically including the following implementation steps:

[0012] T1, cluster analysis: using a clustering algorithm to cluster customers into different customer groups, using a K-Means algorithm combined with an elbow rule to calculate the clustering error under different K values, and using a silhouette coefficient method to calculate the silhouette coefficient of each sample;

[0013] T2, association rule mining: using the Apriori algorithm and the FP-Growth algorithm to mine the association rules between customer behaviors in different customer groups based on the customer data obtained by the cluster analysis in the T1 step; the Apriori algorithm is based on the a priori principle, generates candidate frequent item sets and verifies them in the data set, and gradually mines frequent item sets and association rules; the FP-Growth algorithm compresses the data storage space and improves the mining efficiency by constructing an FP tree, directly mines frequent item sets from the FP tree, and avoids the large candidate set generation process in the Apriori algorithm;

[0014] T3, customer group analysis: classifying customers into customer groups based on the customer behavior results obtained by the association rule mining in the T2 step, determining the customer group category to which each customer belongs, and combining business experience and domain knowledge to manually review and adjust the classification results.

[0015] Further, the cluster analysis step of T1 specifically includes the following implementation modes:

[0016] When using the K-Means algorithm, the optimal K value is determined by randomly initializing the cluster center multiple times and combining the elbow rule or the silhouette coefficient method, K-Means clustering calculation is performed, and different clustering results are obtained;

[0017] The elbow method draws a curve of the relationship between the clustering error and the K value, observes the trend of the curve, when the K value is small, the clustering error decreases sharply with the increase of K, when the K increases to a certain extent, the clustering error decreases slowly, and when the curve shape is similar to the elbow, the K value corresponding to the elbow is selected as the optimal value;

[0018] The silhouette coefficient method calculates the silhouette coefficient for each sample, and the calculation formula of the silhouette coefficient is: si = (bi - ai) / max(ai, bi), wherein ai represents the average distance between sample i and other samples in the same cluster; bi represents the average distance between sample i and the nearest sample in other clusters; the value range of the silhouette coefficient is [-1, 1], and the closer the value is to 1, the better the sample clustering effect is; different K values are traversed, and the K value when the silhouette coefficient is maximum is selected as the optimal K value.

[0019] Further, the customer group analysis step of T3 includes a customer group feature analysis step: first, the characteristics of each customer group are comprehensively analyzed, and the customer groups are divided according to the distribution proportion of different ages, genders, regions and occupations to understand the basic composition of the customer groups; then, in terms of consumption behavior characteristics, the average purchase amount, purchase frequency and preferred product category of the customer group are analyzed to judge the consumption ability and consumption habit of the customer group; secondly, the interest and hobby characteristics of the customer group are mined by analyzing the product categories and search keywords browsed and purchased by the customer group; finally, the customer portrait is displayed in the form of intuitive charts by using data visualization tools.

[0020] Further, the customer group analysis step of T3 further includes a customer group dynamic updating step: first, new customer behavior data is collected in real time, and the new data is imported into the system through message queue and timing task technical means; then, the pre-aggregated storage table is updated, and an incremental update strategy is adopted to update only the changed data; secondly, the big data analysis and customer group classification are re-performed, and the customer portrait is updated by updating the customer group feature analysis; at the same time, an update threshold is set, and when the amount of new data reaches 3% of the total amount of data in the pre-aggregated storage table, the update process is triggered.

[0021] Further, in the data collection step of S1, customer behavior data and basic information data are collected through web crawler technology, log analysis tools and third-party data interface channels, and when using web crawler technology, relevant laws and regulations and the robots protocol of the website need to be followed to ensure that the data is obtained legally and in compliance; according to different website structures and data formats, corresponding crawler programs are written, and the Scrapy framework of Python is adopted; in terms of log analysis tools, the combination technology of Elasticsearch, Logstash and Kibana is adopted, Logstash is responsible for collecting and preprocessing log data, Elasticsearch is used for storing and retrieving log data, and Kibana provides a visual interface for log analysis; when interfacing with third-party data, a data use agreement is signed to ensure the security and compliance of the data, the interface is called according to the interface document specification to obtain data, and the customer information dimension is enriched.

[0022] Furthermore, in the data preprocessing step S2, the regression prediction imputation method uses the linear relationship between other variables in the data and the variable containing the missing value to establish a regression model for prediction imputation; the multiple imputation method generates multiple complete datasets by randomly generating imputation values ​​multiple times, analyzes each dataset, and combines the analysis results to improve the accuracy and reliability of data processing.

[0023] Furthermore, in the step of constructing the pre-aggregated storage table in S4, each node of the B-tree stores multiple key-value pairs and pointers to child nodes. The storage location and query path of the data are determined by comparing the key values, and the data in the specified time interval is quickly located. The hash index enables fast lookup of the customer's unique identifier by mapping the customer's unique identifier to a fixed-length hash value through a hash function, and the location of the stored data is directly located based on the hash value.

[0024] Furthermore, in the S5 data analysis step, the elastic computing resources provided by the cloud computing platform are used to dynamically adjust the computing resource configuration of the analysis task, thereby improving analysis efficiency and reducing costs. When performing big data analysis tasks, the computing resources are dynamically adjusted according to the task load, and excess computing resources are released in a timely manner when the analysis task is completed or the load decreases.

[0025] In summary, this invention provides a user classification method based on a pre-aggregated storage table, which has the following beneficial effects:

[0026] 1. This invention comprehensively collects customer behavior and basic information data by integrating web crawling technology, log analysis tools, and third-party data interfaces during the data collection phase. It strictly adheres to laws, regulations, and website agreements to ensure data legality and compliance. Furthermore, during data preprocessing, it employs the 3σ principle, various missing value handling methods, and data smoothing and normalization techniques to effectively remove noise, fill in missing values, and improve data quality. This provides an accurate and complete data foundation for subsequent feature extraction and analysis, ensuring the precision of user classification and enabling the classification results to truly reflect customer characteristics, thus providing a reliable basis for enterprise decision-making.

[0027] 2. This invention deeply extracts key features such as activity and purchase frequency from preprocessed data, and uses information gain and chi-square tests to screen valuable features, reducing data redundancy and highlighting key classification information. Simultaneously, when constructing the pre-aggregated storage table, data is aggregated and stored according to multiple dimensions, and distributed storage technology ensures data security and reliability. B-trees and hash indexes are designed for different feature dimensions, enabling rapid data location and querying, greatly improving data processing efficiency, facilitating enterprises to quickly obtain the information they need, enhancing the overall effectiveness of data management and analysis, and providing an efficient solution for large-scale data processing.

[0028] 3. This invention accurately segments customer groups through cluster analysis, utilizing the K-Means algorithm combined with the elbow rule and silhouette coefficient method, providing a basis for precision marketing and personalized services. Association rule mining employs the Apriori and FP-Growth algorithms to discover potential connections in customer behavior, optimize product recommendation strategies, and improve sales conversion rates. Customer group analysis not only comprehensively dissects customer characteristics but also visualizes customer profiles through data visualization. Combining business experience with review and adjustment of classification results enables businesses to gain a deeper understanding of customers, develop targeted operational strategies, and improve operational effectiveness and customer satisfaction.

[0029] 4. This invention establishes a dynamic customer group update mechanism to collect new data in real time and uses an incremental update strategy to update the storage table. When the amount of new data reaches a threshold, the customer group profile is re-analyzed and updated to ensure that the classification results closely follow changes in customer behavior. At the same time, during the data analysis process, the elastic computing resources of the cloud computing platform are used to dynamically adjust the configuration according to the task load. Resources are released in a timely manner when the task is completed or the load decreases to avoid waste, reduce computing costs, enhance system adaptability and economy, and enable enterprises to use resources efficiently and maintain an advantage in market competition. Attached Figure Description

[0030] Figure 1 This is a schematic diagram of the process architecture of a user classification method based on a pre-aggregated storage table according to the present invention. Detailed Implementation

[0031] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0032] Example:

[0033] Please see Figure 1 As shown, the present invention provides a technical solution: a user classification method based on a pre-aggregated storage table, including the following classification methods:

[0034] S1. Data Collection: First, utilizing diverse technologies and channels, and based on historical data and browsing patterns, comprehensively collect customer behavior data and basic information data. Through web crawling technology, and according to established rules and strategies, accurately capture customer browsing records from designated web pages, e-commerce platforms, and social media networks, recording detailed information such as the URLs of pages visited, access times, and browsing order. Then, using log analysis tools, deeply analyze website server logs and APP operation logs to obtain customer's operational trajectory within the application, including search records, click behavior, and dwell time. Next, obtain authoritative customer basic information data by connecting with third-party data interfaces, and obtain customer information such as age, gender, and region through demographic data platforms, and customer occupation information through industry data service providers. Finally, the collected customer behavior data includes, but is not limited to, customer browsing records, purchase records, search records, and dwell time, while also collecting basic customer information data. Collecting data through multiple channels comprehensively obtains customer information, providing rich evidence for accurate classification. Simultaneously, precise capture through web crawling technology and deep log analysis capture customer behavior details, and third-party data integration supplements basic information, improving the accuracy and comprehensiveness of classification.

[0035] S2. Data Preprocessing: The customer data collected in step S1 undergoes comprehensive cleaning. The 3σ principle, based on statistical methods, is used to identify and remove noisy data and outliers. The 3σ principle is based on the assumption of a normal distribution; if a data point deviates from the mean by more than three standard deviations, it is considered an outlier. For missing values, different methods are used depending on the data characteristics. For numerical data, mean imputation is used if the data distribution is relatively uniform; regression prediction imputation is used if a linear relationship exists; and multiple imputation methods are used to generate multiple reasonable imputation values ​​for complex data scenarios. Simultaneously, data smoothing techniques are used to eliminate minor fluctuations in the data and process the data for regularity. Normalization techniques are used to improve data quality by applying zero mean and unit variance. Data cleaning and processing improve data quality, remove noise and outliers to avoid interfering with classification results, and ensure data integrity by using appropriate missing value handling methods for different data situations. Normalization makes the data more comparable and facilitates subsequent analysis.

[0036] S3. Feature Extraction: Deep feature extraction is performed on the preprocessed customer data from step S2. Features are extracted from the preprocessed customer behavior data. Statistical analysis of customer login frequency and operation frequency over a period of time yields customer activity and usage time. Purchase frequency is calculated by counting the number of purchases within a certain period. Purchase amount is calculated by summing the purchase amounts of each customer. Preferred product categories are determined by analyzing the categories of the products purchased. Finally, a feature selection algorithm is used to select representative and distinctive features from the extracted features. Deep feature extraction accurately depicts customer behavior and consumption characteristics, while feature selection algorithms identify key features, reducing data redundancy and highlighting information valuable for user classification, making the classification more targeted and effective.

[0037] Feature selection algorithms include the information gain method and the chi-square test. The information gain method calculates the information gain of each feature to the classification target and selects features with high information gain. The chi-square test is based on the assumption of independence between features and classification target and measures the importance of features.

[0038] S4. Constructing a Pre-aggregated Storage Table: The pre-processed customer data from step S3 is used to construct a pre-aggregated storage table based on extracted features. The data is then aggregated and categorized according to time and attribute dimensions, and also aggregated according to behavioral type. Distributed storage technology is used to store the pre-aggregated storage table. Efficient data index structures are designed for different feature dimensions. A B-tree index is used for range queries in the time dimension, while a hash index is used for range queries in the attribute and behavioral dimensions. Constructing a pre-aggregated storage table facilitates data management and analysis, multi-dimensional aggregation storage improves data query and processing efficiency, distributed storage ensures data security and reliability, and different indexing methods meet various query needs, thus improving system performance.

[0039] S5. Data Analysis: Utilizing big data analytics techniques, conduct in-depth analysis of customer behavior data based on the pre-aggregated storage table built in step S4. This includes the following implementation steps:

[0040] T1. Cluster Analysis: Clustering algorithms are used to cluster customers into different customer groups. The K-Means algorithm, combined with the elbow rule, is used to calculate the clustering error under different K values. The silhouette coefficient method is used to calculate the silhouette coefficient of each sample. Cluster analysis can segment customers. The K-Means algorithm combined with the elbow rule and silhouette coefficient method can determine the optimal number of clusters, accurately segment customer groups, provide a basis for precision marketing and personalized services, and improve marketing effectiveness and user satisfaction.

[0041] T2. Association Rule Mining: Using the customer data from the clustering analysis in step T1, the Apriori and FP-Growth algorithms are used to mine association rules between customer behaviors within different customer groups. The Apriori algorithm, based on prior principles, generates candidate frequent itemsets and validates them in the dataset to gradually mine frequent itemsets and association rules. The FP-Growth algorithm constructs an FP tree, compressing data storage space and improving mining efficiency. It directly mines frequent itemsets from the FP tree, avoiding the extensive candidate set generation process of the Apriori algorithm. Mining association rules can uncover potential connections between customer behaviors, reveal customer buying habits and preference combinations, and allow businesses to optimize product recommendations and marketing strategies, thereby improving sales conversion rates and customer loyalty.

[0042] T3. Customer Group Analysis: Based on the customer behavior results mined from the association rules in step T2, customers are categorized into customer groups to determine the category to which each customer belongs. At the same time, the categorization results are manually reviewed and adjusted in conjunction with business experience and domain knowledge. The customer group classification clarifies the customer affiliation, and the review and adjustment based on business experience and domain knowledge ensures that the categorization results meet the actual business needs, enabling enterprises to develop precise operational strategies for different customer groups and improve operational effectiveness.

[0043] The clustering analysis steps of T1 specifically include the following implementation methods:

[0044] When using the K-Means algorithm, the cluster centers are initialized randomly multiple times, and the optimal K value is determined by combining the elbow rule or the silhouette coefficient method. K-Means clustering calculation is then performed to obtain different clustering results.

[0045] The elbow method involves plotting a curve showing the relationship between clustering error and the K value, and observing the trend of the curve. When the K value is small, the clustering error decreases sharply as K increases. When K increases to a certain extent, the decreasing trend of clustering error slows down. When the curve shape resembles an elbow, the K value corresponding to the elbow is selected as the optimal value.

[0046] The silhouette coefficient method calculates the silhouette coefficient for each sample. The formula for calculating the silhouette coefficient is: si = (bi - ai) / max(ai, bi), where ai represents the average distance between sample i and other samples in the same cluster; bi represents the average distance between sample i and the nearest sample in other clusters. The silhouette coefficient ranges from [-1, 1]. The closer the value is to 1, the better the clustering effect. By iterating through different K values ​​and selecting the K value with the largest silhouette coefficient as the optimal K value, the method of randomly initializing cluster centers multiple times and selecting the optimal K value can avoid the clustering results from getting trapped in local optima, improve the accuracy and stability of clustering, more accurately segment customer groups, and help enterprises carry out targeted marketing activities more effectively.

[0047] T3's customer analysis steps include customer characteristic analysis: First, a comprehensive analysis of the characteristics of each customer group is conducted, dividing the customer group according to the distribution ratio of different ages, genders, regions, and occupations to understand the basic composition of the customer group; then, in terms of consumption behavior characteristics, the average purchase amount, purchase frequency, and preferred purchase categories of the customer group are analyzed to determine the customer group's spending power and consumption habits; secondly, by analyzing the product categories and search keywords browsed and purchased by the customer group, the interests and hobbies of the customer group are explored; finally, data visualization tools are used to display the customer profile in an intuitive chart format. Comprehensive customer characteristic analysis helps companies gain a deeper understanding of the characteristics of different customer groups, and visualization makes the analysis results more intuitive and easy to understand, facilitating companies to formulate product strategies and marketing plans that meet the needs of customer groups and enhance market competitiveness.

[0048] T3's customer segment analysis process also includes a dynamic customer segment update step: First, new customer behavior data is collected in real time and imported into the system through message queues and scheduled tasks; then, the pre-aggregated storage table is updated using an incremental update strategy, updating only the changed data; next, big data analysis and customer segment classification are performed again, and customer segment feature analysis is conducted to update the customer profile; at the same time, an update threshold is set, and when the amount of new data reaches 3% of the total data in the pre-aggregated storage table, the update process is triggered. Dynamic customer segment updates ensure that the classification results keep up with changes in customer behavior, promptly reflecting new customer needs and trends. Incremental updates save resources, and setting update thresholds balances update costs and timeliness, enabling enterprises to continuously provide accurate services.

[0049] In S1's data collection process, customer behavior and basic information data are collected through web crawling, log analysis tools, and third-party data interfaces. When using web crawling, relevant laws and regulations and the website's robots.txt protocol must be followed to ensure legal and compliant data acquisition. Corresponding crawler programs are written based on different website structures and data formats, using the Python Scrapy framework. For log analysis tools, a combination of Elasticsearch, Logstash, and Kibana technologies is used. Logstash is responsible for collecting and preprocessing log data, Elasticsearch is used for storing and retrieving log data, and Kibana provides a visual interface for log analysis. When connecting with third-party data interfaces, data usage agreements are signed to ensure data security and compliance. Interfaces are called according to the interface documentation specifications to obtain data, enriching customer information dimensions. Legal and compliant data collection methods ensure the security of enterprise operations. The use of different technical tools efficiently acquires multi-source data, enriching customer information dimensions and laying a solid foundation for building accurate user classification models.

[0050] In the data preprocessing steps of S2, the regression prediction imputation method uses the linear relationship between other variables in the data and the variable containing the missing value to establish a regression model for prediction imputation; the multiple imputation method generates multiple complete datasets by randomly generating imputed values ​​multiple times, and analyzes each dataset. The comprehensive analysis results improve the accuracy and reliability of data processing. By effectively handling the problem of missing data through regression prediction and multiple imputation, the data integrity and accuracy are improved, providing high-quality data for subsequent feature extraction, model building and analysis, and ensuring the reliability of user classification results.

[0051] In the S4 pre-aggregated storage table construction step, each node of the B-tree stores multiple key-value pairs and pointers to child nodes. By comparing key values, the storage location and query path of the data are determined, quickly locating data within a specified time interval. The hash index enables fast lookup of customer unique identifiers. A hash function maps the customer unique identifier to a fixed-length hash value, directly locating the stored data based on the hash value. The design of B-trees and hash indexes greatly improves data query efficiency, quickly locating data and saving time and resources. This facilitates enterprises in quickly obtaining the required customer information, improves data processing and analysis speed, and optimizes business responsiveness.

[0052] In the S5 data analysis process, the elastic computing resources provided by the cloud computing platform are used to dynamically adjust the computing resource configuration of the analysis tasks, thereby improving analysis efficiency and reducing costs. When performing big data analysis tasks, computing resources are dynamically adjusted according to the task load. When the analysis task is completed or the load decreases, excess computing resources are released in a timely manner. By utilizing the elastic computing resources of cloud computing, resources can be flexibly allocated according to task requirements, improving analysis efficiency, reducing computing costs, avoiding resource waste, enhancing the adaptability and economy of the system, and improving the resource utilization efficiency of enterprises.

[0053] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any other way. Any person skilled in the art may make changes or modifications to the above-disclosed technical content to create equivalent embodiments for application in other fields. However, any simple modifications, equivalent changes, and modifications made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the protection scope of the present invention.

Claims

1. A user classification method based on a pre-aggregated storage table, characterized in that: The following classification methods are included: S1. Data Collection: First, we use a variety of technical means and channels, as well as historical data and browsing history, to comprehensively collect customer behavior data and basic information data. Then, through web crawling technology, we accurately capture customer browsing records from designated web pages, e-commerce platforms and social media network resources according to the set rules and strategies, and record in detail the page URLs visited by customers, access time and browsing order information. S2. Data Preprocessing: The customer data collected in step S1 is thoroughly cleaned. The 3σ principle based on statistical methods is used to identify and remove noisy data and outliers. The 3σ principle is based on the assumption of normal distribution of data. If a data point is more than 3 times the standard deviation away from the mean, it is judged as an outlier. S3. Feature Extraction: Deeply extract features from the preprocessed customer data in step S2. Deeply extract features from the preprocessed customer behavior data. By statistically analyzing the number of times a customer logs in and the frequency of operations over a period of time, obtain the customer's activity level and usage time. Calculate the number of times a customer makes a purchase within a period to obtain the purchase frequency. The total purchase amount is calculated by summing up each customer's purchase amount; the preferred product categories are determined by analyzing the categories of the products purchased by the customer; and a feature selection algorithm is used to select representative and distinctive features from the extracted features. S4. Construct a pre-aggregated storage table: Take the customer data preprocessed in step S3, construct a pre-aggregated storage table based on the extracted features, and aggregate and classify the data according to the time dimension and attribute dimension. At the same time, aggregate and store the corresponding data according to the behavior type dimension. Use distributed storage technology to store the pre-aggregated storage table, design an efficient data index structure for different feature dimensions, and use the B-tree index method for range queries in the time dimension. Hash indexing is used for range queries of attributes and behaviors. Each node of a B-tree stores multiple key-value pairs and pointers to child nodes. By comparing key values, the storage location and query path of the data are determined, allowing for quick location of data within a specified time interval. Hash indexes enable fast lookup of customer unique identifiers by mapping the customer unique identifier to a fixed-length hash value using a hash function, which directly locates the stored data based on the hash value. S5. Data Analysis: Utilizing big data analytics techniques, conduct in-depth analysis of customer behavior data based on the pre-aggregated storage table built in step S4. This includes the following implementation steps: T1. Cluster Analysis: Clustering algorithms are used to cluster customers into different customer groups. The K-Means algorithm is used in conjunction with the elbow rule to calculate the clustering error under different K values. The silhouette coefficient method is used to calculate the silhouette coefficient of each sample. T2. Association Rule Mining: Using the customer data from the clustering analysis in step T1, the Apriori and FP-Growth algorithms are used to mine association rules between customer behaviors within different customer groups. The Apriori algorithm, based on prior principles, generates candidate frequent itemsets and validates them in the dataset to gradually mine frequent itemsets and association rules. The FP-Growth algorithm constructs an FP tree, compressing data storage space and improving mining efficiency, directly mining frequent itemsets from the FP tree, avoiding the large number of candidate set generation processes in the Apriori algorithm. T3. Customer Group Analysis: Based on the customer behavior results mined from the association rules in step T2, customers are classified into customer groups to determine the customer group category to which each customer belongs. At the same time, the classification results are manually reviewed and adjusted in combination with business experience and domain knowledge. The clustering analysis step of T1 specifically includes the following implementation methods: When using the K-Means algorithm, the cluster centers are initialized randomly multiple times, and the optimal K value is determined by combining the elbow rule or the silhouette coefficient method. K-Means clustering calculation is then performed to obtain different clustering results. The elbow method involves plotting a curve showing the relationship between clustering error and the K value, and observing the trend of the curve. When the K value is small, the clustering error decreases sharply as the K value increases. When the increase in the K value deepens, the downward trend of the clustering error slows down. When the curve is shaped like an elbow, the K value corresponding to the elbow is selected as the optimal value. The silhouette coefficient method calculates the silhouette coefficient for each sample. The formula for calculating the silhouette coefficient is: si=(bi-ai) / max(ai,bi), where ai represents the average distance between sample i and other samples in the same cluster; bi represents the average distance between sample i and the nearest sample in other clusters; the silhouette coefficient ranges from [-1,1]. The closer the value is to 1, the better the sample clustering effect. Different K values ​​are iterated, and the K value with the largest silhouette coefficient is selected as the optimal K value.

2. The user classification method based on a pre-aggregated storage table according to claim 1, characterized in that: In step S1, log analysis tools are used to perform in-depth analysis of website server logs and APP operation logs to obtain the customer's operation trajectory within the application, including search records, click behavior, and dwell time. Secondly, authoritative basic customer information data is obtained by connecting with third-party data interfaces, and customer age, gender, and regional information is obtained through a demographic data platform, and customer occupation information is obtained through industry data service providers. Finally, the collected customer behavior data includes, but is not limited to, customer browsing history, purchase history, search history, and dwell time, while also collecting basic customer information data. In step S2, for missing values, different methods are used to process them according to the characteristics of the data. For numerical data, if the data distribution is relatively uniform, the mean imputation method is used; if the data has a linear relationship, the regression prediction imputation method is used; and for complex data scenarios, the multiple imputation method generates multiple reasonable imputation values. At the same time, data smoothing technology is used to eliminate small fluctuations in the data and to process the data in a regular way. Normalization techniques are used to improve data quality by processing the data to zero mean and unit variance.

3. The user classification method based on a pre-aggregated storage table according to claim 1, characterized in that: The customer group analysis steps of T3 include customer group characteristic analysis steps: First, a comprehensive analysis of the characteristics of each customer group is conducted, dividing the customer group according to the distribution ratio of different ages, genders, regions, and occupations to understand the basic composition of the customer group; then, in terms of consumption behavior characteristics, the average purchase amount, purchase frequency, and preferred purchase categories of the customer group are analyzed to determine the customer group's consumption power and consumption habits; secondly, by analyzing the product categories and search keywords browsed and purchased by the customer group, the interests and hobbies of the customer group are explored; finally, data visualization tools are used to display the customer group profile in an intuitive chart form.

4. The user classification method based on a pre-aggregated storage table according to claim 3, characterized in that: The customer group analysis steps of T3 also include a dynamic customer group update step: First, new customer behavior data is collected in real time and imported into the system through message queues and scheduled tasks; then, the pre-aggregated storage table is updated using an incremental update strategy, updating only the changed data; second, big data analysis and customer group classification are performed again, and customer group feature analysis is conducted to update the customer profile; at the same time, an update threshold is set, and the update process is triggered when the amount of new data reaches 3% of the total amount of data in the pre-aggregated storage table.

5. The user classification method based on a pre-aggregated storage table according to claim 1, characterized in that: In the data collection step S1, customer behavior data and basic information data are collected through web crawling technology, log analysis tools, and third-party data interface channels. When using web crawling technology, relevant laws and regulations and the website's robots.txt protocol must be followed to ensure legal and compliant data acquisition. Corresponding crawler programs are written according to different website structures and data formats, using the Python Scrapy framework. For log analysis tools, a combination of Elasticsearch, Logstash, and Kibana technologies is used. Logstash is responsible for collecting and preprocessing log data, Elasticsearch is used for storing and retrieving log data, and Kibana provides a visualization interface for log analysis. When connecting with third-party data interfaces, data usage agreements are signed to ensure data security and compliance. Interfaces are called to obtain data according to the interface documentation specifications to enrich the dimensions of customer information.

6. The user classification method based on a pre-aggregated storage table according to claim 1, characterized in that: In the data preprocessing step S2, the regression prediction imputation method uses the linear relationship between other variables in the data and the variable containing the missing value to establish a regression model for prediction imputation. Multiple imputation methods generate multiple complete datasets by randomly generating imputation values ​​multiple times. Each dataset is then analyzed, and the combined analysis results improve the accuracy and reliability of data processing.

7. The user classification method based on a pre-aggregated storage table according to claim 1, characterized in that: In step S3, the feature selection algorithm includes the information gain method and the chi-square test. The information gain method calculates the information gain of each feature to the classification target and selects features with high information gain. The chi-square test is based on the assumption of independence between features and classification target and measures the importance of features.

8. The user classification method based on a pre-aggregated storage table according to claim 1, characterized in that: In the S5 data analysis step, the elastic computing resources provided by the cloud computing platform are used to dynamically adjust the computing resource configuration of the analysis task, thereby improving analysis efficiency and reducing costs. When performing big data analysis tasks, the computing resources are dynamically adjusted according to the task load, and excess computing resources are released in a timely manner when the analysis task is completed or the load decreases.

Images