Recommendation algorithm design based on big data in e-commerce platform

Abstract

The invention belongs to the technical field of e-commerce, and particularly relates to a big data-based recommendation algorithm design in an e-commerce platform, which comprises the following steps: completing data preprocessing based on data collection and preprocessing; constructing user portraits to generate personalized user portraits, and classifying the users by using a clustering algorithm to find group features of the users; analyzing the user similarity and the commodity similarity, and calculating the similarity between the users; carrying out user-based collaborative filtering and article-based collaborative filtering, and carrying out dimensionality reduction processing on user and commodity features by using an SVD or latent semantic model; modeling user behaviors by using a neural network, and performing data compression and feature extraction by using an auto-encoder; the user feedback data is collected for continuous optimization and iteration of the model, and online learning and A / B test are adopted to adjust the recommendation strategy, so that the method has the effects of realizing more accurate and personalized commodity recommendation, and improving the shopping experience of the user and the sales conversion rate of the platform.

Description

Technical Field

[0001] The present invention relates to the technical field of e-commerce, and in particular to a recommendation algorithm design based on big data in an e-commerce platform. Background Art

[0002] Current situation analysis: Product recommendations on traditional e-commerce platforms usually rely on collaborative filtering, content-based recommendations and other strategies, and make recommendations based on user browsing history, purchase history or product features. However, these methods are limited by data scarcity, cold start problems, and insufficient accuracy, making it difficult to accurately meet user personalized needs and improve user experience. In the data-driven era, the massive amount of data that is exploding with information provides valuable resources for accurate recommendations.

[0003] Existing technology is insufficient. Although the existing recommendation algorithms based on big data have made breakthrough progress, there are still some challenges: 1) The analysis of user behavior characteristics is not accurate enough to explore the potential needs of users; 2) The integration of multi-dimensional characteristics of products is not sufficient, and it is difficult to capture the correlation between products; 3) There is a lack of dynamic learning and updating mechanism, which makes it difficult to adapt to changes in user needs, resulting in invalid recommendation results;

[0004] In view of the above technical defects, a recommendation algorithm design solution based on big data in e-commerce platforms is proposed. Summary of the invention

[0005] To solve the above problems, the present invention provides the following technical solutions:

[0006] A recommendation algorithm design based on big data in an e-commerce platform, including:

[0007] Based on data collection and preprocessing, user data, product data, and behavior data are collected and stored in a database, and the data in the database is cleaned and converted to complete data preprocessing;

[0008] Construct user portraits, analyze user interests and preferences based on user behavior data, generate personalized user portraits, and use clustering algorithms to classify users to discover user group characteristics;

[0009] Through similarity calculation, the user similarity and product similarity are analyzed, the similarity between users is calculated, and the similarity between products is calculated through content similarity or collaborative filtering;

[0010] Build analysis models, user-based collaborative filtering and item-based collaborative filtering, and use SVD or latent semantic models to reduce the dimensionality of user and item features;

[0011] Based on deep learning methods, neural networks are used to model user behavior, and autoencoders are used for data compression and feature extraction;

[0012] Through feedback mechanisms and model optimization, user feedback data is collected for continuous optimization and iteration of the model, and online learning and A / B testing are used to adjust the recommendation strategy.

[0013] Furthermore, the data collection and preprocessing include collecting users' browsing history, click history, shopping cart addition, purchase history, search history, users' age, gender, geographic location, user preferences and brands they follow from social media, blogger information, users' comments, ratings, feedback information on products, real-time inventory data and current discount activity data, processing missing data by deletion, filling or interpolation methods, using statistical methods or machine learning algorithms to detect and filter abnormal data, converting data to a unified scale, such as normalizing numerical features to improve the efficiency of the algorithm, converting categorical data to numerical data, using PCA or other dimensionality reduction techniques to simplify data features, reduce dimensions, and improve computing efficiency, extracting useful features from raw data, such as extracting key attributes of products, selecting the features that are most helpful for recommendation effects through correlation analysis or model selection, and integrating data from different sources to form a unified user-product interaction data set.

[0014] Furthermore, the data processing includes collecting real-time data of users and establishing a database, cleaning the data in the database, and cleaning the data sets in the database, including carrying a data cleaning function, as follows:

[0015]

[0016] Among them, Xi′ is the standardization of the current data after sorting, Xi is the standardization of the current data, Xmax is the maximum standardization of the current data, and Xmin is the minimum standardization of the current data.

[0017] Further, the data processing includes obtaining each pair of adjacent user data in each user data subsequence and the acquisition time of the pair of adjacent user data;

[0018] The data slope ratio of the pair of adjacent user data in each user data subsequence is determined according to each pair of adjacent user data in each user data subsequence and the acquisition time of the pair of adjacent user data.

[0019] In specific implementation, the data slope ratio can be determined by the following formula:

[0020]

[0021] in, represents the data slope ratio of the qth pair of adjacent user data in the rth user data subsequence, represents the right adjacent user data value of the qth pair of adjacent user data in the rth user data subsequence, represents the left adjacent user data value of the qth pair of adjacent user data in the rth user data subsequence, + represents the symbol of the right adjacent user data value of the adjacent user data, - represents the symbol of the left adjacent user data value of the adjacent user data, represents the acquisition time of the right adjacent user data value of the qth pair of adjacent user data in the rth user data subsequence, represents the acquisition time of the left adjacent user data value of the qth pair of adjacent user data in the rth user data subsequence.

[0022] Furthermore, the user portrait construction includes collecting information from the user's daily behavior, including browsing history, clicking on products, adding to shopping carts, purchase history, favorites, search history, user comments, product ratings, interactions with other users, including the type of device used by the user, access time, geographic location, and network type; extracting the product categories or brand preferences that the user is interested in based on the user's long-term behavior; understanding the timing of the user's purchase decision based on the user's behavior in different time periods; adjusting the recommendation strategy based on the user's current situation; using an unsupervised learning algorithm to group users with similar purchasing habits and browsing behaviors into one category; analyzing the user's interests and preferences, and constructing the user's interest portrait; inferring the user's current interests based on the user's recent browsing and interactive behaviors; extracting the user's stable preferences based on the user's long-term behavior patterns; analyzing the content of the products browsed or purchased by the user; extracting the key features of the products through natural language processing technology; establishing a model related to the user's interests; converting the user's behavior data and interests and hobbies into a vector representation for calculation; converting user features into high-dimensional vectors and using matrix decomposition and other methods to analyze the matching degree between the user and the product; and continuously optimizing the user portrait through user behavior feedback.

[0023] Furthermore, the similarity calculation includes user similarity, which is mainly used in user-based collaborative filtering algorithms to recommend products that are liked by similar users based on user behavior and preferences. Based on the user's behavior vector, the similarity is measured by calculating the cosine angle between two vectors. Product similarity is used in item-based collaborative filtering algorithms or content-based recommendation algorithms to help the platform recommend other products similar to the currently browsed products to users. Based on the attribute characteristics of the products, product feature vectors are constructed and cosine similarity is calculated. Based on the numerical characteristics of the products, the similarity between products is calculated using Euclidean distance. For the text description of the products, TF-IDF, word embedding or BERT natural language processing technology is used to calculate the similarity of product texts. For products involving visual content, convolutional neural networks are used to extract features from product images, and image similarity is calculated after extracting visual feature vectors through deep learning models. In large-scale e-commerce platforms, user similarity matrices or product similarity matrices are pre-calculated and stored: each row and column of the user similarity matrix corresponds to a different user, and the values ​​in the matrix represent the similarity between two users. Each row and column of the product similarity matrix corresponds to a different product, and the values ​​in the matrix represent the similarity between two products.

[0024] Furthermore, the cosine similarity calculation formula is as follows:

[0025]

[0026] Among them, u is the user's click data on the product, v is the user's purchase data on the product, ui is the number of times the user clicks on product i, vi is the number of times the user purchases product i, and n represents the product sequence.

[0027] Furthermore, based on the numerical features of the products, the Euclidean distance is used to calculate the similarity between the products. The formula is as follows:

[0028]

[0029] Among them, x is the name of product x, y is the name of product y, j is the common feature of product x and product y, and n is the feature type of the product.

[0030] Furthermore, the construction of the analysis model includes defining business goals to provide personalized recommendations for users, improve sales conversion rates, group user groups by behavior and interests for more accurate marketing, predict sales for a certain period of time in the future, optimize inventory management and pricing strategies, understand market trends and user preferences, provide data support for product development, conduct preliminary exploration of data through visualization tools, analyze user behavior characteristics and product sales, use descriptive statistics and distribution analysis to understand the characteristics of the data, check the correlation between features, identify key features that affect the target variable, prepare for subsequent feature engineering and modeling, use methods such as Pearson correlation coefficient, mutual information method or model-based importance ranking to select the features that have the greatest impact on the target variable, extract new features from existing data, encode categorical data, normalize or standardize numerical features, and collaborative filtering-based models, including user-based collaborative filtering and product-based collaborative filtering. Collaborative filtering, through neural collaborative filtering and Wide&Deep model, combines deep learning with collaborative filtering to improve recommendation accuracy, divides the data set into training set and validation set to ensure the training effect and generalization ability of the model, evaluates model performance through K-fold cross validation to prevent model overfitting, uses grid search or random search methods to adjust the model's hyperparameters, and finds the optimal parameter combination. In the real-time recommendation system, the model is deployed to the production environment to ensure that user behavior data can be updated in real time and recommendation results can be generated dynamically. Use A / B testing to verify the effect of the new model, compare the performance of the new and old models, and ensure that the new model effectively improves the indicators. According to user feedback and new data collected by the system, continuously iterate and optimize the model, establish a monitoring system, regularly monitor the performance of the model, and ensure that the model has no performance degradation. As user behavior and market environment change, the model is regularly retrained and updated to maintain the effectiveness and accuracy of the model.

[0031] Furthermore, the feedback mechanism and model optimization include feedback directly provided by users, through product ratings, comments, likes / dislikes or through user behavior data, based on clicks, browsing time, adding to cart, bounce rate, when the model is launched, using A / B testing to randomly divide users into two groups, using the old model and the new model respectively, and compare their respective performances;

[0032] Continuously adjust recommendation strategies or advertising strategies through experimental design to find the optimal parameter configuration, set up user satisfaction surveys to obtain more direct feedback, identify potential problems in the model through real-time monitoring of user behavior, recommend irrelevant products or algorithm deviations during cold start, and store users' real-time feedback data and purchase conversion information in the data warehouse for subsequent analysis;

[0033] Clean the collected feedback data, including deleting abnormal behaviors, filling missing values, and removing noise data, to ensure that the feedback data can correctly reflect the user's true intentions. Generate more labels for the model based on the user's feedback data. For real-time recommendation systems, use implicit feedback to dynamically adjust the user's short-term interest model. By processing the real-time data stream, train new user portraits or interest models.

[0034] For non-real-time models, feedback data is used to retrain the model regularly to ensure that the model adapts to the new behavior patterns of users. Online learning algorithms are used to enable the model to make instant adjustments based on the latest feedback. Parameters are tuned based on feedback data. Based on the similarity threshold or the number of recommended products in the collaborative filtering model, explicit feedback and implicit feedback are combined to optimize according to the weight of the feedback. Multiple recommendation models are combined to form a hybrid recommendation system. The weight of each sub-model is dynamically adjusted through the feedback mechanism to improve the accuracy of the recommendation results.

[0035] Monitor changes in user behavior in real time, track the performance of model recommendation results, and continuously monitor the long-term effects of the model, especially the performance of the model on cold start and long-tail users, to avoid the recommendation of products always being concentrated on a few popular products. Use feedback data to detect whether the model is biased or unfair, detect changes in data distribution, and promptly discover inconsistencies between the model and actual data distribution to avoid model effect degradation;

[0036] We conduct continuous optimization based on feedback data, and constantly improve model structure, feature selection, and algorithm strategies based on user feedback. We use feedback mechanisms to improve personalization, and adjust recommendation algorithms based on user click and purchase history to make recommendations more in line with individual user interests rather than overly dependent on general preferences. We discover new features based on feedback data, and extract features related to user interests through the time users spend on specific products, helping the model further refine user portraits.

[0037] Compared with the prior art, the present invention has the following beneficial effects:

[0038] The invention discloses a recommendation algorithm design based on big data in an e-commerce platform. The design collects user data, product data and behavior data, establishes a database for storage, cleans and converts the data in the database to complete data preprocessing; constructs a user portrait, analyzes the user's interests and preferences based on the user behavior data, generates a personalized user portrait, and classifies the users by using a clustering algorithm to discover the group characteristics of the users; analyzes the user similarity and product similarity, calculates the similarity between users, and calculates the similarity between products by content similarity or collaborative filtering; performs dimensionality reduction processing of user and product features by using SVD or a latent semantic model based on user-based collaborative filtering and item-based collaborative filtering; uses a neural network to model user behavior, and uses an autoencoder to perform data compression and feature extraction; collects user feedback data for continuous optimization and iteration of the model, and uses online learning and A / B testing to adjust the recommendation strategy, which has the effect of achieving more accurate and personalized product recommendations, improving the user's shopping experience and the sales conversion rate of the platform. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to facilitate understanding by those skilled in the art, the present invention is further described below in conjunction with the accompanying drawings;

[0040] Figure 1 It is an overall schematic diagram of a recommendation algorithm designed based on big data in an e-commerce platform of the present invention;

[0041] Figure 2 This is a structural diagram of a computer device in the design of a recommendation algorithm based on big data in an e-commerce platform of the present invention. DETAILED DESCRIPTION

[0042] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0043] like Figure 1-2 As shown in the figure, a recommendation algorithm design based on big data in an e-commerce platform includes:

[0044] Based on data collection and preprocessing, user data, product data, and behavior data are collected and stored in a database, and the data in the database is cleaned and converted to complete data preprocessing;

[0045] Construct user portraits, analyze user interests and preferences based on user behavior data, generate personalized user portraits, and use clustering algorithms to classify users to discover user group characteristics;

[0046] Through similarity calculation, the user similarity and product similarity are analyzed, the similarity between users is calculated, and the similarity between products is calculated through content similarity or collaborative filtering;

[0047] Build analysis models, user-based collaborative filtering and item-based collaborative filtering, and use SVD or latent semantic models to reduce the dimensionality of user and item features;

[0048] Based on deep learning methods, neural networks are used to model user behavior, and autoencoders are used for data compression and feature extraction;

[0049] Through feedback mechanisms and model optimization, user feedback data is collected for continuous optimization and iteration of the model, and online learning and A / B testing are used to adjust the recommendation strategy. Recommended content is adjusted based on the user's current contextual information (such as location, time, and device type). The user's current behavior is analyzed in real time, and related products are dynamically recommended. A variety of recommendation strategies (such as collaborative filtering, content-based, and knowledge-based recommendations) are combined to make up for the shortcomings of a single model. Personalized recommendation combinations are created for users to improve the accuracy and diversity of recommendations. Ensure that the collected and stored data meets relevant privacy protection regulations and security standards, and use anonymization and encryption technologies to protect user data. Through these technical concepts, e-commerce platforms can design efficient, accurate, and personalized recommendation systems to improve user experience and commercial benefits.

[0050] Specifically, the data collection and preprocessing includes collecting the user's browsing history, click history, shopping cart addition, purchase history, search history, user age, gender, geographic location, user preferences and brands from social media, blogger information, user comments, ratings, feedback information on products, real-time inventory data and current discount activity data, processing missing data by deletion, filling or interpolation methods, using statistical methods or machine learning algorithms to detect and filter abnormal data, converting data to a unified scale, such as normalizing numerical features to improve the efficiency of the algorithm, converting categorical data to numerical data, using PCA or other dimensionality reduction techniques to simplify data features, reduce dimensions, and improve computational efficiency, extracting useful features from raw data, such as extracting key attributes of products, selecting the most helpful features for recommendation effects through correlation analysis or model selection, and integrating data from different sources to form a unified user-product interaction data set. Through the above process, the preprocessed data will have higher quality and consistency, providing a solid foundation for subsequent recommendation algorithms. Good data collection and preprocessing are the prerequisites for optimizing the performance of recommendation algorithms.

[0051] Specifically, the data processing includes collecting real-time data of users and establishing a database, cleaning the data in the database, and cleaning the data sets in the database, including carrying a data cleaning function, as follows:

[0052]

[0053] Among them, Xi′ is the standardization of the current data after sorting, Xi is the standardization of the current data, Xmax is the maximum standardization of the current data, and Xmin is the minimum standardization of the current data.

[0054] Specifically, the data processing includes obtaining each pair of adjacent user data in each user data subsequence and the acquisition time of the pair of adjacent user data;

[0055] The data slope ratio of the pair of adjacent user data in each user data subsequence is determined according to each pair of adjacent user data in each user data subsequence and the acquisition time of the pair of adjacent user data.

[0056] In specific implementation, the data slope ratio can be determined by the following formula:

[0057]

[0058] in, represents the data slope ratio of the qth pair of adjacent user data in the rth user data subsequence, represents the right adjacent user data value of the qth pair of adjacent user data in the rth user data subsequence, represents the left adjacent user data value of the qth pair of adjacent user data in the rth user data subsequence, + represents the symbol of the right adjacent user data value of the adjacent user data, - represents the symbol of the left adjacent user data value of the adjacent user data, represents the acquisition time of the right adjacent user data value of the qth pair of adjacent user data in the rth user data subsequence, represents the acquisition time of the left adjacent user data value of the qth pair of adjacent user data in the rth user data subsequence.

[0059] It should be noted that the above-mentioned data slope ratio refers to an indicator used to compare the change rate of each pair of user data points. In user data analysis, the data slope ratio can be used to compare the trend changes of data at different time points. Specifically, the data slope ratio calculates the degree of change between two adjacent data points to determine the data change trend. By calculating the data slope ratio, the trend changes between adjacent data points in the user data subsequence can be more clearly understood to identify the direction and change speed of the data trend. For example, if the data slope ratio is positive for several consecutive time points, it means that the user data in the user data subsequence shows an upward trend, which facilitates subsequent data processing. It should also be noted that in the embodiment of the present application, the data slope ratio can also be used to help identify potential anomalies in the data to improve the accuracy of identifying contaminated data values.

[0060] The volatility of each user data subsequence is measured according to all data slope ratios of the user data subsequence to obtain the sequence volatility of the user data subsequence, and the sequence volatility of each user data subsequence is compared with the preset volatility. When the sequence volatility is greater than the preset volatility, the user data subsequence is determined to be a user data contaminated subsequence.

[0061] In some embodiments, the volatility of each user data subsequence is measured according to the slope ratio of all data of the user data subsequence, and the sequence volatility of the user data subsequence is obtained in the following manner, namely:

[0062] Determine the data slope concentration trend of each user data subsequence;

[0063] Determine a discrete value of a data slope of each user data subsequence;

[0064] Obtain the random volatility factor of the volatility measure of the user data subsequence;

[0065] Obtain the volatility level harmonic coefficient of the volatility measure of the user data subsequence;

[0066] Determine the total number of data slope ratios of each user data subsequence;

[0067] The sequence volatility of each user data subsequence is determined based on the data slope concentration trend of each user data subsequence, the data slope discrete value of each user data subsequence, the random volatility factor of the user data subsequence volatility measure, the volatility level harmonic coefficient of the user data subsequence volatility measure, and the data slope ratio of each user data subsequence to the total number.

[0068] Specifically, the user portrait construction includes collecting information from the user's daily behavior, including browsing history, clicking on products, adding to shopping carts, purchase history, favorites, search history, user comments, product ratings, interactions with other users, including the type of device used by the user, access time, geographic location, and network type; extracting the product categories or brand preferences that the user is interested in based on the user's long-term behavior; understanding the timing of the user's purchase decision based on the user's behavior in different time periods; adjusting the recommendation strategy based on the user's current situation; using an unsupervised learning algorithm to group users with similar purchasing habits and browsing behaviors into one category; analyzing the user's interests and preferences, and constructing the user's interest portrait; inferring the user's current interests based on the user's recent browsing and interactive behaviors; extracting the user's stable preferences based on the user's long-term behavior patterns; analyzing the content of the products browsed or purchased by the user; extracting the key features of the products through natural language processing technology; establishing a model related to the user's interests; converting the user's behavior data and interests and hobbies into a vector representation for calculation; converting user features into high-dimensional vectors and using matrix decomposition and other methods to analyze the matching degree between the user and the product; and continuously optimizing the user portrait through user behavior feedback.

[0069] Based on user portraits, the system can provide users with personalized product recommendations. For different users, the recommendation system can push a list of products that suit their current interests and preferences. The advertising system can accurately locate users based on user portraits, push appropriate advertisements to target users, and improve advertising effectiveness. The platform can customize personalized promotional strategies based on user portraits, such as providing discount information for price-sensitive users and pushing limited-edition products to high-spending users. User portrait construction is a process based on big data analysis. Through static and dynamic feature extraction, combined with clustering and classification algorithms, it can deeply understand user behavior and preferences. Good user portraits can not only provide more accurate personalized recommendations for e-commerce platforms, but also improve user experience and the commercial benefits of the platform.

[0070] Specifically, the similarity calculation includes user similarity, which is mainly used in user-based collaborative filtering algorithms. According to user behavior and preferences, products that are liked by similar users are recommended. Based on the user's behavior vector, the similarity is measured by calculating the cosine angle between two vectors. Product similarity is used in item-based collaborative filtering algorithms or content-based recommendation algorithms to help the platform recommend other products similar to the currently browsed products to users. Based on the attribute characteristics of the products, product feature vectors are constructed and cosine similarity is calculated. Based on the numerical characteristics of the products, the similarity between products is calculated using Euclidean distance. For the text description of the products, TF-IDF, word embedding or BERT natural language processing technology is used to calculate the similarity of product texts. For products involving visual content, convolutional neural networks are used to extract features from product images, and image similarity is calculated after extracting visual feature vectors through deep learning models. In large-scale e-commerce platforms, user similarity matrices or product similarity matrices are pre-calculated and stored: each row and column of the user similarity matrix corresponds to a different user, and the value in the matrix represents the similarity between two users. Each row and column of the product similarity matrix corresponds to a different product, and the value in the matrix represents the similarity between two products.

[0071] Similarity calculation is a key step in the recommendation algorithm. It helps e-commerce platforms recommend suitable products to users by measuring the similarity between users and products. Common methods include cosine similarity, Jaccard similarity, Euclidean distance, etc. In addition, natural language processing, deep learning and other technologies can be combined to improve the accuracy of recommendations. In practical applications, hybrid methods are often used, and similarities are weighted and optimized to ensure the accuracy and personalization of the recommendation system.

[0072] Specifically, the cosine similarity calculation formula is as follows:

[0073]

[0074] Among them, u is the user's click data on the product, v is the user's purchase data on the product, ui is the number of times the user clicks on product i, vi is the number of times the user purchases product i, and n represents the product sequence.

[0075] Specifically, based on the numerical features of the products, the Euclidean distance is used to calculate the similarity between products. The formula is as follows:

[0076]

[0077] Among them, x is the name of product x, y is the name of product y, j is the common feature of product x and product y, and n is the feature type of the product.

[0078] Specifically, the construction of the analysis model includes defining business goals to provide personalized recommendations for users, improving sales conversion rates, grouping user groups by behavior and interests for more accurate marketing, predicting sales over a certain period of time in the future, optimizing inventory management and pricing strategies, understanding market trends and user preferences, providing data support for product development, conducting preliminary exploration of data through visualization tools, analyzing user behavior characteristics and product sales, using descriptive statistics and distribution analysis to understand the characteristics of the data, checking the correlation between features, identifying key features that affect the target variable, preparing for subsequent feature engineering and modeling, using methods such as the Pearson correlation coefficient, mutual information method, or model-based importance ranking to select the features that have the greatest impact on the target variable, extracting new features from existing data, encoding categorical data, normalizing or standardizing numerical features, and collaborative filtering-based models, including user-based collaborative filtering and product-based collaborative filtering. Collaborative filtering, through neural collaborative filtering and Wide&Deep model, combines deep learning with collaborative filtering to improve recommendation accuracy, divides the data set into training set and validation set to ensure the training effect and generalization ability of the model, evaluates model performance through K-fold cross validation to prevent model overfitting, uses grid search or random search method to adjust the model's hyperparameters and find the optimal parameter combination, deploys the model to the production environment in the real-time recommendation system, ensures that user behavior data can be updated in real time, and recommendation results can be generated dynamically, uses A / B testing to verify the effect of the new model, compares the performance of the new and old models, and ensures that the new model effectively improves the indicators, continuously iterates and optimizes the model based on user feedback and new data collected by the system, establishes a monitoring system, regularly monitors the performance of the model, and ensures that the model does not experience performance degradation. As user behavior and market environment change, the model is regularly retrained and updated to maintain the effectiveness and accuracy of the model. Building an analytical model is a systematic process, from data exploration, feature engineering to model selection and verification, to deployment and optimization, each step requires careful design and tuning. By using technologies such as machine learning and deep learning, combined with the big data of e-commerce platforms, we can build accurate and efficient recommendation systems, sales forecasting models, and customer segmentation models, thereby improving user experience and the commercial benefits of the platform. Through feedback mechanisms and model optimization, the accuracy and personalization level of e-commerce platform recommendation systems can be continuously improved. Feedback data is not only an important source of model optimization, but also allows the model to evolve adaptively through periodic retraining, parameter tuning, hybrid model integration, and other means. This process is dynamic and requires real-time monitoring and analysis of user feedback, as well as rapid response and optimization to ensure that the recommendation effect meets user needs and business goals at any time.

[0079] Specifically, the feedback mechanism and model optimization include feedback directly provided by users, through product ratings, comments, likes / dislikes, or through user behavior data, based on clicks, browsing time, adding to shopping carts, and bounce rates. When the model is launched, users are randomly divided into two groups using A / B testing, using the old model and the new model respectively, to compare their respective performances;

[0080] Continuously adjust recommendation strategies or advertising strategies through experimental design to find the optimal parameter configuration, set up user satisfaction surveys to obtain more direct feedback, identify potential problems in the model through real-time monitoring of user behavior, recommend irrelevant products or algorithm deviations during cold start, and store users' real-time feedback data and purchase conversion information in the data warehouse for subsequent analysis;

[0081] Clean the collected feedback data, including deleting abnormal behaviors, filling missing values, and removing noise data, to ensure that the feedback data can correctly reflect the user's true intentions. Generate more labels for the model based on the user's feedback data. For real-time recommendation systems, use implicit feedback to dynamically adjust the user's short-term interest model. By processing the real-time data stream, train new user portraits or interest models.

[0082] For non-real-time models, feedback data is used to retrain the model regularly to ensure that the model adapts to the new behavior patterns of users. Online learning algorithms are used to enable the model to make instant adjustments based on the latest feedback. Parameters are tuned based on feedback data. Based on the similarity threshold or the number of recommended products in the collaborative filtering model, explicit feedback and implicit feedback are combined to optimize according to the weight of the feedback. Multiple recommendation models are combined to form a hybrid recommendation system. The weight of each sub-model is dynamically adjusted through the feedback mechanism to improve the accuracy of the recommendation results.

[0083] Monitor changes in user behavior in real time, track the performance of model recommendation results, and continuously monitor the long-term effects of the model, especially the performance of the model on cold start and long-tail users, to avoid the recommendation of products always being concentrated on a few popular products. Use feedback data to detect whether the model is biased or unfair, detect changes in data distribution, and promptly discover inconsistencies between the model and actual data distribution to avoid model effect degradation;

[0084] We conduct continuous optimization based on feedback data, and constantly improve model structure, feature selection, and algorithm strategies based on user feedback. We use feedback mechanisms to improve personalization, and adjust recommendation algorithms based on user click and purchase history to make recommendations more in line with individual user interests rather than overly dependent on general preferences. We discover new features based on feedback data, and extract features related to user interests through the time users spend on specific products, helping the model to refine user portraits.

[0085] In one embodiment, a computer device is provided, which may be a terminal, and includes a processor, a memory, a communication interface, a display screen, and an input device connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be realized through WI FI, an operator network, NFC (near field communication) or other technologies. The display screen of the computer device may be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device may be a touch layer covered on the display screen, or a key, trackball or touchpad provided on the housing of the computer device, or an external keyboard, touchpad or mouse, etc.

[0086] The working principle of the design of a recommendation algorithm based on big data in an e-commerce platform of the present invention is as follows: by collecting user data, product data, and behavior data and establishing a database for storage, and performing data cleaning and conversion on the data in the database to complete data preprocessing; constructing a user portrait, analyzing the user's interests and preferences based on the user behavior data, generating a personalized user portrait, and using a clustering algorithm to classify the users to discover the group characteristics of the users; analyzing the user similarity and product similarity, calculating the similarity between users, and calculating the similarity between products through content similarity or collaborative filtering; based on user-based collaborative filtering and item-based collaborative filtering, dimensionality reduction processing of user and product features is performed by using SVD or a latent semantic model; using a neural network to model user behavior, and using an autoencoder to perform data compression and feature extraction; collecting user feedback data for continuous optimization and iteration of the model, and using online learning and A / B testing to adjust the recommendation strategy, which has the effect of achieving more accurate and personalized product recommendations, improving the user's shopping experience and the sales conversion rate of the platform.

[0087] The preferred embodiments of the present invention disclosed above are only used to help explain the present invention. The preferred embodiments do not describe all the details in detail, nor do they limit the invention to only specific implementation methods. Obviously, many modifications and changes can be made according to the content of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the present invention, so that those skilled in the art can understand and use the present invention well. The present invention is limited only by the claims and their full scope and equivalents.

Claims

1. A recommendation algorithm design based on big data in an e-commerce platform, characterized in that: include: Based on data collection and preprocessing, user data, product data, and behavior data are collected and stored in a database, and the data in the database is cleaned and converted to complete data preprocessing; Construct user portraits, analyze user interests and preferences based on user behavior data, generate personalized user portraits, and use clustering algorithms to classify users to discover user group characteristics; Through similarity calculation, the user similarity and product similarity are analyzed, the similarity between users is calculated, and the similarity between products is calculated through content similarity or collaborative filtering; Build analysis models, user-based collaborative filtering and item-based collaborative filtering, and use SVD or latent semantic models to reduce the dimensionality of user and item features; Based on deep learning methods, neural networks are used to model user behavior, and autoencoders are used for data compression and feature extraction; Through feedback mechanisms and model optimization, user feedback data is collected for continuous optimization and iteration of the model, and online learning and A / B testing are used to adjust the recommendation strategy.

2. The design of a recommendation algorithm based on big data in an e-commerce platform according to claim 1, characterized in that: The data collection and preprocessing include collecting users' browsing history, click history, shopping cart addition, purchase history, search history, users' age, gender, geographic location, user preferences and brands from social media, blogger information, user comments, ratings, feedback information on products, real-time inventory data and current discount activity data, processing missing data by deletion, filling or interpolation methods, using statistical methods or machine learning algorithms to detect and filter abnormal data, converting data to a unified scale, such as normalizing numerical features to improve the efficiency of the algorithm, converting categorical data to numerical data, using PCA or other dimensionality reduction techniques to simplify data features, reduce dimensions, and improve computing efficiency, extracting useful features from raw data, such as extracting key attributes of products, selecting the features that are most helpful for recommendation effects through correlation analysis or model selection, and integrating data from different sources to form a unified user-product interaction data set.

3. The design of a recommendation algorithm based on big data in an e-commerce platform according to claim 2, characterized in that: The data processing includes collecting the user's real-time data and establishing a database, cleaning the data in the database, and cleaning the data sets in the database, including carrying a data cleaning function, as follows: Among them, X i′ is the standardization of the current data, X i is the standardization of the current data, X max is the maximum standardization of the current data, X min This is the minimum standardization for the current data.

4. The design of a recommendation algorithm based on big data in an e-commerce platform according to claim 1, characterized in that: The data processing includes acquiring each pair of adjacent user data in each user data subsequence and the acquisition time of the pair of adjacent user data; The data slope ratio of the pair of adjacent user data in each user data subsequence is determined according to each pair of adjacent user data in each user data subsequence and the acquisition time of the pair of adjacent user data. In specific implementation, the data slope ratio can be determined by the following formula: in, represents the data slope ratio of the qth pair of adjacent user data in the rth user data subsequence, represents the right adjacent user data value of the qth pair of adjacent user data in the rth user data subsequence, represents the left adjacent user data value of the qth pair of adjacent user data in the rth user data subsequence, + represents the symbol of the right adjacent user data value of the adjacent user data, - represents the symbol of the left adjacent user data value of the adjacent user data, represents the acquisition time of the right adjacent user data value of the qth pair of adjacent user data in the rth user data subsequence, represents the acquisition time of the left adjacent user data value of the qth pair of adjacent user data in the rth user data subsequence.

5. The design of a recommendation algorithm based on big data in an e-commerce platform according to claim 4, characterized in that: The user portrait construction includes collecting information from the user's daily behavior, including browsing history, clicking on products, adding to shopping carts, purchase history, favorites, search history, user comments, product ratings, interactions with other users, including the type of device used by the user, access time, geographic location, and network type; extracting the product categories or brand preferences that the user is interested in based on the user's long-term behavior; understanding the timing of the user's purchase decision based on the user's behavior in different time periods; adjusting the recommendation strategy in combination with the user's current situation; using an unsupervised learning algorithm to group users with similar purchasing habits and browsing behaviors into one category; analyzing the user's interests and preferences, and constructing the user's interest portrait; inferring the user's current interests based on the user's recent browsing and interactive behaviors; extracting the user's stable preferences based on the user's long-term behavior patterns; analyzing the content of products browsed or purchased by the user; extracting the key features of the products through natural language processing technology; establishing a model related to the user's interests; converting the user's behavior data and interests and hobbies into a vector representation for calculation; converting user features into high-dimensional vectors and using matrix decomposition and other methods to analyze the matching degree between the user and the product; and continuously optimizing the user portrait through user behavior feedback.

6. The design of a recommendation algorithm based on big data in an e-commerce platform according to claim 5, characterized in that: The similarity calculation includes user similarity, which is mainly used in user-based collaborative filtering algorithms. According to the user's behavior and preference, the products that similar users like are recommended. Based on the user's behavior vector, the similarity is measured by calculating the cosine angle between two vectors. The product similarity is used in the collaborative filtering algorithm based on the item or the recommendation algorithm based on the content, which helps the platform recommend other products similar to the currently browsed products to the user. Based on the attribute characteristics of the product, the product feature vector is constructed and the cosine similarity is calculated. Based on the numerical characteristics of the product, the similarity between the products is calculated using the Euclidean distance. For the text description of the product, the similarity of the product text is calculated using TF-IDF, word embedding or BERT natural language processing technology. For products involving visual content, a convolutional neural network is used to extract features from the product image, and the image similarity is calculated after the visual feature vector is extracted by a deep learning model. In large-scale e-commerce platforms, a user similarity matrix or a product similarity matrix is ​​pre-calculated and stored: each row and each column of the user similarity matrix corresponds to a different user, and the value in the matrix represents the similarity between two users. Each row and each column of the product similarity matrix corresponds to a different product, and the value in the matrix represents the similarity between two products.

7. The design of a recommendation algorithm based on big data in an e-commerce platform according to claim 6, characterized in that: The cosine similarity calculation formula is as follows: Among them, u is the user's click data on the product, v is the user's purchase data on the product, and u i is the number of clicks on product i by the user, v i is the number of times the user purchases product i, and n represents the product sequence.

8. The design of a recommendation algorithm based on big data in an e-commerce platform according to claim 7, characterized in that: Based on the numerical features of the products, the Euclidean distance is used to calculate the similarity between products. The formula is as follows: Among them, x is the name of product x, y is the name of product y, j is the common feature of product x and product y, and n is the feature type of the product.

9. The design of a recommendation algorithm based on big data in an e-commerce platform according to claim 8, characterized in that: The construction of the analysis model includes defining business goals to provide personalized recommendations for users, improving sales conversion rates, grouping user groups by behavior and interests for more accurate marketing, predicting sales over a certain period of time in the future, optimizing inventory management and pricing strategies, understanding market trends and user preferences, providing data support for product development, conducting preliminary exploration of data through visualization tools, analyzing user behavior characteristics and product sales, using descriptive statistics and distribution analysis to understand the characteristics of the data, checking the correlation between features, identifying key features that affect the target variable, preparing for subsequent feature engineering and modeling, using methods such as the Pearson correlation coefficient, mutual information method, or model-based importance ranking to select the features that have the greatest impact on the target variable, extracting new features from existing data, encoding categorical data, normalizing or standardizing numerical features, and collaborative filtering-based models, including user-based collaborative filtering and product-based collaborative filtering. For the same filtering, deep learning is combined with collaborative filtering through neural collaborative filtering and Wide&Deep models to improve the accuracy of recommendations. The data set is divided into training set and validation set to ensure the training effect and generalization ability of the model. The model performance is evaluated through K-fold cross-validation to prevent model overfitting. The hyperparameters of the model are adjusted using grid search or random search methods to find the optimal parameter combination. In the real-time recommendation system, the model is deployed to the production environment to ensure that user behavior data can be updated in real time and recommendation results can be generated dynamically. A / B testing is used to verify the effect of the new model and compare the performance of the new and old models to ensure that the new model effectively improves the indicators. According to user feedback and new data collected by the system, the model is continuously iterated and optimized. A monitoring system is established to regularly monitor the performance of the model to ensure that the model has not experienced performance degradation. As user behavior and market environment change, the model is regularly retrained and updated to maintain the effectiveness and accuracy of the model.

10. The design of a recommendation algorithm based on big data in an e-commerce platform according to claim 8, characterized in that: The feedback mechanism and model optimization include feedback directly provided by users, through product ratings, comments, likes / dislikes, or through user behavior data, based on clicks, browsing time, adding to cart, and bounce rate. When the model is launched, users are randomly divided into two groups using A / B testing, using the old model and the new model respectively, and their respective performances are compared; Continuously adjust recommendation strategies or advertising strategies through experimental design to find the optimal parameter configuration, set up user satisfaction surveys to obtain more direct feedback, identify potential problems in the model through real-time monitoring of user behavior, recommend irrelevant products or algorithm deviations during cold start, and store users' real-time feedback data and purchase conversion information in the data warehouse for subsequent analysis; Clean the collected feedback data, including deleting abnormal behaviors, filling missing values, and removing noise data, to ensure that the feedback data can correctly reflect the user's true intentions. Generate more labels for the model based on the user's feedback data. For real-time recommendation systems, use implicit feedback to dynamically adjust the user's short-term interest model. By processing the real-time data stream, train new user portraits or interest models. For non-real-time models, feedback data is used to retrain the model regularly to ensure that the model adapts to the new behavior patterns of users. Online learning algorithms are used to enable the model to make instant adjustments based on the latest feedback. Parameters are tuned based on feedback data. Based on the similarity threshold or the number of recommended products in the collaborative filtering model, explicit feedback and implicit feedback are combined to optimize according to the weight of the feedback. Multiple recommendation models are combined to form a hybrid recommendation system. The weight of each sub-model is dynamically adjusted through the feedback mechanism to improve the accuracy of the recommendation results. Monitor changes in user behavior in real time, track the performance of model recommendation results, and continuously monitor the long-term effects of the model, especially the performance of the model on cold start and long-tail users, to avoid the recommendation of products always being concentrated on a few popular products. Use feedback data to detect whether the model is biased or unfair, detect changes in data distribution, and promptly discover inconsistencies between the model and actual data distribution to avoid model effect degradation; We conduct continuous optimization based on feedback data, and constantly improve model structure, feature selection, and algorithm strategies based on user feedback. We use feedback mechanisms to improve personalization, and adjust recommendation algorithms based on user click and purchase history to make recommendations more in line with individual user interests rather than overly dependent on general preferences. We discover new features based on feedback data, and extract features related to user interests through the time users spend on specific products, helping the model further refine user portraits.

Images