Live broadcast e-commerce intelligent management platform and method based on big data

Abstract

The invention discloses a live broadcast e-commerce intelligent management platform and method based on big data, and particularly relates to the field of live broadcast e-commerce. User behaviors, transactions and commodity data in a plurality of historical data sources are collected, data deviation is corrected through data preprocessing and an anomaly detection algorithm, data quality is ensured, and user experience is improved. Inputting the corrected data into a machine learning model for training, evaluating prediction performance through cross validation, optimizing model hyper-parameters, collecting data of an e-commerce platform in real time, evaluating the accuracy of a recommendation system in combination with data updating abnormity and model performance stability, and dynamically adjusting model parameters, so as to improve the recommendation accuracy of the e-commerce platform. The performance of the recommendation system is analyzed in a fixed time period, the recommendation algorithm and model are continuously optimized, the accuracy and efficiency of the system are ensured, the operation efficiency, recommendation precision and user experience of live broadcast e-commerce are effectively improved, and prediction errors caused by data deviation are reduced.

Description

Technical Field

[0001] The present invention relates to the field of live broadcast e-commerce, and in particular to a live broadcast e-commerce intelligent management platform and method based on big data. Background Art

[0002] Intelligent management of live e-commerce based on big data refers to the use of artificial intelligence technology to optimize and support decision-making in all aspects of live e-commerce by collecting, analyzing and processing massive amounts of consumer behavior data, sales data, inventory information, etc. This includes accurate product recommendations, real-time optimization of marketing strategies, intelligent inventory management, user portrait analysis, and providing personalized shopping experiences. Its core goal is to improve the operational efficiency of live e-commerce, increase sales conversion rates, and enhance users' shopping experience in a data-driven way. However, artificial intelligence technology relies on a large amount of data for training, but the quality and representativeness of this data are crucial. If there are deviations in the data source, such as false transactions, abnormal user behavior, or too single a consumer group, the artificial intelligence technology algorithm may make wrong predictions or decisions, affecting the recommendation effect, marketing strategy, and even worsening sales performance. Summary of the invention

[0003] The purpose of the present invention is to provide a live e-commerce intelligent management platform and method based on big data to solve the shortcomings of the background technology.

[0004] In order to achieve the above object, the present invention provides the following technical solution: a live e-commerce intelligent management method based on big data, comprising the following steps:

[0005] S1: Collect user behavior data, transaction data, and product data from several different historical data sources, pre-process the collected data, and use anomaly detection algorithms to automatically identify and correct potential data deviations;

[0006] S2: Input the corrected historical data into the machine learning model for model training. During the training process, the prediction performance of the model is evaluated through cross-validation technology, and the hyperparameters are optimized;

[0007] S3: Collect real-time data from the live e-commerce platform, evaluate the accuracy of the recommendation system based on the degree of update anomalies of the real-time data and the stability of the model prediction performance, and dynamically adjust the parameters of the recommendation algorithm and machine learning model based on the evaluation results;

[0008] S4: Based on the adjusted recommendation algorithm and machine learning model, further analyze the accuracy of the recommendation system within a fixed time period and continuously improve it based on the analysis results.

[0009] Preferably, in S1, potential data deviations are automatically identified and corrected through the isolation forest algorithm, specifically including: collecting user behavior data, transaction data, and commodity data from multiple data sources, extracting key features for anomaly detection after preprocessing the data, and standardizing the extracted features; dividing the data into a training set and a test set, building a model using the isolation forest algorithm, and identifying potential abnormal data points by randomly selecting data features and segmenting them; optimizing the hyperparameters of the isolation forest algorithm, including the number of trees in the forest, nestimators, the number of samples used by each tree, maxsamples, and the proportion of abnormal points in the data set, contamination; using the trained isolation forest model to detect the entire data, and marking abnormal data points based on a preset threshold; manually verifying and correcting the detected abnormal data in combination with business logic, and saving the corrected data in the database to ensure that subsequent artificial intelligence models can use high-quality data.

[0010] Preferably, in S2, the corrected historical data is input into a machine learning model for model training. During the training process, the prediction performance of the model is evaluated by a cross-validation technique, specifically: the corrected historical data set is divided into features and labels, the features are input variables, and the labels are predicted target variables. The data set is divided into k subsets using k-fold cross-validation, and k-1 subsets are selected as training sets each time, and the remaining 1 subset is used as a validation set, and this is repeated k times. In each validation process, a different subset is used as a validation set to ensure that the model can be evaluated under different data distributions;

[0011] Use k-1 subsets as training sets to train the machine learning model, and use the remaining 1 subset as the validation set to evaluate the performance of the model. In each iteration, the trained model is predicted on the validation set and the performance index is calculated. The performance index is calculated for the prediction result of each validation set. For each compromised validation set, the prediction performance index of the model is calculated. For the regression task, the mean square error MSE is calculated. The formula is: Where n is the number of samples in the validation set, y i is the actual value, is the predicted value of the model, and MSE is the mean square error, which is used to measure the difference between the model prediction and the true value;

[0012] After completing k cross-validation, calculate the average of all fold performance indicators as the overall performance measure of the model. For the MSE of k-fold cross-validation, the average calculation expression is: Average In the formula, MSE jis the mean square error of the jth fold, k is the number of folds. For classification tasks, the average classification accuracy is calculated as:

[0013] Preferably, in S3, real-time data in the live e-commerce platform is collected, and the accuracy of the recommendation system is evaluated based on the degree of update anomalies of the real-time data and the stability of the model prediction performance, specifically:

[0014] The real-time data in the live e-commerce platform is collected, and the data update anomaly index is generated according to the update anomaly degree of the real-time data. The method for obtaining the data update anomaly index is as follows:

[0015] Collect real-time data related to the live e-commerce platform, including user behavior data, product inventory data, sales data, and transaction data; use historical data to determine the normal data update range, and calculate the historical mean and variance of each feature as a benchmark reference. The calculation expressions are: σ f In the formula, f i is the characteristic value at a certain moment in the historical data, μ f is the historical mean of feature f, σ f is the historical standard deviation of feature f, n is the total number of periods of historical data, monitor the real-time data of each feature, and compare the real-time data at the current moment with the historical benchmark. For a feature f at time t, calculate the deviation between the real-time data and the historical mean. The calculation expression is: f(t) is the eigenvalue corresponding to time t in the real-time data, D f Update the anomaly index for the data.

[0016] Preferably, the mean classification accuracy and the data update anomaly index are normalized, and the accuracy coefficient of the recommendation system is calculated using the normalized mean classification accuracy and the data update anomaly index.

[0017] Preferably, the acquired accuracy coefficient of the recommendation system is compared with a preset accuracy coefficient reference threshold. If the accuracy coefficient of the recommendation system is greater than or equal to the preset accuracy coefficient reference threshold, it means that the accuracy of the recommendation system is high, and no warning signal is generated at this time; if the accuracy coefficient of the recommendation system is less than the preset accuracy coefficient reference threshold, it means that the accuracy of the recommendation system is low, and a warning signal is generated at this time, indicating that the prediction performance of the model has declined, and the model parameters or recommendation algorithm are adjusted.

[0018] Preferably, in S4, the accuracy of the recommendation system within a fixed time period is further analyzed based on the adjusted recommendation algorithm and machine learning model, specifically:

[0019] When the recommendation algorithm and machine learning model need to be adjusted, that is, the accuracy coefficient of the recommendation system generated within a fixed time period is less than the preset accuracy coefficient reference threshold, the accuracy coefficients of the recommendation systems generated in subsequent fixed time periods that are less than the preset accuracy coefficient reference threshold are collected, the accuracy coefficients in the data set are compared and analyzed with the accuracy coefficient reference threshold, the anomaly index of the recommendation system is calculated, and whether further adjustment is needed is determined based on the anomaly index.

[0020] Preferably, the acquired abnormality index is compared with a preset abnormality index reference threshold. If the abnormality index is greater than or equal to the preset abnormality index reference threshold, it means that the abnormality degree of the recommendation system is high and the system performance is unstable. At this time, an adjustment signal is generated, prompting further adjustment of the recommendation algorithm and machine learning model. If the abnormality index is less than the preset abnormality index reference threshold, it means that the abnormality degree of the recommendation system is low and there is no serious problem with the accuracy of the recommendation system. At this time, no adjustment signal is generated and no immediate adjustment is required.

[0021] The present invention also provides a live broadcast e-commerce intelligent management platform based on big data, including a data correction module, a model performance evaluation module, a dynamic adjustment module and an optimization and improvement module;

[0022] Data correction module: collects user behavior data, transaction data, and product data from several different historical data sources, pre-processes the collected data, and uses anomaly detection algorithms to automatically identify and correct potential data deviations;

[0023] Model performance evaluation module: input the corrected historical data into the machine learning model for model training. During the training process, the model's prediction performance is evaluated through cross-validation technology, and hyperparameters are optimized;

[0024] Dynamic adjustment module: collects real-time data from the live e-commerce platform, evaluates the accuracy of the recommendation system based on the degree of update anomalies of the real-time data and the stability of the model prediction performance, and dynamically adjusts the parameters of the recommendation algorithm and machine learning model based on the evaluation results;

[0025] Optimization and improvement module: Based on the adjusted recommendation algorithm and machine learning model, further analyze the accuracy of the recommendation system within a fixed time period, and continuously improve it based on the analysis results.

[0026] In the above technical solution, the technical effects and advantages provided by the present invention are:

[0027] 1. The present invention collects multi-dimensional user behavior data, transaction data, and commodity data, and uses the isolation forest algorithm to automatically identify and correct potential data deviations to ensure data accuracy. On this basis, the corrected historical data is used for machine learning model training, and the model performance is optimized through cross-validation technology to ensure the accuracy of the model on different data sets. Real-time data of the live e-commerce platform is collected in real time, and the performance of the recommendation system is dynamically evaluated and the algorithm and model parameters are adjusted in combination with the degree of data update anomalies and the stability of the model prediction performance, so as to ensure the accuracy of the recommendation results.

[0028] 2. The present invention not only improves the operational efficiency and sales conversion rate of live e-commerce through a data-driven approach, but also improves the user experience by continuously monitoring the performance of the recommendation system, dynamically adjusting and optimizing the algorithm. Through the calculation and evaluation of the abnormal index, the system can promptly identify and solve potential problems, ensure the long-term stability and efficient operation of the recommendation system, and thus reduce the negative impact caused by data deviation and model performance degradation. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0030] Figure 1 This is a method flow chart of the live e-commerce intelligent management method based on big data of the present invention.

[0031] Figure 2 This is a module diagram of the live e-commerce intelligent management platform based on big data of the present invention. DETAILED DESCRIPTION

[0032] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0033] Example 1, please refer to Figure 1 As shown, the live e-commerce intelligent management method based on big data described in this embodiment includes the following steps:

[0034] S1: Collect user behavior data, transaction data, and product data from several different historical data sources, pre-process the collected data, and use anomaly detection algorithms to automatically identify and correct potential data deviations;

[0035] S2: Input the corrected historical data into the machine learning model for model training. During the training process, the prediction performance of the model is evaluated through cross-validation technology, and the hyperparameters are optimized;

[0036] S3: Collect real-time data from the live e-commerce platform, evaluate the accuracy of the recommendation system based on the degree of update anomalies of the real-time data and the stability of the model prediction performance, and dynamically adjust the parameters of the recommendation algorithm and machine learning model based on the evaluation results;

[0037] S4: Based on the adjusted recommendation algorithm and machine learning model, further analyze the accuracy of the recommendation system within a fixed time period and continuously improve it based on the analysis results.

[0038] Among them, in S1, user behavior data, transaction data and product data are collected from several different historical data sources, the collected data are preprocessed, and the anomaly detection algorithm is used to automatically identify and correct potential data deviations.

[0039] Select key data sources from different historical data sources, including: User behavior data: browsing, clicking, adding to shopping cart, purchase records, etc. Transaction data: order quantity, transaction amount, payment method, etc. Product data: product name, category, price, inventory, evaluation, etc. Integrate data sources from different channels, such as e-commerce platforms, social media, historical sales systems, etc. through API, database query or data import tools to ensure the diversity of data sources. Since different data sources may have different formats and structures, the data format needs to be unified first. This includes converting data from different sources into a unified format (such as date format, numerical unit, etc.) and defining clear field mappings for each data source. Remove invalid data and noise data, such as: remove null values, duplicates, or data with incorrect formats. Remove irrelevant fields, such as orders or products that lack important information. Merge duplicate user behavior records or duplicate transaction records. Perform preliminary filtering for obviously abnormal data points. For example: abnormal user purchase behavior (such as large purchases in a short period of time). Abnormal inventory changes of products (inventory suddenly becomes negative). Abnormally high or low transaction amounts (such as obvious payment anomalies).

[0040] Use appropriate anomaly detection algorithms to automatically identify abnormal behaviors and deviations in data. Common algorithms include: Based on statistical analysis: such as the Z-Score algorithm based on mean and variance to identify outliers. Based on machine learning: such as Isolation Forest, DBSCAN and other algorithms to identify possible abnormal groups. Based on time series analysis: For time-sensitive data, use time series models (such as ARIMA, LSTM) to detect abnormal trends.

[0041] Automatically identify and correct potential data biases through the Isolation Forest algorithm, including:

[0042] Collect relevant data from multiple data sources, such as user behavior data, transaction data, product data, etc. Ensure that these data cover diversity and different dimensions of features (for example, purchase frequency, transaction amount, product inventory, etc.). Perform basic data cleaning to remove missing values, duplicate values, and obviously erroneous data (such as incorrectly formatted dates, blank transaction records, etc.).

[0043] Select key features for anomaly detection. For example: For user behavior data: user activity, purchase frequency, average order amount, etc. For transaction data: transaction amount, order quantity, payment method, etc. For product data: inventory changes, sales volume, product ratings, etc. After extracting these features, standardize the data and scale the features to the same numerical range so that the isolation forest algorithm can better handle features of different scales.

[0044] Divide the data into training and test sets to evaluate the effectiveness of the algorithm. A certain proportion of the data can be randomly selected as the training set.

[0045] The isolation forest algorithm is used to build the model. The core idea of ​​the isolation forest is to identify those data points that are easy to be isolated (i.e., outliers) by randomly selecting data features and segmenting them. Unlike other algorithms, the isolation forest is more suitable for high-dimensional data and has higher computational efficiency.

[0046] The model is trained using the preprocessed and standardized data. The model will automatically learn the normal patterns in the data and try to “isolate” those data points that are clearly abnormal.

[0047] Adjust the hyperparameters of the isolation forest, such as nestimators: the number of trees in the forest, usually set to 100-200 trees.

[0048] maxsamples: The number of samples used per tree, usually set to a small fraction of the dataset size (such as 256).

[0049] Contamination: It is used to specify the proportion of outliers in the data set. If you are unsure, you can set it to automatic detection.

[0050] Through cross-validation or grid search, optimize hyperparameters to improve the detection effect of the model.

[0051] Use the trained isolation forest model to detect anomalies on the entire data. The model generates an anomaly score for each data point (usually a value between 0 and 1). The higher the score, the more likely the data point is an anomaly.

[0052] Based on the output of the model, a threshold is set, usually 0.5 or 0.7 is used as the threshold to distinguish normal data from abnormal data. Any point exceeding this threshold will be marked as an anomaly.

[0053] Flag data points detected as anomalous by the Isolation Forest model, such as unusually high transaction amounts, frequent inventory changes, or unusual user behavior.

[0054] Combined with business logic, some abnormal data marked by Isolation Forest are manually verified to ensure the accuracy of detection.

[0055] For obvious abnormal data, an automatic correction strategy can be adopted. For example, for abnormally high transaction amounts, a reasonable alternative value can be set based on the historical average transaction amount.

[0056] For abnormal inventory fluctuations, use the average or median of historical inventory data to make corrections.

[0057] For minor anomalies (such as boundary anomalies), the mean or median of adjacent normal data points can be used for filling.

[0058] For some complex anomalies, manual review can be used to handle them. Using the marked abnormal data points, manual confirmation is performed to determine whether they are indeed abnormal, and adjustments are made based on specific business scenarios.

[0059] Save the corrected data to the database to ensure that subsequent AI models and systems can use high-quality and accurate data.

[0060] The higher the mean classification accuracy, the better the classification and recommendation capabilities of the recommendation system on different data sets, and the higher the prediction accuracy. The classification accuracy measures the proportion of correct recommendations made by the recommendation system in multiple tests or cross-validation. Therefore, a high classification accuracy means that the system can more effectively map user behavior and product features to user preferences, and the recommendation results are more in line with user needs.

[0061] As the mean classification accuracy increases, the recommendation system becomes more accurate in determining which products are suitable for users. This indicates that the system has a stronger ability to learn and understand user behavior and historical data, and can capture more detailed patterns, thereby providing personalized and highly relevant product recommendations, enhancing the user's shopping experience and satisfaction.

[0062] Therefore, the higher the mean classification accuracy, the better the accuracy of the recommendation system, and the higher the match between the recommended products and user needs. This not only helps to improve user engagement, but also increases conversion rates and sales, making the overall performance of the system more reliable and efficient.

[0063] In order to optimize the hyperparameters of the model, use cross-validation with grid search or random search, and perform multiple cross-validations based on different hyperparameter combinations. For example: Grid search: traverse multiple specified hyperparameter combinations (such as tree depth, learning rate, etc.) for cross-validation. Random search: randomly select multiple hyperparameter combinations to reduce computational complexity. Select the hyperparameter combination with the best performance based on the average performance index of cross-validation. For example, select the hyperparameter combination with the smallest average MSE or the largest average Accuracy.

[0064] Using the best hyperparameter combination found, retrain the final model on the entire training set (instead of the k-1 subsets). Evaluate the final performance of the model on an independent test set to ensure that the model has good predictive power on unseen data.

[0065] In this application, k-fold cross validation is used to comprehensively evaluate the performance of the model on different data sets to prevent the model from overfitting due to a specific data partition. During the cross validation process, the performance indicators (such as MSE, accuracy, etc.) of each fold are calculated, and the overall performance of the model is measured by the average value. The model is further optimized by combining hyperparameter search, and finally the best hyperparameters are selected for model training and its effect on real data is verified.

[0066] S3: Collect real-time data from the live e-commerce platform, evaluate the accuracy of the recommendation system based on the degree of update anomalies of the real-time data and the stability of the model prediction performance, and dynamically adjust the parameters of the recommendation algorithm and machine learning model based on the evaluation results.

[0067] The real-time data in the live e-commerce platform is collected, and the data update anomaly index is generated according to the update anomaly degree of the real-time data. The method for obtaining the data update anomaly index is as follows:

[0068] Collect real-time data related to the live e-commerce platform, including user behavior data, product inventory data, sales data, and transaction data; use historical data to determine the normal data update range, and calculate the historical mean and variance of each feature as a benchmark reference. The calculation expressions are: In the formula, f i is the characteristic value at a certain moment in the historical data (such as the order volume at a certain minute), μ f is the historical mean of feature f, σ f is the historical standard deviation of feature f, n is the total number of periods of historical data, monitor the real-time data of each feature, and compare the real-time data at the current moment with the historical benchmark. For a feature f at a specific time t, calculate the deviation between the real-time data and the historical mean. The calculation expression is: f(t) is the eigenvalue corresponding to time t in the real-time data, D f Update the anomaly index for the data.

[0069] The larger the data update anomaly index, the greater the deviation between real-time data and historical normal data, which means that the data on which the recommendation system relies has experienced large abnormal fluctuations. The recommendation system relies on high-quality, accurate real-time data to generate personalized recommendations. Therefore, when the data update anomaly index is large, the recommendation system may not be able to obtain the correct user behavior, inventory status, or product information in a timely manner.

[0070] As the data anomaly index increases, the accuracy of the recommendation system will gradually decrease. Data anomalies may cause the recommendation system to rely on outdated or erroneous data for calculations, such as recommending items that are already sold out, or misestimating user preferences, causing the recommendation results to deviate from the actual needs of users. In addition, fluctuations in key features in real-time data (such as abnormal user behavior) may reduce the predictive ability of the model, resulting in recommendation results that are inconsistent with user expectations. Therefore, when the data update anomaly index is large, the accuracy and reliability of the recommendation system will be significantly affected. In this case, the system needs to respond quickly and correct data anomalies in a timely manner to restore the normal operation of the recommendation system and ensure that users can obtain accurate and personalized recommendation results.

[0071] The mean classification accuracy and the data update anomaly index are normalized, and the accuracy coefficient of the recommendation system is calculated based on the normalized mean classification accuracy and the data update anomaly index.

[0072] For example, the present invention can use the following formula to calculate the accuracy coefficient of the recommendation system, and the calculation expression is: In the formula, LK is the accuracy coefficient of the recommendation system, AverageAccuracy is the average classification accuracy, and D fis the data update anomaly index, a1 and a2 are the proportional coefficients of the mean classification accuracy and the data update anomaly index, and a2>a1>0.

[0073] The acquired accuracy coefficient of the recommendation system is compared with the preset accuracy coefficient reference threshold. If the accuracy coefficient of the recommendation system is greater than or equal to the preset accuracy coefficient reference threshold, it means that the accuracy of the recommendation system is high, and no warning signal is generated at this time; if the accuracy coefficient of the recommendation system is less than the preset accuracy coefficient reference threshold, it means that the accuracy of the recommendation system is low, and a warning signal is generated at this time, indicating that the prediction performance of the model has declined, and it is recommended to adjust the model parameters or recommendation algorithm.

[0074] Based on the early warning signals, the system can make the following adjustments:

[0075] Re-adjust model hyperparameters: Re-optimize model hyperparameters such as learning rate, regularization parameter, tree depth, etc. through grid search or random search.

[0076] Switching recommendation algorithms: If the current algorithm does not work well, the system can choose to switch to an alternative algorithm, such as switching from a collaborative filtering-based algorithm to a content-based recommendation algorithm, or combining multiple algorithms (such as a hybrid recommendation system).

[0077] Retraining the model: Under new data conditions, the model may need to be retrained through online learning or batch learning to adapt to the new data distribution and user behavior.

[0078] S4: Based on the adjusted recommendation algorithm and machine learning model, further analyze the accuracy of the recommendation system within a fixed time period and continuously improve it based on the analysis results.

[0079] When the recommendation algorithm and machine learning model need to be adjusted, that is, the accuracy coefficient of the recommendation system generated within a fixed time period is less than the preset accuracy coefficient reference threshold, the accuracy coefficients of the recommendation systems generated within the subsequent fixed time period that are less than the preset accuracy coefficient reference threshold are collected, the accuracy coefficients in the data set are compared and analyzed with the accuracy coefficient reference threshold, and the abnormal index of the recommendation system is calculated. According to the abnormal index, it is determined whether further adjustments are needed. The specific calculation expression is: Where LF is the abnormality index, LKM is the accuracy coefficient reference threshold, D is the total number of accuracy coefficients of the recommendation systems in the set that are less than the preset accuracy coefficient reference threshold, and LK ris the accuracy coefficient of the recommendation system that is less than a preset accuracy coefficient reference threshold, r is the label of the accuracy coefficient of the recommendation system that is less than the preset accuracy coefficient reference threshold in the data set, and r=1, 2, 3, ..., z, where z is a positive integer greater than 0.

[0080] The obtained anomaly index is compared with the pre-set anomaly index reference threshold. If the anomaly index is greater than or equal to the pre-set anomaly index reference threshold, it means that the abnormality level of the recommendation system is high and the system performance is unstable. At this time, an adjustment signal is generated, prompting further optimization or adjustment of the recommendation algorithm and machine learning model; if the anomaly index is less than the pre-set anomaly index reference threshold, it means that the abnormality level of the recommendation system is low and there is no serious problem with the accuracy of the recommendation system. At this time, no adjustment signal is generated and no immediate adjustment is required.

[0081] In this embodiment, user behavior, transaction and commodity data are first collected from multiple historical data sources, and data deviations are corrected through preprocessing and anomaly detection algorithms. The corrected data is then input into the machine learning model for training, and the prediction performance and hyperparameters are evaluated through cross-validation technology. Then, the e-commerce platform data is collected in real time, and the accuracy of the recommendation system is dynamically evaluated based on the degree of data update anomalies and model prediction stability, and the algorithm and model parameters are adjusted based on the evaluation results. Finally, the performance of the recommendation system is further analyzed and continuously improved within a fixed time period to ensure the long-term accuracy and stability of the recommendation system.

[0082] Example 2, please refer to Figure 2 As shown, the live e-commerce intelligent management platform based on big data described in this embodiment includes a data correction module, a model performance evaluation module, a dynamic adjustment module and an optimization and improvement module;

[0083] Data correction module: collects user behavior data, transaction data, and product data from several different historical data sources, pre-processes the collected data, and uses anomaly detection algorithms to automatically identify and correct potential data deviations;

[0084] Model performance evaluation module: input the corrected historical data into the machine learning model for model training. During the training process, the prediction performance of the model is evaluated through cross-validation technology, and the parameters are optimized;

[0085] Dynamic adjustment module: collects real-time data from the live e-commerce platform, evaluates the accuracy of the recommendation system based on the degree of update anomalies of the real-time data and the stability of the model prediction performance, and dynamically adjusts the parameters of the recommendation algorithm and machine learning model based on the evaluation results;

[0086] Optimization and improvement module: Based on the adjusted recommendation algorithm and machine learning model, further analyze the accuracy of the recommendation system within a fixed time period, and continuously improve it based on the analysis results.

[0087] The above formulas are all dimensionless and numerical calculations. The formula is a formula for the most recent real situation obtained by collecting a large amount of data and performing software simulation. The preset parameters in the formula are set by technicians in this field according to actual conditions.

[0088] The above embodiments can be implemented in whole or in part by software, hardware, firmware or any other combination. When implemented by software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website site, computer, server or data center to another website site, computer, server or data center by wired or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that contains one or more available media sets. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state hard disk.

[0089] It should be understood that the term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. A and B can be singular or plural. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship, but it may also indicate an "and / or" relationship. Please refer to the context for specific understanding.

[0090] It should be understood that in the various embodiments of the present application, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0091] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0092] The above description is only a specific implementation manner of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application.

Claims

1. Intelligent management method of live e-commerce based on big data, Features: The following steps are involved: S1: Collect user behavior data, transaction data, and product data from several different historical data sources, pre-process the collected data, and use anomaly detection algorithms to automatically identify and correct potential data deviations; S2: Input the corrected historical data into the machine learning model for model training. During the training process, the prediction performance of the model is evaluated through cross-validation technology, and the hyperparameters are optimized; S3: Collect real-time data from the live e-commerce platform, evaluate the accuracy of the recommendation system based on the degree of update anomalies of the real-time data and the stability of the model prediction performance, and dynamically adjust the parameters of the recommendation algorithm and machine learning model based on the evaluation results; S4: Based on the adjusted recommendation algorithm and machine learning model, further analyze the accuracy of the recommendation system within a fixed time period and continuously improve it based on the analysis results.

2. According to the big data-based live e-commerce intelligent management method of claim 1, Features: In S1, the isolation forest algorithm is used to automatically identify and correct potential data deviations, including: collecting user behavior data, transaction data, and product data from multiple data sources, extracting key features for anomaly detection after preprocessing the data, and standardizing the extracted features; dividing the data into training sets and test sets, building a model using the isolation forest algorithm, and identifying potential abnormal data points by randomly selecting data features and segmenting them; optimizing the hyperparameters of the isolation forest algorithm, including the number of trees in the forest, the number of samples used by each tree, and the proportion of abnormal points in the data set; using the trained isolation forest model to detect the entire data and mark abnormal data points based on preset thresholds; manually verifying and correcting the detected abnormal data in combination with business logic, and saving the corrected data in the database to ensure that subsequent artificial intelligence models can use high-quality data.

3. According to the big data-based live e-commerce intelligent management method of claim 1, Features: In S2, the corrected historical data is input into the machine learning model for model training. During the training process, the prediction performance of the model is evaluated by cross-validation technology. Specifically, the corrected historical data set is divided into features and labels. Features are input variables, and labels are predicted target variables. The data set is divided into k subsets using k-fold cross-validation. Each time, k-1 subsets are selected as training sets, and the remaining 1 subset is used as a validation set. This is repeated k times. In each validation process, a different subset is used as a validation set to ensure that the model can be evaluated under different data distributions. Use k-1 subsets as training sets to train the machine learning model, and use the remaining 1 subset as the validation set to evaluate the performance of the model. In each iteration, the trained model is predicted on the validation set and the performance index is calculated. The performance index is calculated for the prediction result of each validation set. For each compromised validation set, the prediction performance index of the model is calculated. For the regression task, the mean square error MSE is calculated. The formula is: Where n is the number of samples in the validation set, y i is the actual value, is the predicted value of the model, and MSE is the mean square error, which is used to measure the difference between the model prediction and the true value; After completing k cross-validation, calculate the average of all fold performance indicators as the overall performance measure of the model. For the MSE of k-fold cross-validation, the average calculation expression is: In the formula, MSE j is the mean square error of the jth fold, k is the number of folds. For classification tasks, the mean classification accuracy is calculated as:

4. According to the big data-based live e-commerce intelligent management method of claim 1, Features: In S3, real-time data from the live e-commerce platform is collected, and the accuracy of the recommendation system is evaluated based on the degree of update anomalies of real-time data and the stability of the model prediction performance. Specifically: The real-time data in the live e-commerce platform is collected, and the data update anomaly index is generated according to the update anomaly degree of the real-time data. The method for obtaining the data update anomaly index is as follows: Collect real-time data related to the live e-commerce platform, including user behavior data, product inventory data, sales data, and transaction data; use historical data to determine the normal data update range, and calculate the historical mean and variance of each feature as a benchmark reference. The calculation expressions are: In the formula, f i is the characteristic value at a certain moment in the historical data, μ f is the historical mean of feature f, σ f is the historical standard deviation of feature f, n is the total number of periods of historical data, monitor the real-time data of each feature, and compare the real-time data at the current moment with the historical benchmark. For a feature f at time t, calculate the deviation between the real-time data and the historical mean. The calculation expression is: f(t) is the eigenvalue corresponding to time t in the real-time data, D f Update the anomaly index for the data.

5. According to the big data-based live e-commerce intelligent management method of claim 4, Features: The mean classification accuracy and the data update anomaly index are normalized, and the accuracy coefficient of the recommendation system is calculated based on the normalized mean classification accuracy and the data update anomaly index.

6. According to the big data-based live e-commerce intelligent management method of claim 5, Features: The acquired accuracy coefficient of the recommendation system is compared with the preset accuracy coefficient reference threshold. If the accuracy coefficient of the recommendation system is greater than or equal to the preset accuracy coefficient reference threshold, it means that the accuracy of the recommendation system is high, and no warning signal is generated at this time; if the accuracy coefficient of the recommendation system is less than the preset accuracy coefficient reference threshold, it means that the accuracy of the recommendation system is low, and a warning signal is generated at this time, indicating that the prediction performance of the model has declined, and the model parameters or recommendation algorithm are adjusted.

7. According to the big data-based live e-commerce intelligent management method of claim 1, Features: In S4, based on the adjusted recommendation algorithm and machine learning model, the accuracy of the recommendation system within a fixed time period is further analyzed, specifically: When the recommendation algorithm and machine learning model need to be adjusted, that is, the accuracy coefficient of the recommendation system generated within a fixed time period is less than the preset accuracy coefficient reference threshold, the accuracy coefficients of the recommendation systems generated in subsequent fixed time periods that are less than the preset accuracy coefficient reference threshold are collected, the accuracy coefficients in the data set are compared and analyzed with the accuracy coefficient reference threshold, the anomaly index of the recommendation system is calculated, and whether further adjustment is needed is determined based on the anomaly index.

8. According to the big data-based live e-commerce intelligent management method of claim 7, Features: The obtained anomaly index is compared with the pre-set anomaly index reference threshold. If the anomaly index is greater than or equal to the pre-set anomaly index reference threshold, it means that the abnormality level of the recommendation system is high and the system performance is unstable. At this time, an adjustment signal is generated, prompting further adjustment of the recommendation algorithm and machine learning model; if the anomaly index is less than the pre-set anomaly index reference threshold, it means that the abnormality level of the recommendation system is low and there is no serious problem with the accuracy of the recommendation system. At this time, no adjustment signal is generated and no immediate adjustment is required.

9. A live e-commerce intelligent management platform based on big data, used to implement the live e-commerce intelligent management method based on big data as described in any one of claims 1 to 8, Features: It includes data correction module, model performance evaluation module, dynamic adjustment module and optimization and improvement module; Data correction module: collects user behavior data, transaction data, and product data from several different historical data sources, pre-processes the collected data, and uses anomaly detection algorithms to automatically identify and correct potential data deviations; Model performance evaluation module: input the corrected historical data into the machine learning model for model training. During the training process, the model's prediction performance is evaluated through cross-validation technology, and hyperparameters are optimized; Dynamic adjustment module: collects real-time data from the live e-commerce platform, evaluates the accuracy of the recommendation system based on the degree of update anomalies of the real-time data and the stability of the model prediction performance, and dynamically adjusts the parameters of the recommendation algorithm and machine learning model based on the evaluation results; Optimization and improvement module: Based on the adjusted recommendation algorithm and machine learning model, further analyze the accuracy of the recommendation system within a fixed time period, and continuously improve it based on the analysis results.

Images