An adaptive information recommendation data processing method, system, terminal and medium

Abstract

The application relates to the technical field of data processing, and discloses a self-adaptive information recommendation data processing method and system, a terminal and a medium. The method comprises the following steps: obtaining an original feature vector with missing values of a user and performing missing value interpolation processing to obtain a complete dense feature vector, and then inputting the complete dense feature vector into a machine learning prediction model to perform reasoning, generate information recommendation result data and a prediction result label; comparing the prediction result label with a real feedback label of the user, calculating a current cumulative error rate and a standard deviation, determining a historical optimal error rate benchmark, then performing statistical threshold comparison, and generating a retraining trigger signal if a second determination condition is met; and in response to the retraining trigger signal, generating updated model parameters based on training data, and deploying the updated model parameters to the machine learning prediction model. The application can reduce the maintenance cost and resource consumption of operation, and improve the utilization efficiency of computing resources.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to an adaptive information recommendation data processing method, system, terminal and medium. Background Technology

[0002] In product distribution scenarios, product operators typically act as intermediaries between external providers and customers, relying on their understanding of customer needs to recommend suitable products. With the development of the digital economy, leveraging machine learning to build recommendation systems to improve recommendation efficiency and customer experience has become an industry trend. Existing technologies commonly employ algorithms such as RandomForest, Collaborative filtering, XGBoost, and Bayesian Networks to construct recommendation models.

[0003] Existing solutions generate product recommendation strategies based on customers' historical insurance policies, family protection gaps, and marketing stages. While these solutions implement recommendation logic based on user profiles, they remain static recommendation architectures. Specifically, these solutions suffer from the following technical drawbacks: They lack real-time monitoring capabilities for concept drift; when product market conditions, customer preferences, or product strategies change, the system cannot proactively detect model performance degradation. The recommendation model lacks an online feedback loop mechanism, making it impossible to dynamically adjust model parameters based on subsequent contract feedback. To maintain recommendation effectiveness, it typically relies on manual, periodic full retraining (e.g., weekly or monthly model updates). Even if the model does not exhibit significant degradation, the system may still perform a large amount of unnecessary training, wasting computational resources. Full retraining involves reloading large-scale historical data and feature calculations, easily leading to high CPU or GPU resource consumption and system maintenance costs. Recommendation results primarily remain at the product or marketing strategy level, lacking an explanation mechanism for the feature contributions of individual customers, making it difficult for business personnel to understand the basis for recommendations.

[0004] Therefore, existing technologies still need to be improved and developed. Summary of the Invention

[0005] The main purpose of this application is to provide an adaptive information recommendation data processing method, system, terminal and medium, which aims to solve the problem that existing static recommendation methods lack the ability to perceive changes in data distribution (concept drift) in real time, and need to rely on periodic full retraining to maintain performance, resulting in excessive consumption of computing resources and low maintenance efficiency.

[0006] The first aspect of this application provides an adaptive information recommendation data processing method, which includes the following steps: Obtain the original feature vector of the user with missing values, and perform missing value imputation on the original feature vector to obtain a complete dense feature vector without missing values; The dense feature vector is input into the currently deployed machine learning prediction model for inference, outputting the probability distribution for each candidate information item, and generating information recommendation result data and prediction result labels based on the probability distribution; Obtain real user feedback tags, compare the predicted result tags with the real user feedback tags, generate error indicator variables for the current sample, calculate the current cumulative error rate and standard deviation based on the accumulated error indicator variables, determine the historical best error rate benchmark, compare the current cumulative error rate and standard deviation with the historical best error rate benchmark using statistical thresholds, if the first judgment condition is met, generate a warning log, if the second judgment condition is met, generate a retraining trigger signal; In response to the retraining trigger signal, the latest accumulated feedback sample set is obtained as training data, the parameters of the machine learning prediction model are optimized based on the training data, updated model parameters are generated, the updated model parameters are deployed to the machine learning prediction model, and the historical best error rate benchmark is reset to the statistical initial value at the current moment.

[0007] Optionally, in one embodiment of this application, the step of performing missing value imputation on the original feature vector to obtain a complete dense feature vector without missing values ​​specifically includes: The original feature vector is subjected to missing mask identification to obtain the known feature set and the missing index set; Obtain reference samples from the complete historical samples, and calculate the masked Euclidean distance between the current sample and each historical sample based on the reference samples and the known feature set to obtain a set of distance values; Determine the nearest neighbor sample set based on the distance value set; Based on the nearest neighbor sample set and the missing index set, an estimated imputation value set is obtained; The estimated imputation value set and the original feature vector are merged and recombined to obtain a complete dense feature vector.

[0008] Optionally, in one embodiment of this application, obtaining the estimated imputation value set based on the nearest neighbor sample set and the missing index set specifically includes: For each missing dimension in the missing index set, the weight coefficient of each nearest sample is calculated based on the distance between each nearest sample in the nearest sample set and the current sample: ; in, For the first The weight coefficients of each neighbor, For the first The mask Euclidean distance between the current sample and its nearest neighbor samples. This is a preset, minimal smoothing constant; Based on the weighting coefficients and the known values ​​of each nearest sample in the nearest sample set for the missing dimension, calculate the estimated imputation value for the missing dimension: ; in, For missing dimensions The estimated fill value, Given the known values ​​of the neighbor samples in the missing dimension. For the nearest neighbor sample set; Iterate through all missing dimensions in the missing index set to obtain a set of estimated imputation values ​​formed by the estimated imputation values ​​of each missing dimension.

[0009] Optionally, in one embodiment of this application, the step of inputting the dense feature vector into the currently deployed machine learning prediction model for inference and outputting the probability distribution for each candidate information item specifically includes: The complete dense feature vector is input into the currently deployed machine learning prediction model, and the original score vector of each candidate information item is output: ; in, This is the original score vector output. No. The original scores of each candidate information item For a complete dense feature vector, This is the current set of model parameters; The original score vector is normalized using the Softmax function, mapping the original scores of each candidate information item to a probability distribution: ; in, To complete dense feature vectors Under the conditions The probability of each candidate information item. It is an exponential function with the natural constant as its base. The sum of the original scores for all candidate information items. For the first Original scores for each candidate product; By traversing all candidate information items, the probability distribution vector formed by the probabilities of each candidate information item is obtained.

[0010] Optionally, in one embodiment of this application, each of the error indication variables takes the value of 1 or 0; The calculation of the current cumulative error rate and standard deviation based on the accumulated error indicator variables specifically includes: The current cumulative error rate is calculated based on the accumulated error indicator variables: ; in, This represents the current cumulative error rate. For error indicator variables, This represents the total number of cumulative feedback samples. This is the loop variable in the summation formula; The standard deviation is calculated based on the accumulated error indicator variables: ; in, The standard deviation is denoted as .

[0011] Optionally, in one embodiment of this application, the first determination condition is: ; The second determination condition is: ; in, The minimum error rate among the historical best error rate benchmarks. This represents the standard deviation corresponding to the minimum error rate.

[0012] Optionally, in one embodiment of this application, the step of optimizing the parameters of the machine learning prediction model based on the training data to generate updated model parameters specifically includes: With the goal of minimizing prediction loss, the parameters of the machine learning prediction model are optimized based on the training data to generate updated model parameters, wherein the optimization objective function is: ; in, For the updated model parameters, The model parameters to be optimized are... For the first The true feedback labels of each training sample For machine learning prediction models, the prediction function is... For the first The complete dense feature vector of each training sample.

[0013] A second aspect of this application also provides an adaptive information recommendation data processing system, wherein the adaptive information recommendation data processing system is used to implement the adaptive information recommendation data processing method described in any one of the above solutions; the adaptive information recommendation data processing system includes: The data processing module is used to obtain the user's original feature vector with missing values, and to perform missing value imputation processing on the original feature vector to obtain a complete dense feature vector without missing values. The recommendation generation module is used to input the dense feature vector into the currently deployed machine learning prediction model for inference, output the probability distribution for each candidate information item, and generate information recommendation result data and prediction result labels based on the probability distribution. The drift detection module is used to acquire user real feedback labels, compare the predicted result labels with the user real feedback labels, generate error indicator variables for the current sample, calculate the current cumulative error rate and standard deviation based on the accumulated error indicator variables, determine the historical best error rate benchmark, compare the current cumulative error rate and standard deviation with the historical best error rate benchmark using statistical thresholds, and generate a warning log if the first judgment condition is met, and generate a retraining trigger signal if the second judgment condition is met. An automatic retraining module is used to respond to the retraining trigger signal, acquire the latest accumulated feedback sample set as training data, optimize the parameters of the machine learning prediction model based on the training data, generate updated model parameters, deploy the updated model parameters to the machine learning prediction model, and reset the historical best error rate benchmark to the statistical initial value at the current moment.

[0014] A third aspect of this application also provides a terminal, wherein the terminal includes: a memory, a processor, and an adaptive information recommendation data processing program stored in the memory and executable on the processor, wherein when the adaptive information recommendation data processing program is executed by the processor, it implements the steps of the adaptive information recommendation data processing method as described above.

[0015] A fourth aspect of this application also provides a computer-readable storage medium, wherein the computer-readable storage medium stores an adaptive information recommendation data processing program, which, when executed by a processor, implements the steps of the adaptive information recommendation data processing method described above.

[0016] Beneficial Effects: This application provides an adaptive information recommendation data processing method, system, terminal, and medium. By introducing a drift detection mechanism based on statistical threshold discrimination, this application achieves real-time online monitoring of the prediction performance of the recommendation system. Model retraining is triggered only when a significant concept drift in the data distribution is detected, avoiding the large-scale repeated loading of historical data and excessive consumption of computing resources caused by periodic full retraining in traditional solutions. This reduces maintenance costs and resource overhead during long-term system operation and improves the utilization efficiency of computing resources. At the same time, this application enables the recommendation model to dynamically evolve with changes in the market environment or user preferences. While maintaining a high level of prediction accuracy, it achieves a leap from periodic manual intervention in model maintenance to autonomous on-demand updates by the system, enhancing the long-term operational stability and adaptability of the system. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 A flowchart illustrating a preferred embodiment of the adaptive information recommendation data processing method of this application; Figure 2 This is a diagram illustrating the overall architecture and data flow of the adaptive information recommendation data processing system of this application. Figure 3 This is a visualization example of the radar profile generated for the first product in the adaptive information recommendation data processing method of this application. Figure 4 This is a visualization example of the radar profile generated for the second type of product in the adaptive information recommendation data processing method of this application. Figure 5 This is a visualization example of the radar profile generated for the third type of product in the adaptive information recommendation data processing method of this application. Figure 6 This is a structural diagram of a preferred embodiment of the adaptive information recommendation data processing system of this application; Figure 7 This is a structural diagram of a preferred embodiment of the terminal of this application.

[0019] Explanation of reference numerals in the attached figures: 100. Data processing module; 200. Recommendation generation module; 300. Drift detection module; 400. Automatic retraining module. Detailed Implementation

[0020] To make the objectives, technical solutions, and effects of this application clearer and more explicit, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. The described embodiments are only possible technical implementations of this application and not all possible implementations. Based on the embodiments in this application, those skilled in the art can obtain other embodiments without creative effort, and these embodiments are also within the protection scope of this application.

[0021] First, let's introduce the terms used in the embodiments of this application: KNNimpute, K-Nearest-Neighbor Imputation, is a K-nearest neighbor imputation algorithm that fills in missing K-nearest neighbor values. LightGBM, Light Gradient Boosting Machine, is a lightweight gradient boosting machine model. DDM, Drift Detection Method, is a concept of drift detection method, i.e., drift detection algorithm; SHAP, or SAPley Additive exPlanation, is a feature attribution algorithm.

[0022] First, existing product recommendation systems suffer from a lack of awareness of environmental changes: most are rigid architectures lacking feedback loops. In the product market, customer preferences are greatly affected by policy, seasonality, and economic fluctuations (i.e., concept drift). Current recommendation systems generally lack drift detection capabilities, causing prediction accuracy to decline subtly after a period of operation due to the inability to detect changes in data distribution, resulting in model aging. Developers often have to address this through periodic manual maintenance, which is significantly lagging. Second, recommendation results lack interpretability: existing black-box recommendation systems are mostly limited to simple single-point model predictions, failing to systematically reverse-engineer and reconstruct customer profiles within the product's business context. Therefore, the system often only provides cold product rankings, making it difficult for operators to intuitively understand the logic behind the recommendations (i.e., why they are recommended). This lack of multi-dimensional customer characteristics in actual communication hinders the achievement of high-quality personalized marketing.

[0023] Current product recommendation solutions mostly focus on improving algorithm accuracy, neglecting the maintenance of model timeliness. This invention provides a self-evolving framework through a drift detection module, which has extremely high technical barriers and commercial value in the current agent distribution field, which lacks a continuous feedback loop.

[0024] This application introduces a DDM-based drift detection module to continuously monitor prediction errors in the real-time feedback stream. Model retraining is triggered only when a statistically significant drift is detected, thus avoiding the ineffective computation and resource waste caused by traditional periodic full updates. A visualization and explanation module is also introduced, generating customer profiles and radar chart explanations through SHAP feature attribution, enhancing the interpretability and decision-making support capabilities of the recommendation system in product business scenarios. A closed-loop adaptive mechanism of recommendation-feedback-drift detection automatic retraining is constructed, enabling the system to dynamically evolve according to market changes, reducing the frequency of manual maintenance, and improving long-term operational stability and computational resource utilization efficiency. Therefore, this application not only improves the timeliness and interpretability of the product recommendation system but also addresses the technical deficiencies of existing static recommendation systems in terms of computational resource utilization, model maintenance costs, and online adaptive capabilities.

[0025] This application's embodiments introduce a real-time drift detection module based on the DDM (Drift Detection Method), breaking through the static limitations of traditional recommendation systems. It not only processes missing data through KNNimpute and generates initial recommendations using a machine learning prediction model (LightGBM), but more importantly, it continuously monitors the feedback stream from the business side. When it detects abnormal recommendation accuracy (i.e., interest drift), it automatically triggers an alert and initiates model retraining logic. This closed-loop mechanism ensures that the recommendation model can dynamically evolve with the market environment, always maintaining a high level of prediction accuracy.

[0026] It is worth noting that this application introduces a drift detection module (DDM) to detect in real time whether the product recommendation system has experienced concept drift, and automatically triggers a retraining mechanism when an abnormal decline in model performance is detected, thereby achieving a self-evolutionary closed loop for the recommendation model; and introduces a visualization explanation module to generate customer profiles and recommendation explanations through SHAP feature attribution, thereby enhancing the business interpretability of product recommendation results.

[0027] This application also uses LightGBM as the recommendation engine to reduce the computational overhead in large-scale product recommendation scenarios; the drift detection module monitors only based on real-time error rate statistics, eliminating the need for frequent full retraining, thereby reducing system maintenance costs; and the model is only triggered when a significant drift is detected, which reduces unnecessary training resource consumption compared to periodic manual maintenance.

[0028] The technical solutions of this application will be described in detail below with reference to specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.

[0029] The adaptive information recommendation data processing method described in the preferred embodiment of this application, such as... Figure 1 As shown, the adaptive information recommendation data processing method includes the following steps: In step S10, the original feature vector of the user with missing values ​​is obtained, and the missing value interpolation process is performed on the original feature vector to obtain a complete dense feature vector without missing values.

[0030] In one possible implementation, step S10 specifically includes: performing missing mask identification on the original feature vector to obtain a known feature set and a missing index set; obtaining reference samples from historical complete samples, and calculating the mask Euclidean distance between the current sample and each historical sample based on the reference samples and the known feature set to obtain a distance value set; determining a nearest neighbor sample set based on the distance value set; obtaining an estimated imputation value set based on the nearest neighbor sample set and the missing index set; and merging and recombining the estimated imputation value set and the original feature vector to obtain a complete dense feature vector.

[0031] Specifically, such as Figure 2 As shown, the data processing module, acting as the entry point of the pipeline, is responsible for transforming the incomplete customer feature vectors entered by operators into dense vectors that can be processed by the model. Let the original input feature vector be... , For the first Given a set of original features, the system first needs to identify the known feature set. ,in, Indexed by feature dimensions, For new customers in known dimensions The feature values ​​on the vector. For example, the original feature vector. It is a set of customer attributes containing missing values ​​(e.g., [35 years old, male, null]), while the feature set This is a set of indices (e.g., {1, 2}) that mark the locations of these known attributes. Subsequently, the system retrieves samples from the complete historical sample database. And calculate the mask Euclidean distance between the two: ; in, Represents the current customers to be recommended and the first The spatial distance between historical reference customers, the smaller the value, the closer the background characteristics (such as age, occupation, etc.) of the two are; New customers in known dimensions The feature values, and This refers to the true value of the historical reference sample in the corresponding dimension.

[0032] After obtaining the distance sorting, select the nearest one. A set of neighbors For missing dimensions After performing weighted interpolation, this module outputs the complete dense feature vector. It flows to the next level.

[0033] In one possible implementation, the step of calculating the estimated imputation value set specifically includes: for each missing dimension in the missing index set, calculating the weight coefficient of each nearest sample based on the distance values ​​between each nearest sample in the nearest sample set and the current sample. ; in, For the first The weight coefficients of each neighbor are determined by their distance. An inverse relationship ensures that more similar samples contribute more; For the first The mask Euclidean distance between the current sample and its nearest neighbor samples. It is a preset minimal smoothing constant used to avoid the computational risk of the denominator being zero; Based on the weighting coefficients and the known values ​​of each nearest sample in the nearest sample set for the missing dimension, calculate the estimated imputation value for the missing dimension: ; in, For missing dimensions The estimated imputation value (system-generated imputation feature value). Given the known values ​​of the neighbor samples in the missing dimension. For the nearest neighbor sample set; Iterate through all missing dimensions in the missing index set to obtain a set of estimated imputation values ​​formed by the estimated imputation values ​​of each missing dimension.

[0034] In step S20, the dense feature vector is input into the currently deployed machine learning prediction model for inference, outputting the probability distribution for each candidate information item, and generating information recommendation result data and prediction result labels based on the probability distribution.

[0035] It should be noted that the machine learning prediction model in this embodiment is the LightGBM prediction model.

[0036] In one possible implementation, the step of outputting the probability distribution for each candidate information item specifically includes: The complete dense feature vector is input into the currently deployed machine learning prediction model, and the original score vector of each candidate information item is output: ; in, This is the original score vector output. No. The original scores of each candidate information item Measuring customer satisfaction with the third Potential insurance intentions for each product For a complete dense feature vector, This refers to the set of parameters for the current model under the current model version. The original score vector is normalized using the Softmax function, mapping the original scores of each candidate information item to a probability distribution: ; in, To complete dense feature vectors Under the conditions The probability of each candidate information item. It is an exponential function with the natural constant as its base. The sum of the original scores for all candidate information items. For the first Original scores for each candidate product; By traversing all candidate information items, the probability distribution vector formed by the probabilities of each candidate information item is obtained.

[0037] Specifically, such as Figure 2 As shown, the recommendation generation module will generate the repaired dense feature vector. Input to a pre-built prediction function (This solution is implemented using the LightGBM framework.) This function, based on the currently learned feature association weights, is within the system library. For each candidate product, a raw predicted strength value (i.e., Logit value) is calculated.

[0038] To eliminate the dimensional differences between the original scores of various products and to give the results an intuitive business meaning, a Softmax transformation logic is introduced. This transformation converts the discrete original scores... This is mapped to a globally unified insurance probability distribution, ensuring that the sum of the probabilities of all candidate products is 1, thereby enabling a quantitative comparison of customer preferences. Based on the obtained probability distribution... Sort the products in descending order and select the ones with the highest scores. The specific insurance product will be provided to the operators as a final recommendation.

[0039] The significant difference between this module and the traditional static model lies in the function. Parameter set in This is not a fixed value. Once the backend drift detection module detects that the prediction deviation exceeds the preset threshold through real-time comparison (i.e., it captures market interest drift), the system will trigger a retraining mechanism. Perform automatic optimization and iteration.

[0040] Once the drift detection module detects a shift in market preferences (i.e., the prediction bias exceeds a threshold), the system immediately initiates a parameter evolution process. Its core logic is to automatically update the function by minimizing the prediction loss using the latest customer feedback samples. parameter set For details, please refer to step S40.

[0041] In step S30, the user's real feedback label is obtained, the predicted result label is compared with the user's real feedback label, and an error indicator variable for the current sample is generated. The current cumulative error rate and standard deviation are calculated based on the accumulated error indicator variables, and the historical best error rate benchmark is determined. The current cumulative error rate and the standard deviation are compared with the historical best error rate benchmark using statistical thresholds. If the first judgment condition is met, a warning log is generated. If the second judgment condition is met, a retraining trigger signal is generated.

[0042] In one possible implementation, each of the error indicator variables takes the value of 1 or 0; the steps of calculating the current cumulative error rate and standard deviation specifically include: The current cumulative error rate is calculated based on the accumulated error indicator variables: ; in, The current cumulative error rate measures the average prediction bias of the pipeline since the last update. This is an error indicator variable; it takes the value 1 if the predicted recommended product does not match the product actually signed by the customer, and 0 otherwise. This represents the total number of cumulative feedback samples. This is the loop variable in the summation formula; The standard deviation is calculated based on the accumulated error indicator variables: ; in, The standard deviation is denoted as .

[0043] In one possible implementation, the first determination condition is: ; The second determination condition is: ; in, The minimum error rate among the historical best error rate benchmarks. This represents the standard deviation corresponding to the minimum error rate.

[0044] Specifically, see Figure 2The drift detection module in this application drives the recommendation engine to update itself by continuously monitoring the feedback flow from the business side. When the first... When a feedback sample is received, the system calculates the real-time error rate. and its standard deviation .

[0045] Continuously maintain historical performance low points And execute the warning and retraining logic: Warning logic: if the condition is met... The condition is determined to be a suspected drift; retraining logic: if the following conditions are met... This was determined to be a significant interest drift.

[0046] In this logic, and This serves as the baseline for the lowest prediction error rate achieved by the system during stable operation. Combined with a preset confidence level (2x or 3x), it forms the dynamic trigger threshold. Once the retraining logic is triggered, the system will automatically regenerate the model parameters. This completes the self-evolutionary closed loop of the entire recommendation chain.

[0047] In step S40, in response to the retraining trigger signal, the latest accumulated feedback sample set is obtained as training data, the parameters of the machine learning prediction model are optimized based on the training data, updated model parameters are generated, the updated model parameters are deployed to the machine learning prediction model, and the historical best error rate benchmark is reset to the statistical initial value at the current moment.

[0048] In one possible implementation, the step of generating updated model parameters specifically includes: optimizing the parameters of the machine learning prediction model based on the training data with the objective of minimizing the prediction loss, thereby generating updated model parameters, wherein the optimization objective function is: ; in, For the updated model parameters, The model parameters to be optimized are... For the first The actual feedback labels of each training sample (real customer contract feedback). For machine learning prediction models, the prediction function is... For the first The complete dense feature vector of each training sample.

[0049] See Figure 3 , Figure 4 and Figure 5 To make the recommendation results interpretable, the black-box weights of the decision-making layer are transformed into a radar profile that is perceptible to the business. Figure 2The customer profile radar chart generated for the first product uses multiple business feature dimensions of the customer as coordinate axes. The value on each coordinate axis represents the marginal contribution of that feature to recommending the product, and after Min-Max standardization, it is mapped to the [0,1] interval. The three radar charts together constitute the core output of the visualization and explanation module.

[0050] First, calculate each business characteristic. Contribution weight: ; in, Representing the The marginal contribution of each feature (such as driving experience or vehicle price) to the current recommendation score; For the total feature set, For features not included Any subset of features; The model represents only a subset Expected prediction at input.

[0051] After obtaining the contribution ranking, the system extracts core features and performs Min-Max standardization to eliminate the differences in measurement units between different business indicators: ; in, It is the calculated normalized score, which directly corresponds to the vertex coordinates on the radar chart; and These are the minimum and maximum values ​​of the feature in the full sample database, respectively. This module assists operators in decision-making through visual feedback, while the customer's final contract signing feedback will activate the monitoring layer.

[0052] This application pioneers a drift monitoring closed loop. Unlike all existing static recommendation systems, this solution deeply integrates the DDM detection mechanism with product recommendation for the first time, solving the industry problem of model aging immediately after deployment. Proactive model retraining: The system no longer relies on manual periodic maintenance but autonomously determines the retraining timing through statistical thresholds, achieving low-cost, highly sensitive adaptive updates. A combination of high fault tolerance and interpretability: Based on drift detection, it incorporates KNN missing value imputation and SHAP feature attribution to construct a closed-loop system covering the entire lifecycle from data input to model update and front-end interpretation.

[0053] It is understood that the drift detection in this application embodiment is not limited to DDM, and can also be replaced by ADWIN (Adaptive Window Method) or EDDM. Retraining can be performed using full data retraining or an incremental retraining mode based on weight decay. Although the recommendation engine focuses on monitoring, the execution engine LightGBM can also be replaced by machine learning prediction models such as XGBoost or CatBoost that can predict candidate information items.

[0054] Next, referring to the accompanying drawings, an adaptive information recommendation data processing system according to an embodiment of this application is described, which is used to implement the adaptive information recommendation data processing method described in any of the above schemes.

[0055] Figure 6 This is a structural diagram of the adaptive information recommendation data processing system according to an embodiment of this application.

[0056] like Figure 6 As shown, the adaptive information recommendation data processing system includes: a data processing module 100, a recommendation generation module 200, a drift detection module 300, and an automatic retraining module 400.

[0057] Specifically, the data processing module 100 is used to obtain the user's original feature vector with missing values, and to perform missing value interpolation on the original feature vector to obtain a complete dense feature vector without missing values. The recommendation generation module 200 is used to input the dense feature vector into the currently deployed machine learning prediction model for inference, output the probability distribution for each candidate information item, and generate information recommendation result data and prediction result labels based on the probability distribution. The drift detection module 300 is used to acquire user real feedback labels, compare the predicted result labels with the user real feedback labels, generate error indicator variables for the current sample, calculate the current cumulative error rate and standard deviation based on the accumulated error indicator variables, determine the historical best error rate benchmark, compare the current cumulative error rate and standard deviation with the historical best error rate benchmark using statistical thresholds, and generate a warning log if the first judgment condition is met, and generate a retraining trigger signal if the second judgment condition is met. The automatic retraining module 400 is used to respond to the retraining trigger signal, acquire the latest accumulated feedback sample set as training data, optimize the parameters of the machine learning prediction model based on the training data, generate updated model parameters, deploy the updated model parameters to the machine learning prediction model, and reset the historical best error rate benchmark to the statistical initial value at the current moment.

[0058] Figure 7A structural diagram of a terminal provided in an embodiment of this application. The terminal may include: The memory 501, the processor 502, and the computer program stored on the memory 501 and capable of running on the processor 502.

[0059] When the processor 502 executes the program, it implements the adaptive information recommendation data processing method provided in the above embodiments.

[0060] Furthermore, the terminal also includes: Communication interface 503 is used for communication between memory 501 and processor 502.

[0061] The memory 501 is used to store computer programs that can run on the processor 502.

[0062] Memory 501 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.

[0063] If the memory 501, processor 502, and communication interface 503 are implemented independently, then the communication interface 503, memory 501, and processor 502 can be interconnected via a bus to complete communication between them. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EIS) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, Figure 7 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0064] Optionally, in a specific implementation, if the memory 501, processor 502, and communication interface 503 are integrated on a single chip, then the memory 501, processor 502, and communication interface 503 can communicate with each other through an internal interface.

[0065] Processor 502 may be a central processing unit (CPU), an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of this application.

[0066] This embodiment also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the adaptive information recommendation data processing method described above.

[0067] One embodiment of this application provides a computer program product, including a computer program that, when executed by a processor, implements the features described in this application. Figure 1 The adaptive information recommendation data processing method provided in any of the corresponding embodiments.

[0068] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0069] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "N" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0070] Any process or method described in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or N executable instructions for implementing custom logic functions or processes, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as should be understood by those skilled in the art to which embodiments of this application pertain.

[0071] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable storage medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable storage medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable storage media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Alternatively, the computer-readable storage medium could be paper or other suitable media on which the program can be printed, since the program can be obtained electronically by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in a computer memory.

[0072] It should be understood that the various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, the N steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0073] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0074] Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0075] The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc. Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of this application.

[0076] It should be understood that the application of this application is not limited to the examples above. Those skilled in the art can make improvements or modifications based on the above description, and all such improvements and modifications should fall within the protection scope of the appended claims.

[0077] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. An adaptive information recommendation data processing method, characterized in that, The adaptive information recommendation data processing method includes: Obtain the original feature vector of the user with missing values, and perform missing value imputation on the original feature vector to obtain a complete dense feature vector without missing values; The dense feature vector is input into the currently deployed machine learning prediction model for inference, outputting the probability distribution for each candidate information item, and generating information recommendation result data and prediction result labels based on the probability distribution; Obtain real user feedback tags, compare the predicted result tags with the real user feedback tags, generate error indicator variables for the current sample, calculate the current cumulative error rate and standard deviation based on the accumulated error indicator variables, determine the historical best error rate benchmark, compare the current cumulative error rate and standard deviation with the historical best error rate benchmark using statistical thresholds, if the first judgment condition is met, generate a warning log, if the second judgment condition is met, generate a retraining trigger signal; In response to the retraining trigger signal, the latest accumulated feedback sample set is obtained as training data, the parameters of the machine learning prediction model are optimized based on the training data, updated model parameters are generated, the updated model parameters are deployed to the machine learning prediction model, and the historical best error rate benchmark is reset to the statistical initial value at the current moment.

2. The adaptive information recommendation data processing method according to claim 1, characterized in that, The step of performing missing value imputation on the original feature vector to obtain a complete dense feature vector without missing values ​​specifically includes: The original feature vector is subjected to missing mask identification to obtain the known feature set and the missing index set; Obtain reference samples from the complete historical samples, and calculate the masked Euclidean distance between the current sample and each historical sample based on the reference samples and the known feature set to obtain a set of distance values; Determine the nearest neighbor sample set based on the distance value set; Based on the nearest neighbor sample set and the missing index set, an estimated imputation value set is obtained; The estimated imputation value set and the original feature vector are merged and recombined to obtain a complete dense feature vector.

3. The adaptive information recommendation data processing method according to claim 2, characterized in that, The step of obtaining the estimated imputation value set based on the nearest neighbor sample set and the missing index set specifically includes: For each missing dimension in the missing index set, the weight coefficient of each nearest sample is calculated based on the distance between each nearest sample in the nearest sample set and the current sample: ； in, For the first The weight coefficients of each neighbor, For the first The mask Euclidean distance between the current sample and its nearest neighbor samples. This is a preset, minimal smoothing constant; Based on the weighting coefficients and the known values ​​of each nearest sample in the nearest sample set for the missing dimension, calculate the estimated imputation value for the missing dimension: ； in, For missing dimensions The estimated fill value, Given the known values ​​of the neighbor samples in the missing dimension. For the nearest neighbor sample set; Iterate through all missing dimensions in the missing index set to obtain a set of estimated imputation values ​​formed by the estimated imputation values ​​of each missing dimension.

4. The adaptive information recommendation data processing method according to claim 2, characterized in that, The step of inputting the dense feature vector into the currently deployed machine learning prediction model for inference and outputting the probability distribution for each candidate information item specifically includes: The complete dense feature vector is input into the currently deployed machine learning prediction model, and the original score vector of each candidate information item is output: ； in, This is the original score vector output. No. The original scores of each candidate information item For a complete dense feature vector, This is the current set of model parameters; The original score vector is normalized using the Softmax function, mapping the original scores of each candidate information item to a probability distribution: ； in, To complete dense feature vectors Under the conditions The probability of each candidate information item. It is an exponential function with the natural constant as its base. The sum of the original scores for all candidate information items. For the first Original scores for each candidate product; By traversing all candidate information items, the probability distribution vector formed by the probabilities of each candidate information item is obtained.

5. The adaptive information recommendation data processing method according to claim 1, characterized in that, Each of the aforementioned error indicator variables takes the value of 1 or 0; The calculation of the current cumulative error rate and standard deviation based on the accumulated error indicator variables specifically includes: The current cumulative error rate is calculated based on the accumulated error indicator variables: ； in, This represents the current cumulative error rate. For error indicator variables, This represents the total number of cumulative feedback samples. This is the loop variable in the summation formula; The standard deviation is calculated based on the accumulated error indicator variables: ； in, The standard deviation is denoted as .

6. The adaptive information recommendation data processing method according to claim 5, characterized in that, The first determination condition is: ； The second determination condition is: ； in, The minimum error rate among the historical best error rate benchmarks. This represents the standard deviation corresponding to the minimum error rate.

7. The adaptive information recommendation data processing method according to claim 1, characterized in that, The step of optimizing the parameters of the machine learning prediction model based on the training data to generate updated model parameters specifically includes: With the goal of minimizing prediction loss, the parameters of the machine learning prediction model are optimized based on the training data to generate updated model parameters, wherein the optimization objective function is: ； in, For the updated model parameters, The model parameters to be optimized are... For the first The true feedback labels of each training sample For machine learning prediction models, the prediction function is... For the first The complete dense feature vector of each training sample.

8. An adaptive information recommendation data processing system, characterized in that, The adaptive information recommendation data processing system is used to implement the adaptive information recommendation data processing method according to any one of claims 1-7, wherein the adaptive information recommendation data processing system comprises: The data processing module is used to obtain the user's original feature vector with missing values, and to perform missing value imputation processing on the original feature vector to obtain a complete dense feature vector without missing values. The recommendation generation module is used to input the dense feature vector into the currently deployed machine learning prediction model for inference, output the probability distribution for each candidate information item, and generate information recommendation result data and prediction result labels based on the probability distribution. The drift detection module is used to acquire user real feedback labels, compare the predicted result labels with the user real feedback labels, generate error indicator variables for the current sample, calculate the current cumulative error rate and standard deviation based on the accumulated error indicator variables, determine the historical best error rate benchmark, compare the current cumulative error rate and standard deviation with the historical best error rate benchmark using statistical thresholds, and generate a warning log if the first judgment condition is met, and generate a retraining trigger signal if the second judgment condition is met. An automatic retraining module is used to respond to the retraining trigger signal, acquire the latest accumulated feedback sample set as training data, optimize the parameters of the machine learning prediction model based on the training data, generate updated model parameters, deploy the updated model parameters to the machine learning prediction model, and reset the historical best error rate benchmark to the statistical initial value at the current moment.

9. A terminal, characterized in that, The terminal includes: a memory, a processor, and an adaptive information recommendation data processing program stored in the memory and executable on the processor. When the adaptive information recommendation data processing program is executed by the processor, it implements the steps of the adaptive information recommendation data processing method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores an adaptive information recommendation data processing program, which, when executed by a processor, implements the steps of the adaptive information recommendation data processing method as described in any one of claims 1-7.

Images