Chemical and plastics product personalized recommendation method based on deep reinforcement learning

Abstract

The present invention relates to the technical field of chemical and plastics product recommendation, provides a chemical and plastics product personalized recommendation method based on deep reinforcement learning, comprising: S1 receiving a search condition of a user; S2, constructing a user state vector; S3, obtaining a candidate product set on the basis of the user state vector and the search condition; S4, using a pre-trained deep reinforcement learning model to sort candidate products, and generating a recommendation result; S5, displaying the recommendation result to the user, and acquiring user feedback; and S6, determining whether a recommendation stop condition is met, if yes, stopping recommending, and outputting a final recommendation result, otherwise, updating parameters of the deep reinforcement learning model and the user state vector on the basis of the user feedback, and returning to step S3. The present invention can improve the correlation, diversity and timeliness of the recommendation result, better meet the personalized needs of a user in the chemical and plastics industry, and can continuously learn and adapt to changes in user needs.

Description

Chemical and plastic product personalized recommendation method based on deep reinforcement learning TECHNICAL FIELD

[0001] The present application relates to the technical field of chemical and plastic product recommendation, and particularly relates to a chemical and plastic product personalized recommendation method based on deep reinforcement learning. BACKGROUND

[0002] With the rapid development of chemical and plastic industries, the types of chemical and plastic products are increasingly diverse, and market demand is constantly changing. In order to help enterprises and users better select suitable chemical and plastic products, personalized recommendation technology has become increasingly important in this field. Currently, chemical and plastic product recommendation mainly adopts two methods: content-based recommendation and collaborative filtering recommendation. Content-based recommendation matches product attributes (such as chemical composition, physical properties, application fields, etc.) and user preferences, which can handle new products, but it is difficult to capture users' potential interests. Collaborative filtering analyzes users' historical behavior data to find relationships between similar users or similar products, which can discover users' potential interests, but performs poorly when dealing with new users or new products.

[0003] In addition, there are knowledge-based recommendation and hybrid recommendation methods. Knowledge-based recommendation uses the knowledge of domain experts to build a knowledge base or ontology model of chemical and plastic products, and makes recommendations based on the relationships between products (such as upstream and downstream relationships, substitution relationships, etc.), which can provide more professional recommendations, but the cost of building and maintaining the knowledge base is high. Hybrid recommendation methods combine the advantages of multiple technologies, such as combining content features and collaborative filtering, or combining knowledge bases with machine learning algorithms, which can compensate for the shortcomings of single technologies to some extent and improve recommendation effectiveness. In recent years, deep learning technology has also been widely used in recommendation systems, such as using deep neural networks to learn low-dimensional representations of users and products, or using recurrent neural networks to model users' sequential behavior, which can automatically learn complex features and improve the accuracy of recommendations.

[0004] However, existing chemical and plastic product recommendation technologies still have some limitations. These methods often have difficulty in considering multiple factors such as professional attributes of chemical and plastic products, industry chain relationships, user enterprise characteristics, and long and short-term user interests, resulting in less than ideal relevance, diversity, and timeliness of the recommended results. SUMMARY

[0005] In view of this, the present application provides a plastic product personalized recommendation method based on deep reinforcement learning, which realizes the dynamic optimization of the plastic product recommendation process by constructing a comprehensive user state representation, expanding the candidate set using a plastic product knowledge graph, designing a multi-level action space, using a multi-scale Q network to capture the value of different time scales, and designing a detailed reward function. The present application aims to improve the relevance, diversity and timeliness of the recommendation results, better meet the personalized needs of the plastic industry users, and continuously learn and adapt to changes in user demand.

[0006] The technical solution of the present application is implemented as follows:

[0007] The present application provides a plastic product personalized recommendation method based on deep reinforcement learning, comprising:

[0008] S1 receives the search conditions input by the user on the plastic product platform;

[0009] S2 constructs a user state vector according to the search conditions;

[0010] S3 filters a preliminary product set that meets the conditions from the plastic product database based on the user state vector and the search conditions, and expands the preliminary product set according to the upstream and downstream relationships of the products to obtain a candidate product set;

[0011] S4 uses a pre-trained deep reinforcement learning model to input the user state vector and the candidate product set, sorts the candidate products, and generates a recommendation result;

[0012] S5 shows the recommendation result to the user and obtains user feedback;

[0013] S6 judges whether the recommendation stopping condition is reached, if yes, stops the recommendation, outputs the final recommendation result, if not, updates the parameters of the deep reinforcement learning model and the user state vector according to the user feedback, and returns to step S3.

[0014] Preferably, step S2 comprises:

[0015] S21 parses the search conditions, extracts keywords and attribute requirements, and the attribute requirements include product categories, performance indicators, processing techniques and application fields;

[0016] S22 obtains user enterprise information, including enterprise size, enterprise type, position in the industrial chain and main product categories;

[0017] S23 extracts user historical behavior data, including recent search data, historical purchase data, historical conversation data and historical preference data;

[0018] S24 obtains current conversation context information, including time features, geographic features and market features;

[0019] S25 constructs a product association matrix representing the upstream and downstream relationship strength between different chemical products;

[0020] S26 converts the keywords, attribute requirements, user enterprise information, user historical behavior data, current session context information, and product association matrix into numerical features to construct an initial feature vector;

[0021] S27 uses an attention mechanism to weight each part of the initial feature vector to obtain a weighted feature vector;

[0022] S28 inputs the weighted feature vector into a pre-trained autoencoder network to obtain a low-dimensional user state vector representation.

[0023] Preferably, in step S28, the autoencoder network adopts a variational autoencoder structure, and the loss function during pre-training is as follows:

[0024] L VAE =L recon +L sim +L chain

[0025] L sim =∑max(0,d(z i ,z j )-α×d(z i ,z k )+m)

[0026] L chain =∑ | f(z a ,z b )-r(a,b) p

[0027] f(z a ,z b )=σ(W·[z a ;z b ;z a ⊙z b ])

[0028] In the formula, L VAE is the total loss of the variational autoencoder; the first part L recon represents the reconstruction loss, x i is the original feature of the input data, x' i is the reconstructed feature, and w i is the weight based on the importance of the chemical product attribute; the second part L sim represents the product similarity loss, z i , z jz k is the encoded feature vector, (i, j) is the similar product pair, k is the randomly selected dissimilar product, d(·) denotes the distance metric in the feature space, m is the marginal parameter, a is the adjustable similarity proportion coefficient, v d denotes the importance weight of the dth feature dimension, which is predefined according to the characteristics of the plastic product, denotes the encoded feature vector of the dth dimension; the third part L chain denotes the upstream and downstream industrial chain relationship loss, z a z b is the encoded feature vector, f(·) denotes the industrial chain relationship prediction function, r(a, b) is the relationship strength of products a and b in the actual industrial chain, p is an adjustable exponential parameter, W is a learnable weight matrix, [;] denotes vector splicing, ⊙ denotes element-wise product, and σ is a sigmoid activation function;

[0029] wherein, the reconstruction loss L recon measures whether the variational autoencoder accurately retains the basic attribute information of the product and whether the low-dimensional representation after encoding contains the key information of the original data; the product similarity loss L sim measures whether the products with similar performance indicators or application fields are close in the encoding space, and whether the products of different categories but can be mutually replaced maintain similarity in the feature space; the upstream and downstream industrial chain relationship loss L chain measures whether the encoding space retains the industrial chain relationship between products.

[0030] Preferably, step S3 comprises:

[0031] S31, according to the keywords and attribute requirements obtained by analyzing the search conditions, constructs a database query statement containing fuzzy matching and accurate matching conditions, executes a query in the plastic product database, obtains a product list satisfying the conditions, preliminarily sorts the query results, the sorting basis is the comprehensive matching degree with the search conditions, and selects the top-M products after sorting as a preliminary product set;

[0032] S32, a preset plastic product knowledge graph is called, the nodes in the graph represent products, and the edges represent the relationships between products, wherein the relationship types between products include upstream and downstream relationships, substitution relationships, common relationships, and performance similarity relationships;

[0033] S33, each product node in the preliminary product set is located in the plastic product knowledge graph, and from each located node, a weighted random walk is performed, all product nodes visited by the walk are added to the preliminary product set, and duplicate products are removed to obtain a preliminary candidate set;

[0034] S34 calculate the similarity between the user state vector and each product in the preliminary candidate set, sort the products according to the similarity, and select the top-N products with the highest similarity as the candidate product set;

[0035] S35 organize the candidate product set into a three-layer structure, the first layer is the product category, the second layer is the specific product type, and the third layer is the specific specification or model, and add metadata to each product, including its adjacent node information in the plastic product knowledge graph.

[0036] Preferably, the deep reinforcement learning model comprises:

[0037] The state space includes user feature vectors, current session information, context information, and recommended product lists, wherein the current session information includes search conditions and interaction history.

[0038] The action space is designed in a three-layer structure, and the action definition of each layer is: the first layer selects the product category, the second layer selects the specific product type, and the third layer selects the specific specification or model.

[0039] The state encoder takes as input the information in the state space, and outputs a fixed-dimensional state representation vector.

[0040] The action encoder takes as input the information of each product in the candidate product set, and outputs a fixed-dimensional action representation vector.

[0041] The policy network takes as input the state representation vector and the action representation vector, and outputs the probability distribution of selecting each action in the current state.

[0042] The value network takes as input the state representation vector, and outputs the value estimate of the current state.

[0043] The multi-scale Q network includes three Q networks for short-term, medium-term and long-term, which take as input the state representation vector and the action representation vector, and output the Q value corresponding to the time scale.

[0044] Preferably, step S4 comprises:

[0045] S41 merge the user state vector, the current session information, the context information, and the recommended product list, and use the state encoder to encode the merged information into a fixed-dimensional state representation vector s.

[0046] S42 for each product in the candidate product set, extract its feature information, and use the action encoder to encode each product into a fixed-dimensional action vector representation a.

[0047] S43 inputs the state representation vector s into the first layer policy network, outputs the probability distribution π1(a|s) on all product categories, selects a product category using the ε-greedy strategy; merges the state representation vector s and the selected product category information, inputs into the second layer policy network, outputs the probability distribution π2(a|s) on all product types under the selected category, selects a specific product type using the ε-greedy strategy; merges the state representation vector s, the selected product category and the specific product type information, inputs into the third layer policy network, outputs the probability distribution π3(a|s) on all specific specifications or models under the selected specific product type, selects a specific specification or model using the ε-greedy strategy;

[0048] S44 for each selected specific product, respectively calculates the Q value Q s (s,a)、Q m (s,a)、Q l (s,a), and uses the attention mechanism to weight the Q values of the three time scales:

[0049] Q(s,a)=Attention(Q s (s,a),Q m (s,a),Q l (s,a))

[0050] Wherein, Q s (s,a) represents the short-term Q network, Q m (s,a) represents the medium-term Q network, and Q l (s,a) represents the long-term Q network.

[0051] S45 calculates the comprehensive score S for each candidate product, sorts the candidate products in descending order according to the comprehensive score S, and selects the top-K products in the order as the preliminary recommendation result;

[0052] S46 reorders the preliminary recommendation result using the maximum marginal relevance algorithm, calculates the similarity matrix between products, iteratively selects products that are both relevant and diverse, and obtains the current recommendation result.

[0053] Preferably, in step S45, the calculation formula of the comprehensive score S is:

[0054] S(a)=λ×(π1(a|s)×π2(a|s)×π3(a|s))+(1-λ)×Q(s,a)

[0055] Wherein, λ is a balance factor for adjusting the relative importance of the policy network and the Q value.

[0056] Preferably, the types of user feedback include click behavior, inquiry behavior, order behavior, explicit rating, browsing duration, collection behavior, comparison behavior, and sharing behavior.

[0057] Preferably, in step S6, the parameters of the deep reinforcement learning model and the user state vector are updated according to the user feedback, including:

[0058] According to the reward function, the comprehensive reward value R of each type of user feedback is calculated;

[0059] The Q value is updated using the TD learning method:

[0060] Q s (s,a)=Q s (s,a)+α s ×(R+γ s ×max(Q s (s',a')-Q s (s,a)))

[0061] Q m (s,a)=Q m (s,a)+α m ×(R+γ m ×max(Q m (s',a')-Q m (s,a)))

[0062] Q l (s,a)=Q l (s,a)+α l ×(R+γ l ×max(Q l (s',a')-Q l (s,a)))

[0063] Q(s,a)=Attention(Q s (s,a),Q m (s,a),Q l (s,a))

[0064] In the formula, s is the current state, a is the current action, s' is the next state, a' is the next possible action, α s , α m , α l are the learning rates of the short-term, medium-term, and long-term Q networks respectively, γ s , γ m , γ l are the discount factors of the short-term, medium-term, and long-term Q networks respectively, and γ s < γ m < γ l .

[0065] The parameters θ of the policy network are updated using a policy gradient method, and the parameters of the value network are updated using a TD learning method

[0066] The behavior information in the user feedback is integrated into the user state vector, and the integrated user state vector is reduced and reconstructed using an autoencoder network to obtain an updated user state vector.

[0067] Preferably, the reward function is as follows:

[0068] R = w1 x r c + w2 x r i + w3 x r o + w4 x r f + w5 x r r + w6 x r d + w7 x r n

[0069] In the formula, r c is a click reward item, r i is a quote reward item, r o is an order reward item, r f is an explicit rating reward item, r r is a relevance reward item, r d is a diversity reward item, r n is a novelty reward item, and w1 to w7 are corresponding weights.

[0070] Each reward item also corresponds to its reward value:

[0071] r c = {0.1 if the user clicks to view product details; 0.2 if the user clicks and stays for more than a threshold T1; 0.3 if the user clicks and further interacts; 0 if there is no click behavior};

[0072] r i = {0.3 if the user initiates an online quote; 0.4 if the user initiates a phone quote; 0.5 if the user communicates effectively with customer service after quoting, with a communication duration exceeding a threshold T2; 0 if there is no quote behavior};

[0073] r o = {1.0 if the user places an order; 1.2 if the user places an order and the order amount exceeds a threshold V1; 1.5 if the user places an order and it is the first purchase; 0 if there is no order behavior};

[0074] r f = (user rating - 3) / 2, with a user rating of 5 points;

[0075] r r= cos sim (product e , user sv ), where cos sim is the cosine similarity, product e is the embedding vector representation of the product, and user sv is the user state vector;

[0076] r d = 1 - max(cos sim (product e , prev rp )), where prev rp is the set of previously recommended products;

[0077] r n = 1 - pop(product), where pop(product) is the normalized popularity of the product.

[0078] The present application has the following beneficial effects over the prior art:

[0079] (1) By comprehensively considering the professional attributes of chemical and plastic products, the relationship of the industrial chain, the characteristics of user enterprises, and the long-term and short-term user interests, etc., the dynamic optimization of the recommendation process is realized. This method can improve the relevance, diversity and timeliness of the recommendation results, better meet the personalized needs of users in the chemical and plastic industry, and has the ability to continuously learn and adapt to changes in user needs;

[0080] (2) The variational autoencoder structure is used to construct the user state vector, and by introducing the reconstruction loss, product similarity loss and upstream and downstream industrial chain relationship loss, the encoded user state vector can better retain the basic attribute information, similarity and industrial chain relationship of the product. This method improves the accuracy and comprehensiveness of the user state representation, providing a more reliable input for the subsequent recommendation process;

[0081] (3) The candidate product set is expanded using the chemical and plastic product knowledge graph, and the upstream and downstream relationships, substitution relationships, common relationships and performance similarity relationships between products are considered through the method of weighted random walk. This method significantly increases the diversity and relevance of the recommendation results, and can provide users with more comprehensive and professional product selection;

[0082] (4) A three-layer structure of the action space is designed, corresponding to product categories, specific product types and specific specifications or models. This multi-level action design enables the recommendation system to gradually narrow down the recommendation range, improving the accuracy and efficiency of the recommendation, and also facilitates the system to understand and learn the user's selection logic;

[0083] (5) The multi-scale Q network is adopted, including three Q networks of short-term, medium-term and long-term, which can capture the value of user behavior at different time scales. This design enables the recommendation system to meet the immediate needs of users while also considering the long-term interests of users, thereby providing more balanced and sustainable recommendation results;

[0084] (6) A detailed reward function is designed, which comprehensively considers various behaviors of users such as clicks, inquiries, orders, and the relevance, diversity and novelty of the recommendation results. This comprehensive reward design can guide the system to more accurately understand and evaluate the real needs and satisfaction of users, thereby continuously optimizing the recommendation strategy. BRIEF DESCRIPTION OF DRAWINGS

[0085] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor.

[0086] Fig. 1 is a flowchart of the method of the present application;

[0087] Fig. 2 is a recommended process implementation diagram of the present application. DETAILED DESCRIPTION

[0088] The technical solutions in the embodiments of the present application will be described in detail below, and obviously, the described embodiments are only some of the embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0089] As shown in Figs. 1 and 2, the present application provides a deep reinforcement learning-based personalized recommendation method for plastic products, which includes:

[0090] S1 receives the search conditions input by the user on the plastic product platform;

[0091] S2 constructs a user state vector according to the search conditions;

[0092] S3 based on the user state vector and the search conditions, filters out a preliminary product set that meets the conditions from the plastic product database, and according to the upstream and downstream relationships of the products, expands the preliminary product set to obtain a candidate product set;

[0093] S4 uses a pre-trained deep reinforcement learning model, takes the user state vector and the candidate product set as input, sorts the candidate products, and generates a recommendation result;

[0094] S5 shows the recommendation result to the user and acquires user feedback;

[0095] S6 judges whether the recommendation stopping condition is reached, if yes, stops the recommendation, outputs the final recommendation result, if not, updates the parameters of the deep reinforcement learning model and the user state vector according to the user feedback, and returns to step S3.

[0096] In the chemical and plastic product platform, when recommending chemical and plastic products, the characteristics of the chemical and plastic products need to be considered, for example, there are various chemical and plastic products, including various resins, additives, and aids, each product has specific chemical and physical properties, and is suitable for specific application scenarios, the same type of product may have multiple specifications and models, the same type of product may have multiple specifications and models, and each product has specific technical indicators, such as melt index and density, therefore, the deep reinforcement learning technology is introduced to make recommendations, specifically, a user state vector is constructed to effectively represent the search intention and browsing history of the user when designing the state space, a three-layer action space is designed to adapt to the characteristics of the chemical and plastic products, a reward function with multiple reward items is designed to reflect the real needs of the user, and multiple time scale Q networks are used to balance between immediate feedback and long-term user value.

[0097] Specifically, the user inputs a search condition in the chemical and plastic product platform, the search condition may be a short keyword or phrase, such as "polypropylene", "impact-resistant PP", "food-grade HDPE", "transparent ABS", etc., or a sentence, for example, "high-strength PP for use in automobiles".

[0098] In an embodiment of the present application, step S2 includes:

[0099] S21 parses the search condition, extracts the keyword and attribute requirements, and the attribute requirements include product category, performance index, processing technology, and application field.

[0100] Specifically, when parsing the search condition, the NLP technology is used to parse the search condition, for example, for "high-strength PP for use in automobiles", the keywords "PP" and "polypropylene" are extracted, and the attribute requirements are identified: product category (PP), performance index (high strength), and application field (automobile). When parsing the search condition, the pre-set chemical and plastic product knowledge graph can also be called to identify professional terms or abbreviations. And the processing technology is identified according to the product standard document.

[0101] S22 acquires user enterprise information, including enterprise size, enterprise type, position in the industrial chain, and main product category.

[0102] Specifically, extract enterprise-related information from user registration information or third-party databases, such as enterprise size, which can be divided into large, medium, and small; enterprise type, which can include manufacturers, traders, and end users; position in the industrial chain, which can be upstream raw material suppliers, midstream processors, or downstream application manufacturers; and main product category, which can be specific chemical and plastic product categories.

[0103] S23 extracts user historical behavior data, including recent search data, historical purchase data, historical conversation data, and historical preference data.

[0104] Specifically, extract recent search records, historical purchase orders, and conversation records with customer service from user behavior logs, as well as user preference data obtained from long-term behavior analysis, such as commonly used product types, products of interest or collection, and performance indicators of interest.

[0105] S24 obtains current conversation context information, including time characteristics, geographic characteristics, and market characteristics.

[0106] Specifically, record the time of the user's current search, the geographic location corresponding to the user's IP address, and the overall market situation of the current chemical and plastic market, such as fluctuations in raw material prices, changes in supply and demand relationships, etc.

[0107] S25 constructs a product association matrix to represent the upstream and downstream relationship strength between different chemical and plastic products.

[0108] Specifically, based on the chemical and plastic product knowledge graph, a matrix is constructed to represent the upstream and downstream relationship strength between different products. The matrix is N*N, where N is the total number of chemical and plastic products in the knowledge graph. Each element M[i][j] in the matrix represents the relationship strength between product i and product j. For each element M[i][j] in the matrix, the relationship strength is calculated according to the following factors:

[0109] Direct relationship: If product i is a direct upstream or downstream of product j, give a high relationship strength (such as 0.8-1.0); if product i and j are direct substitution relationships, give a medium relationship strength (such as 0.6-0.8); if product i and j have common relationships or similar performance relationships, give a lower relationship strength (such as 0.4-0.6).

[0110] Indirect relationship: Use graph algorithms (such as shortest path) to calculate the distance between two products, and the relationship strength decreases with the increase of path length, which can use an exponential decay function.

[0111] Market factors: Consider the correlation of two products in the market, such as price linkage; the correlation coefficient obtained from historical data analysis can be used to adjust the relationship strength.

[0112] Allow domain experts to manually adjust the relationship strength of certain product pairs.

[0113] Normalize the calculated relationship strength to ensure all values are between 0 and 1. Since there may be no direct relationship between most products, set a threshold, such as 0.1, and set the relationship strength below this threshold to 0, resulting in a sparse matrix.

[0114] S26 converts the keywords, attribute requirements, user enterprise information, user historical behavior data, current session context information, and product association matrix into numerical features to construct an initial feature vector.

[0115] Specifically, convert the keywords and attribute requirements into one-hot encoding or word embedding vectors, convert the enterprise information into categorical features, convert the historical behavior data into statistical features, convert the context information into continuous features, and convert the product association matrix into a sparse matrix representation. Then concatenate these features into a high-dimensional initial feature vector.

[0116] S27 uses an attention mechanism to weight each part of the initial feature vector to obtain a weighted feature vector.

[0117] The system uses an attention mechanism to weight each part of the initial feature vector. This allows the system to dynamically adjust the importance of different features, for example, it may place more emphasis on current search conditions for new users, and more emphasis on historical behavior data for old users.

[0118] Specifically, divide the initial feature vector into K sub-vectors, corresponding to enterprise information, historical behavior, context information, etc. Assign a learnable attention weight to each sub-vector, calculate the weighted sum, and obtain a weighted feature vector. The attention weight is dynamically adjusted through the backpropagation process to adapt to different users.

[0119] S28 inputs the weighted feature vector into a pre-trained autoencoder network to obtain a low-dimensional user state vector representation.

[0120] In this embodiment, the autoencoder network adopts a variational autoencoder structure, and its pre-training loss function is as follows:

[0121] L VAE =L recon +L sim +L chain

[0122] L sim =∑max(0,d(z i ,z j )-α×d(z i ,zk )+m)

[0123] L chain =∑ | f(z a ,z b )-r(a,b) p

[0124] f(z a ,z b )=σ(W·[z a ;z b ;z a ⊙z b ])

[0125] In the formula, L VAE The total loss of the variational autoencoder; Part 1 L recon Let x represent the reconstruction loss. i x' represents the original features of the input data. i To reconstruct features, w i Weights based on the importance of chemical and plastic product attributes; Part Two L sim z represents the product similarity loss. i ,z j ,z k Let be the encoded feature vector, (i,j) be similar product pairs, k be randomly selected dissimilar products, d(·) represent the distance metric in the feature space, m be the marginal parameter, α be an adjustable similarity scaling factor, and v be the variable. d This represents the importance weight of the d-th feature dimension, which is predefined based on the characteristics of the chemical and plastic products. This represents the feature vector of the d-th dimension after encoding; the third part L chain This indicates losses in the upstream and downstream supply chain relationships, z a ,z b Let f(·) be the encoded feature vector, r(a,b) be the prediction function for the industrial chain relationship, r(a,b) be the strength of the relationship between products a and b in the actual industrial chain, p be an adjustable exponential parameter, W be a learnable weight matrix, [;] be the vector concatenation, ⊙ be the element-wise multiplication, and σ be the sigmoid activation function.

[0126] Among them, the reconstruction loss L recon The variational autoencoder measures whether it accurately preserves the basic attribute information of the product, and whether the encoded low-dimensional representation contains the key information of the original data; product similarity loss L sim This measures whether products with similar performance metrics or application areas are close in the coding space, and whether products of different categories but interchangeable remain similar in the feature space; the loss of upstream and downstream supply chain relationships (L) chainThe measurement of whether the coding space preserves the inter-product industrial chain relationship is specifically the measurement of whether the raw material and its derived product maintain a certain correlation in the coding space, and whether the relationship strength between the upstream and downstream products in the industrial chain is reflected in the coding space.

[0127] In this embodiment, the constructed loss function mainly focuses on reconstruction loss, similarity loss and industrial chain loss. The reconstruction loss can adjust the reconstruction accuracy according to the importance of the plastic product attribute, and at the same time, the influence of large error is reduced through the denominator item. The product similarity loss introduces an adjustable similarity proportion coefficient α, so that the model can more flexibly process different degrees of product similarity. The upstream and downstream industrial chain relationship loss uses a special relationship prediction function, which can capture complex industrial chain relationships and control the sensitivity of the loss through an adjustable exponential parameter p. Among them, the product similarity loss uses distance measurement, which considers the importance weight of the plastic product characteristics, so that the influence of key attributes can be highlighted when calculating the product similarity. The upstream and downstream industrial chain relationship loss uses an industrial chain relationship prediction function, which comprehensively considers the direct comparison, difference and interaction of product features, and can more comprehensively depict the industrial chain relationship between products. Training the autoencoder network with this loss function can not only effectively compress the feature dimension, but also accurately preserve the key attributes of plastic products, the similarity relationship between products and the upstream and downstream industrial chain relationship in the low-dimensional representation, thereby providing more accurate and meaningful feature representation for subsequent recommendation tasks.

[0128] Specifically, in an embodiment of the present application, step S3 comprises:

[0129] S31, according to the keywords and attribute requirements obtained by analyzing the search conditions, constructs a database query statement containing fuzzy matching and accurate matching conditions, executes a query in the plastic product database, obtains a product list satisfying the conditions, preliminarily sorts the query results, the sorting basis is the comprehensive matching degree with the search conditions, and selects the top-M products after sorting as a preliminary product set.

[0130] For example, for "high-strength PP used in cars", the keyword is "PP", and the attribute requirements include "high-strength" and "car application". The database query statement is constructed, which contains:

[0131] Fuzzy matching condition: such as '%PP%'. Accurate matching condition: such as strength>[high-strength threshold] AND application='car'.

[0132] Perform a query in the chemical and plastic product database to obtain a list of products that meet the conditions. Sort the query results preliminarily based on the comprehensive matching degree with the search conditions. A weighted scoring method can be used, such as: total score = w1 * keyword matching degree + w2 * attribute matching degree + w3 * product popularity. Select the top-M products after sorting as the preliminary product set. The value of M can be dynamically adjusted according to system performance and demand, such as M = 100.

[0133] S32 calls the preset chemical and plastic product knowledge graph, where the nodes represent products and the edges represent the relationships between products, including upstream and downstream relationships, substitution relationships, common relationships, and performance similarity relationships.

[0134] A chemical and plastic product knowledge graph is constructed and maintained in advance. The nodes in the graph represent various chemical and plastic products, such as PP, HDPE, ABS, etc. The edges represent the relationships between products, including:

[0135] Upstream and downstream relationships: such as crude oil-ethylene-polyethylene. Substitution relationships: such as PP and PE can be substituted for each other in certain applications. Common relationships: such as products using the same catalyst or production equipment. Performance similarity relationships: such as different brands of PP with similar mechanical properties. Each relationship may have a different weight reflecting the strength of the relationship.

[0136] S33 locates each product node in the preliminary product set in the chemical and plastic product knowledge graph, and performs a weighted random walk from each located node. All product nodes visited by the walk are added to the preliminary product set, and duplicate products are removed to obtain a preliminary candidate set.

[0137] Specifically, the weighted random walk is as follows:

[0138] Set the maximum number of steps, such as 5 steps, and the number of walks, such as 10 times; at each step, randomly select the next node based on the weight of the edge; record all nodes visited during the walk.

[0139] S34 calculates the similarity between the user state vector and each product in the preliminary candidate set, sorts the products based on the similarity, and selects the top-N products with the highest similarity as the candidate product set.

[0140] Specifically, the similarity calculation method is cosine similarity, and the value of N is set according to demand, such as N = 50.

[0141] S35 organizes the candidate product set into a three-layer structure, with the first layer being product categories, the second layer being specific product types, and the third layer being specific specifications or models. Add metadata to each product, including its adjacent node information in the chemical and plastic product knowledge graph.

[0142] Specifically, the metadata includes direct adjacent node information in the plastic product knowledge graph, a relationship type and intensity of the adjacent node and the current product.

[0143] In this embodiment, through these steps, the system can not only find products directly related to the user's search conditions, but also expand the candidate set through the knowledge graph and random walk to include possible related upstream and downstream products or substitute products, thereby providing a rich and diverse candidate set for subsequent personalized recommendation.

[0144] Specifically, in an embodiment of the present application, the deep reinforcement learning model is constructed as follows:

[0145] The state space includes a user feature vector, current session information, context information and a recommended product list, wherein the current session information includes search conditions and interaction history; these information comprehensively reflects the user's current demand and historical preferences.

[0146] The action space adopts a three-layer structure design, and the action definition of each layer is: the first layer selects a product category, the second layer selects a specific product type, and the third layer selects a specific specification or model.

[0147] The state encoder, whose input is each item of information in the state space, outputs a fixed-dimensional state representation vector; specifically, MLP or transformer structure is used.

[0148] The action encoder, whose input is each product information in the candidate product set, outputs a fixed-dimensional action representation vector; the specific structure is similar to that of the state encoder.

[0149] The policy network, whose input is the state representation vector and the action representation vector, outputs the probability distribution of selecting each action in the current state; specifically, a three-layer structure is adopted, and a softmax output layer is used in each layer to generate the action probability distribution of the corresponding layer.

[0150] The value network, whose input is the state representation vector, outputs the value estimate of the current state; specifically, MLP structure is used.

[0151] The multi-scale Q network includes three Q networks of short-term, medium-term and long-term, whose input is the state representation vector and the action representation vector, and whose output is the Q value of the corresponding time scale. Among them, each Q network can use a double Q learning structure, including an online network and a target network.

[0152] In this embodiment, the pre-training process of the deep reinforcement learning model is as follows:

[0153] Data collection: collect historical user interaction data, including search, click, inquiry and purchase behavior.

[0154] Behavioral simulation: Use user simulator to generate large amount of simulated interaction data to enrich training dataset.

[0155] Supervised pre-training: Pre-train state encoder and action encoder using historical data. Perform behavior cloning training on policy network to mimic expert behavior.

[0156] Offline reinforcement learning: Use offline policy evaluation and improvement methods such as conservative Q-learning (CQL) or batch-constrained Q-learning (BCQ) to perform initial training of Q network on historical data.

[0157] Online fine-tuning: Deploy model for online learning, use techniques such as prioritized experience replay and target network to stabilize learning process. Regularly evaluate model performance and conduct A / B testing.

[0158] Specifically, in an embodiment of the present application, step S4 includes:

[0159] S41 merge user state vector, current session information, context information and recommended product list, and use state encoder to encode the merged information into fixed-dimensional state representation vector s. The dimension of s can be 512.

[0160] S42 for each product in the candidate product set, extract its feature information, and use action encoder to encode each product into fixed-dimensional action vector representation a; Specifically, product feature information includes chemical properties, physical properties, application fields, etc. The dimension of a is 256.

[0161] S43 input the state representation vector s into the first layer policy network, output the probability distribution π1(a|s) on all product categories, and use the ε-greedy strategy to select a product category; merge the state representation vector s and the selected product category information, input into the second layer policy network, output the probability distribution π2(a|s) on all product types under the selected category, and use the ε-greedy strategy to select a specific product type; merge the state representation vector s, the selected product category and the specific product type information, input into the third layer policy network, output the probability distribution π3(a|s) on all specific specifications or models under the selected specific product type, and use the ε-greedy strategy to select a specific specification or model.

[0162] S44 for each selected specific product, use short-term, medium-term and long-term Q networks to calculate Q values Q s (s,a)、Q m (s,a)、Q l (s,a) respectively, and use attention mechanism to weight the Q values of the three time scales:

[0163] Q(s,a)=Attention(Qs (s,a),Q m (s,a),Q l (s,a))

[0164] where Q s (s,a) denotes a short-term Q-network, Q m (s,a) denotes a medium-term Q-network, Q l (s,a) denotes a long-term Q-network.

[0165] S45 calculates a comprehensive score S for each candidate product, ranks the candidate products in descending order according to the comprehensive score S, and selects the top-K products in the ranking as a preliminary recommendation result; for example, K = 20.

[0166] The calculation formula of the comprehensive score S is:

[0167] S(a) = λ x (pi1(a|s) x pi2(a|s) x pi3(a|s)) + (1 - lambda) x Q(s,a)

[0168] In the formula, lambda is a balance factor for adjusting the relative importance of the policy network and the Q value.

[0169] Specifically, in the calculation formula of the comprehensive score S, pi1(a|s) x pi2(a|s) x pi3(a|s) represents the output of the policy network. In the hierarchical action space, selecting a complete action (i.e., a specific product) can be regarded as a chain of conditional probabilities. According to the chain rule in probability theory, that is:

[0170] P(a|s) = P(a1, a2, a3|s) = P(a1|s) x P(a2|s, a1) x P(a3|s, a1, a2)

[0171] Where a1, a2, a3 represent the selection of the three levels respectively.

[0172] In order to simplify the calculation, it is assumed that the selection of each layer only depends on the state s, that is, approximately:

[0173] P(a|s) = P(a1|s) x P(a2|s) x P(a3|s) = pi1(a|s) x pi2(a|s) x pi3(a|s)

[0174] This probability is also considered as the joint probability of simultaneously selecting a specific product category, product type, and specific specification / model under the given state s.

[0175] S46 reorders the preliminary recommendation result using the maximum marginal relevance algorithm, calculates the similarity matrix between products, iteratively selects products that are both relevant and diverse, and obtains the current recommendation result.

[0176] Specifically, the reordering process is as follows:

[0177] The similarity matrix between the products is calculated using cosine similarity.

[0178] Iteratively select products that are both relevant and diverse:

[0179] MMR = argmax [μ x cos sim (D i , Q) - (1 - μ) x max cos sim (D i , D j )]

[0180] where cos sim denotes cosine similarity, D i is the candidate product, Q is the query, D j is the selected product, and μ controls the balance between relevance and diversity.

[0181] Repeat the iteration step until a sufficient number of products are selected. The current recommendation result is obtained.

[0182] Specifically, in an embodiment of the present application, the current recommendation result is presented to the user, and user feedback is obtained, including click behavior, inquiry behavior, order behavior, explicit rating, browsing time, collection behavior, comparison behavior, and sharing behavior.

[0183] where the click behavior includes recording whether the user clicks to view the detailed information of the recommended product; recording the timestamp and click order; distinguishing between the first click and repeated clicks. The inquiry behavior includes recording whether the user initiates an inquiry on the recommended product; recording the timestamp and inquiry method (such as online inquiry, telephone inquiry, etc.); recording the specific content of the inquiry (such as asking for the price, delivery period, minimum order quantity, etc.). The order behavior includes recording whether the user places an order to purchase the recommended product; recording the timestamp, order amount, and purchase quantity of the order; recording the payment status and shipping status of the order. The explicit rating includes recording the user's rating of the recommended product (1-5 points). The browsing time includes recording the user's stay time on the recommended product page; distinguishing between valid browsing time and total stay time. The collection behavior includes recording whether the user adds the recommended product to the collection; recording the timestamp of the collection. The comparison behavior includes recording whether the user adds the recommended product to the comparison list; recording which products are compared. The sharing behavior includes recording whether the user shares the recommended product; recording the sharing platform and timestamp.

[0184] Specifically, in an embodiment of the present application, in step S6, the parameters of the deep reinforcement learning model and the user state vector are updated according to the user feedback, including:

[0185] According to the reward function, the comprehensive reward value R of each type of user feedback is calculated. The reward function is as follows:

[0186] R = w1xr + w2xr + w3xr + w4xr + w5xr + w6xr + w7xr c i o f r d n

[0187] In the formula, r c is the click reward item, r i is the inquiry reward item, r o is the order reward item, r f is the display rating reward item, r r is the relevance reward item, r d is the diversity reward item, r n is the novelty reward item, and w1 to w7 are the corresponding weights.

[0188] Each reward item also corresponds to its reward value:

[0189] r c = {0.1 if the user clicks to view product details; 0.2 if the user clicks and stays for more than threshold T1; 0.3 if the user clicks and performs further interaction; 0 if there is no click behavior};

[0190] r i = {0.3 if the user initiates online inquiry; 0.4 if the user initiates telephone inquiry; 0.5 if the user communicates effectively with customer service after inquiry, and the communication time exceeds threshold T2; 0 if there is no inquiry behavior};

[0191] r o = {1.0 if the user places an order; 1.2 if the user places an order and the order amount exceeds threshold V1; 1.5 if the user places an order and it is the first purchase; 0 if there is no order behavior};

[0192] r f = (user rating - 3) / 2, the user rating is on a 5-point scale;

[0193] r r = cos sim (product e , user sv ), where cos sim is the cosine similarity, product e is the embedded vector representation of the product, and user sv is the user state vector. ​​​​​​

[0194] r d = 1 - max(cos sim (product e ,prev rp )), where prev rp is the set of previously recommended products;

[0195] r n = 1 - pop(product), where pop(product) is the normalized popularity of the product.

[0196] For the weights of each reward term in the reward function, an online learning method can be used to dynamically adjust the weights, for example: collecting user feedback and behavior data; calculating the correlation between each weight corresponding reward term and overall user satisfaction; adjusting the weight according to the correlation, increasing the weight of the reward term with high correlation and decreasing the weight of the reward term with low correlation; using gradient descent method to optimize the weight, maximizing the cumulative reward:

[0197] where, is the learning rate. E is the cumulative reward. Set the upper and lower limits of the weight to ensure that each reward term has a certain influence.

[0198] In this embodiment, each reward term is set to correspond to different user feedback and characteristics of the recommendation system itself to comprehensively evaluate the effect of the recommendation. Among them, the click reward corresponds to the click behavior and browsing behavior, the inquiry reward corresponds to the inquiry behavior, the order reward corresponds to the purchase behavior, the user feedback reward corresponds to the explicit evaluation, the relevance reward corresponds to the matching degree of the recommended products and the user interest, the diversity reward corresponds to the diversification degree of the recommended results, and the novelty reward corresponds to the freshness of the recommended products.

[0199] Use the TD learning method to update the Q value:

[0200] Q s (s,a) = Q s (s,a) + a s x (R + g s x max(Q s (s',a') - Q s (s,a)))

[0201] Q m (s,a) = Q m (s,a) + a m x (R + g m x max(Q m (s',a') - Q m (s,a)))

[0202] Ql (s,a) = Q l (s,a) + a l x(R + y l x max(Q l (s',a') - Q l (s,a))

[0203] Q(s,a) = Attention(Q s (s,a),Q m (s,a),Q l (s,a))

[0204] where s is the current state, a is the current action, s' is the next state, a' is the next possible action, a s , a m , a l are the learning rates of short-term, medium-term and long-term Q networks respectively, g s , g m , g l are the discount factors of short-term, medium-term and long-term Q networks respectively, and g s < g m < g l .

[0205] For the three-layer policy network (pi1, pi2, pi3), the policy gradient method is used to update the parameters of the policy network g. Take the first layer as an example:

[0206] where a is the learning rate, Q(s,a) is the multi-scale Q value, and V(s) is the state value estimated by the value network.

[0207] For pi2, pi3, the update method is similar to the first layer.

[0208] The TD learning method is used to update the value network parameters

[0209] where b is the learning rate of the value network, g is the discount factor, and l is the TD parameter. r t is the immediate reward.

[0210] The behavior information in the user feedback is integrated into the user state vector, and the self-encoder network is used to reduce and reconstruct the integrated user state vector to obtain the updated user state vector.

[0211] Specifically, the user feedback is quantified by the method of step S2 to form a user feedback vector, the current user state vector is spliced with the user feedback vector to obtain an integrated user state vector, the integrated user state vector is reduced in dimension and reconstructed by using a self-encoder network, i.e., a variational autoencoder, and then a re-encoded user state vector is obtained.

[0212] Specifically, the attention mechanism parameters also need to be updated: the attention weights are dynamically adjusted according to the prediction accuracy of the short-term, medium-term and long-term Q values.

[0213] The recommendation in the application is a real-time multi-round iterative process, and an option key is provided in the platform page, and the user selects to continue recommendation or stop recommendation according to the current recommendation result. If the user selects to continue recommendation, the model parameter updating and the user state vector updating are performed according to the above steps, and then a new round of recommendation is performed by returning to step S3. If the user selects to stop recommendation, it is judged whether the recommendation stopping condition is reached, the recommendation result of the current round is taken as the final recommendation result, and the total process of the current recommendation is ended.

[0214] The above only describes the preferred embodiments of the application and is not intended to limit the application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the application shall be included in the protection scope of the application.

Claims

1. A method for personalized recommendation of plastic products based on deep reinforcement learning, characterized in that, Comprise: S1 receives the search conditions input by the user on the plastic product platform; S2 constructs a user state vector according to the search conditions; S3 based on the user state vector and the search conditions, filters out the preliminary product set that meets the conditions from the plastic product database, and expands the preliminary product set according to the upstream and downstream relationships of the products to obtain the candidate product set; S4 uses a pre-trained deep reinforcement learning model to input the user state vector and the candidate product set, sorts the candidate products, and generates a recommendation result; S5 shows the recommendation result to the user and obtains user feedback; S6 judges whether the recommendation stopping condition is reached, if yes, stops the recommendation and outputs the final recommendation result, if not, updates the parameters of the deep reinforcement learning model and the user state vector according to the user feedback, and returns to step S3. 2.The deep reinforcement learning-based plastic product individualization recommendation method of claim 1, wherein, Step S2 includes: S21 parses the search conditions, extracts keywords and attribute requirements, and the attribute requirements include product category, performance index, processing technology and application field; S22 obtains user enterprise information, including enterprise size, enterprise type, position in the industrial chain and main product category; S23 extracts user historical behavior data, including recent search data, historical purchase data, historical conversation data and historical preference data; S24 obtains the current conversation context information, including time characteristics, geographical characteristics and market characteristics; S25 constructs a product correlation matrix to represent the upstream and downstream relationship strength between different plastic products; S26 converts the keywords, attribute requirements, user enterprise information, user historical behavior data, current conversation context information and product correlation matrix into numerical features to construct an initial feature vector; S27 uses an attention mechanism to weight each part of the initial feature vector to obtain a weighted feature vector; S28 inputs the weighted feature vector into a pre-trained autoencoder network to obtain a low-dimensional user state vector representation. 3.The deep reinforcement learning-based plastic product individualization recommendation method of claim 2, wherein, In step S28, the autoencoder network adopts a variational autoencoder structure, and the loss function during pre-training is as follows: L VAE = L recon + L sim + L chain L sim =∑max(0,d(z i ,z j )-α×d(z i ,z k )+m) L chain =Σ|f(z a ,z b )-r(a,b)| p f(z a ,z b )=σ(W·[z a ;z b ;z a ⊙z b ]) wherein L VAE is the total loss of the variational autoencoder; the first part L recon represents the reconstruction loss, x i is the original feature of the input data, x' i is the reconstructed feature, w i is the weight based on the importance of the plastic product attribute; the second part L sim represents the product similarity loss, z i ,z j ,z k is the encoded feature vector, (i,j) is a pair of similar products, k is a randomly selected dissimilar product, d(·) represents the distance measure in the feature space, m is the marginal parameter, and a is the adjustable similarity proportion coefficient, v d represents the importance weight of the dth feature dimension, which is predefined according to the characteristics of the plastic product, denotes the feature vector of the dth dimension after encoding; the third part L chain denotes the upstream and downstream industrial chain relationship loss, z a ,z b is the feature vector after encoding, f(·) denotes the industrial chain relationship prediction function, r(a, b) is the relationship strength of products a and b in the actual industrial chain, p is an adjustable exponential parameter, W is a learnable weight matrix, [; ] represents vector concatenation, ⊙ represents element-wise multiplication, and σ is the sigmoid activation function; where the reconstruction loss L recon measures whether the variational autoencoder accurately preserves the basic attribute information of the product and whether the low-dimensional representation after encoding contains the key information of the original data; product similarity loss L sim measures whether products with similar performance indicators or application fields are close in the encoding space, and whether products of different categories but can be replaced by each other maintain similarity in the feature space; upstream and downstream industrial chain relationship loss L chain measures whether the encoding space preserves the industrial chain relationship between products. 4.The deep reinforcement learning-based plastic product individualization recommendation method of claim 2, wherein, Step S3 includes: S31 constructs a database query statement according to the keywords and attribute requirements obtained by parsing the search conditions, which contains fuzzy matching and exact matching conditions, executes the query in the plastic product database, obtains the product list that meets the conditions, preliminarily sorts the query results, and selects the top-M products after sorting as the preliminary product set; S32 calls a pre-set plastic product knowledge graph, where nodes represent products and edges represent relationships between products, and the relationship types between products include upstream and downstream relationships, substitution relationships, common relationships and performance similarity relationships; S33 locates each product node in the preliminary product set in the plastic product knowledge graph, performs a weighted random walk from each located node, adds all product nodes visited by the walk to the preliminary product set, removes duplicate products, and obtains a preliminary candidate set; S34 calculates the similarity between the user state vector and each product in the preliminary candidate set, sorts the products according to the similarity, and selects the top-N products with the highest similarity as the candidate product set; S35 organizes the candidate product set into a three-layer structure, the first layer is the product category, the second layer is the specific product type, and the third layer is the specific specification or model, and adds metadata to each product, including its adjacent node information in the plastic product knowledge graph. 5.The deep reinforcement learning-based plastic product individualization recommendation method of claim 4, wherein, The deep reinforcement learning model comprises: a state space comprising a user feature vector, current session information, context information and a recommended product list, wherein the current session information comprises search conditions and interaction history; an action space designed in a three-layer structure, wherein the action of each layer is defined as: the first layer selects a product category, the second layer selects a specific product type, and the third layer selects a specific specification or model; a state encoder whose input is each item of information in the state space and whose output is a fixed-dimension state representation vector; an action encoder whose input is the information of each product in the candidate product set and whose output is a fixed-dimension action representation vector; a policy network whose input is the state representation vector and the action representation vector and whose output is a probability distribution of selecting each action in the current state; a value network whose input is the state representation vector and whose output is a value estimate of the current state; a multi-scale Q network comprising three Q networks of short-term, medium-term and long-term, whose input is the state representation vector and the action representation vector and whose output is a Q value of the corresponding time scale. 6.The deep reinforcement learning-based plastic product individualization recommendation method of claim 5, wherein, Step S4 comprises: S41 merging the user state vector, current session information, context information and recommended product list, and using the state encoder to encode the merged information into a fixed-dimension state representation vector s; S42 extracting the feature information of each product in the candidate product set and using the action encoder to encode each product into a fixed-dimension action vector representation a; S43 inputting the state representation vector s into the first layer policy network to output a probability distribution π1(a|s) over all product categories, using an ε-greedy strategy to select a product category; merging the state representation vector s and the selected product category information and inputting them into the second layer policy network to output a probability distribution π2(a|s) over all product types under the selected category, using an ε-greedy strategy to select a specific product type; merging the state representation vector s, the selected product category and the specific product type information and inputting them into the third layer policy network to output a probability distribution π3(a|s) over all specific specifications or models under the selected specific product type, using an ε-greedy strategy to select a specific specification or model; S44 For each selected specific product, calculate Q values Q s (s, a), Q m (s, a), Q l (s, a), and weight the Q values of the three time scales using an attention mechanism: Q(s, a) = Attention(Q s (s, a), Q m (s, a), Q l (s, a)) where Q s (s,a) denotes a short-term Q-network, Q m (s,a) denotes a medium-term Q-network, Q l (s,a) denotes a long-term Q-network; S45 calculating a comprehensive score S for each candidate product, sorting the candidate products in descending order according to the comprehensive score S, and selecting the top-K products in the sorted order as the preliminary recommendation result; S46 reordering the preliminary recommendation result using the maximum marginal relevance algorithm, calculating a similarity matrix between products, and iteratively selecting products that are both relevant and diverse to obtain the current recommendation result. 7.The deep reinforcement learning-based plastic product individualization recommendation method of claim 6, wherein, In step S45, the calculation formula of the comprehensive score S is: S(a) = λ × (π1(a|s) × π2(a|s) × π3(a|s)) + (1-λ) × Q(s,a) where λ is a balance factor used to adjust the relative importance of the policy network and the Q value. 8.The deep reinforcement learning-based plastic product individualization recommendation method of claim 6, wherein, Types of user feedback: click behavior, inquiry behavior, order behavior, explicit rating, browsing time, collection behavior, comparison behavior, and sharing behavior. 9.The deep reinforcement learning-based plastic product individualization recommendation method of claim 8, wherein, In step S6, the parameters of the deep reinforcement learning model and the user state vector are updated according to the user feedback, including: According to the reward function, the comprehensive reward value R of each type of user feedback is calculated; Update the Q value using the TD learning method: Q s (s,a) = Q s (s,a) + a s × (R + y s × max(Q s (s',a') - Q s (s,a))) Q m (s,a) = Q m (s,a) + a m × (R + y m × max(Q m (s',a') - Q m (s,a))) Q l (s,a) = Q l (s,a) + a l × (R + y l × max(Q l (s',a') - Q l (s,a))) Q(s, a) = Attention(Q s (s, a), Q m (s, a), Q l (s, a)) where s is the current state, a is the current action, s' is the next state, a' is the next possible action, a s , a m , a l are the learning rates of the short-term, medium-term, and long-term Q networks, respectively, a s , a m , a l are the discount factors of the short-term, medium-term, and long-term Q networks, respectively, and a s < a m < a l ; Update the parameters θ of the policy network using the policy gradient method and the value network parameters φ using the TD learning method; Integrate the behavior information in the user feedback into the user state vector, and use the autoencoder network to reduce and reconstruct the integrated user state vector to obtain the updated user state vector. 10.The deep reinforcement learning-based plastic product individualization recommendation method of claim 9, wherein, The reward function is as follows: R = w1xr c + w2xr i + w3xr o + w4xr f + w5xr r + w6xr d + w7xr n wherein r c is a click reward term, r i is a query reward term, r o is an order reward term, r f is an explicit rating reward term, r r is a relevance reward term, r d is a diversity reward term, r n is a novelty reward term, and w1to w7are corresponding weights. Each reward item also corresponds to its reward value: r c = {0.1, if the user clicks to view product details; 0.2, if the user clicks and stays for more than a threshold T1; 0.3, if the user clicks and further interacts; r i = {0.3, if the user initiates an online inquiry; 0.4, if the user initiates a telephone inquiry; 0.5, if the user has an effective communication with the customer service after the inquiry, and the communication duration exceeds a threshold T2; 0, no inquiry behavior}. r o = {1.0, if the user places an order to purchase; 1.2, if the user places an order to purchase and the order amount exceeds a threshold V1; 1.5, if the user places an order and it is the first purchase; 0, no order behavior; r f = (user rating - 3) / 2, user rating is on a 5-point scale; r r = cos sim (product e ,user sv ), where cos sim is the cosine similarity, product e is the embedding vector representation of the product, and user sv is the user state vector; r d = 1 - max(cos sim (product e , prev rp )), where prev rp is the set of previously recommended products; r n = 1 - pop(product), where pop(product) is the normalized popularity of the product.

Images