Material feeding method and device, terminal equipment and medium

Abstract

The application discloses a material putting method and device, a terminal device and a computer readable storage medium. The method comprises the following steps: obtaining a historical clicked material sequence of a current user; determining a material putting optimal action sequence based on the historical clicked material sequence; putting a plurality of to-be-put materials in the material putting optimal action sequence, and evaluating the material putting value of the plurality of to-be-put materials after the putting is completed. The application can realize accurate putting of multiple types of materials.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a material dispensing method, apparatus, terminal equipment, and computer-readable storage medium. Background Technology

[0002] In personalized recommendations within the financial sector, there are scenarios where multiple materials that a target user might be interested in need to be recommended simultaneously. For example, a single section in a bank app might recommend multiple wealth management products, financial news, and promotional activities. These materials often come from different categories and fields to cater to users' diverse interests, thereby increasing user engagement. Simultaneously, it's also necessary to enhance the diversity and richness of the materials delivered on the operational side.

[0003] The mainstream solution in this scenario is a combination of deep learning-based single-material click-through rate (CTR) prediction model ranking and manual adjustment of material category distribution ratios. Specifically, for a given user, their user profile data and information about each material in the material pool are stored in storage. When the user navigates to the target location, the client sends a request to the server to retrieve the material list. Upon receiving the request, the server retrieves the corresponding data from the storage medium and imports a pre-prepared deep learning model file into its memory. The model then uses the user profile and all information about a single material as input to calculate the probability of the user clicking on that material. The predicted CTR values ​​of all candidate materials are then sorted in reverse chronological order, and the top N materials are returned to the client for display. Furthermore, to avoid users receiving only one type of material, the ranking results are manually adjusted based on actual business categories. For example, to ensure exposure for a particular material category, its weight is adjusted, i.e., the predicted CTR value for that material is artificially increased.

[0004] It is evident that the above-mentioned single-material click-through rate prediction model, which combines ranking with category-based placement adjustment, has three main drawbacks: First, relying solely on manual adjustment of material placement weights is costly, requiring experts to determine the weights for each category, and it cannot quickly adapt to the changing customer interests and business objectives in real-world scenarios, resulting in a slow response time. Second, the effect is poor; manual weight adjustments can damage the model's click-through rate prediction accuracy because, in reality, the two may produce contradictory placement results. Furthermore, introducing uniform category weights lacks personalization. Third, it does not consider the relationship between the final placement of multiple materials. Users actually browse materials in a certain order, and the current single-material click-through rate prediction technology does not take into account the correlation between materials.

[0005] In general, the existing material delivery methods cannot achieve accurate delivery of multiple types of materials to each user. Summary of the Invention

[0006] The main objective of this invention is to provide a material dispensing method, apparatus, terminal equipment, and computer-readable storage medium, which aims to achieve accurate dispensing of various types of materials.

[0007] To achieve the above objectives, the present invention provides a material dispensing method, the method comprising the following steps:

[0008] Get the current user's historical clicked material sequence;

[0009] Based on the historical clicked material sequence, determine the optimal action sequence for material delivery;

[0010] The material to be delivered is delivered in the optimal action sequence of material delivery, and the material delivery value of the material to be delivered is evaluated after delivery is completed.

[0011] Optionally, the step of determining the optimal action sequence for material delivery based on the historical clicked material sequence includes:

[0012] Obtain the user representation vector corresponding to the historical clicked material sequence;

[0013] Based on the material vectors in the preset material space, and combined with the user representation vectors corresponding to the historical clicked material sequences, an action sequence containing multiple materials is obtained;

[0014] The first preset number of materials in the action sequence are taken as the materials to be delivered, and the optimal action sequence for material delivery is determined based on the materials to be delivered.

[0015] Optionally, before the step of obtaining the user representation vector corresponding to the historical clicked material sequence, the method further includes:

[0016] Construct user profile vectors for material delivery targets;

[0017] The step of obtaining the user representation vector corresponding to the historical clicked material sequence includes:

[0018] The user profile vector is combined with the historical click material sequence to obtain the user representation vector.

[0019] Optionally, before the step of obtaining an action sequence containing multiple materials based on the material vectors in a preset material space and combined with the user representation vectors corresponding to the historical clicked material sequences, the method further includes:

[0020] Construct the material space;

[0021] Obtain the material vector of each material in the material space.

[0022] Optionally, the step of obtaining an action sequence containing multiple materials based on material vectors in a preset material space and combined with user representation vectors corresponding to the historical clicked material sequences includes:

[0023] Repeat the following steps according to a preset number of times: multiply the material vector of each material in the material space with the user representation vector corresponding to the historical clicked material sequence to obtain multiple multiplication results; obtain the material in the material space corresponding to the maximum value among the multiple multiplication results, place the material at the end of the preset initial action sequence, and delete the material from the material space to obtain the action sequence containing multiple materials.

[0024] Optionally, the step of evaluating the material deployment value of the plurality of materials to be deployed includes:

[0025] Based on the multidimensional vectors corresponding to the multiple materials to be delivered, and combined with the user profile vector and the historical click material sequence, the material delivery value of this delivery is obtained.

[0026] Optionally, before the step of obtaining the current user's historical clicked material sequence, the method further includes:

[0027] Build a material delivery model;

[0028] The material delivery model is trained using a preset strategy gradient algorithm, and the step of executing the current user's historical clicked material sequence is executed after the training is completed.

[0029] To achieve the above objectives, the present invention also provides a material dispensing device, the material dispensing device comprising:

[0030] The acquisition module is used to acquire the historical clicked material sequence of the current user;

[0031] The determination module is used to determine the optimal action sequence for material delivery based on the historical clicked material sequence;

[0032] The delivery module is used to deliver multiple materials to be delivered in the optimal sequence of material delivery actions, and after the delivery is completed, to evaluate the delivery value of the multiple materials to be delivered.

[0033] To achieve the above objectives, the present invention also provides a terminal device, the terminal device including a memory, a processor, and a material dispensing program stored in the memory and executable on the processor, wherein the material dispensing program, when executed by the processor, implements the steps of the material dispensing method as described above.

[0034] Furthermore, to achieve the above objectives, the present invention also proposes a computer-readable storage medium storing a material dispensing program, which, when executed by a processor, implements the steps of the material dispensing method as described above.

[0035] To achieve the above objectives, the present invention also provides a computer program product, the computer program product comprising a computer program, which, when executed by a processor, implements the steps of the material dispensing method as described above.

[0036] This invention provides a material delivery method, apparatus, terminal device, computer-readable storage medium, and computer program product. The method involves: acquiring the historical clicked material sequence of the current user; determining the optimal action sequence for material delivery based on the historical clicked material sequence; delivering multiple materials to be delivered from the optimal action sequence; and evaluating the material delivery value of the multiple materials to be delivered after delivery is completed.

[0037] Compared to existing technologies that combine models with manual methods for single-material delivery, this invention can directly determine multiple materials to be delivered based on the user's historical click sequence, and then deliver these materials, evaluating their delivery value afterward. Therefore, this invention improves delivery accuracy by basing material delivery on the user's historical click history. Furthermore, it can deliver multiple materials to the user, increasing the richness of the delivery and making it more aligned with the user's actual browsing and usage needs, effectively enhancing the user experience. Additionally, this invention eliminates the need for manual intervention in the weighting of each material, thus improving both delivery efficiency and accuracy. Attached Figure Description

[0038] Figure 1 This is a schematic diagram of the hardware operating environment involved in the embodiments of the present invention;

[0039] Figure 2 This is a schematic flowchart of an embodiment of the material delivery method of the present invention;

[0040] Figure 3 This is a schematic diagram of the functional modules of an embodiment of the material dispensing device of the present invention.

[0041] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0042] It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0043] like Figure 1 As shown, Figure 1 This is a schematic diagram of the device structure of the hardware operating environment involved in the embodiments of the present invention.

[0044] The terminal device in this embodiment of the invention can be a mobile phone, tablet computer, computer, server or other network device, etc. The terminal device in this embodiment can be used to achieve accurate delivery of multiple types of materials.

[0045] like Figure 1 As shown, the terminal device may include: a processor 1001, such as a CPU; a network interface 1004; a user interface 1003; a memory 1005; and a communication bus 1002. The communication bus 1002 is used to enable communication between these components. The user interface 1003 may include a display screen and an input unit such as a keyboard. Optionally, the user interface 1003 may also include a standard wired interface or a wireless interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface). The memory 1005 may be high-speed RAM or non-volatile memory, such as a disk drive. Optionally, the memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0046] Those skilled in the art will understand that Figure 1 The equipment structure shown does not constitute a limitation on the material dispensing equipment and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0047] like Figure 1 As shown, the memory 1005, serving as a computer storage medium, may include an operation module, a network communication module, a user interface module, and a material dispensing program. The operation module is a program that manages and controls the hardware and software resources of the equipment, supporting the operation of the material dispensing program and other software or programs. Figure 1 In the device shown, the user interface 1003 is mainly used for data communication with the client; the network interface 1004 is mainly used for establishing a communication connection with the server; and the processor 1001 can be used to call the material dispensing program stored in the memory 1005 and perform the following operations:

[0048] Get the current user's historical clicked material sequence;

[0049] Based on the historical clicked material sequence, determine the optimal action sequence for material delivery;

[0050] The material to be delivered is delivered in the optimal action sequence of material delivery, and the material delivery value of the material to be delivered is evaluated after delivery is completed.

[0051] Furthermore, the processor 1001 can be used to call the material dispensing program stored in the memory 1005 and perform the following operations:

[0052] Obtain the user representation vector corresponding to the historical clicked material sequence;

[0053] Based on the material vectors in the preset material space, and combined with the user representation vectors corresponding to the historical clicked material sequences, an action sequence containing multiple materials is obtained;

[0054] The first preset number of materials in the action sequence are taken as the materials to be delivered, and the optimal action sequence for material delivery is determined based on the materials to be delivered.

[0055] Furthermore, before the step of obtaining the user representation vector corresponding to the historical clicked material sequence, the processor 1001 can call the material delivery program stored in the memory 1005 and perform the following operations:

[0056] Construct user profile vectors for material delivery targets;

[0057] The processor 1001 can be used to call the material dispensing program stored in the memory 1005 and perform the following operations:

[0058] The user profile vector is combined with the historical click material sequence to obtain the user representation vector.

[0059] Furthermore, before the step of obtaining an action sequence containing multiple materials based on the material vector in the preset material space and combined with the user representation vector corresponding to the historical clicked material sequence, the processor 1001 can call the material delivery program stored in the memory 1005 and perform the following operations:

[0060] Construct the material space;

[0061] Obtain the material vector of each material in the material space.

[0062] Furthermore, the processor 1001 can be used to call the material dispensing program stored in the memory 1005 and perform the following operations:

[0063] Repeat the following steps according to a preset number of times: multiply the material vector of each material in the material space with the user representation vector corresponding to the historical clicked material sequence to obtain multiple multiplication results; obtain the material in the material space corresponding to the maximum value among the multiple multiplication results, place the material at the end of the preset initial action sequence, and delete the material from the material space to obtain the action sequence containing multiple materials.

[0064] Furthermore, the processor 1001 can be used to call the material dispensing program stored in the memory 1005 and perform the following operations:

[0065] Based on the multidimensional vectors corresponding to the multiple materials to be delivered, and combined with the user profile vector and the historical click material sequence, the material delivery value of this delivery is obtained.

[0066] Furthermore, prior to the step of obtaining the current user's historical clicked material sequence, the processor 1001 can be used to call the material delivery program stored in the memory 1005 and perform the following operations:

[0067] The material delivery model is trained using a preset strategy gradient algorithm, and after training is completed, the step of obtaining the current user's historical clicked material sequence is executed.

[0068] In the realm of personalized recommendations within the financial sector, it is often necessary to simultaneously recommend multiple materials that a target user might be interested in. For instance, in a section of a bank's app, multiple wealth management products, financial information, and promotional activities might be recommended and displayed to the user. These materials often come from different categories and fields to satisfy users' diverse interests. At the same time, when recommending materials to users from the operations side, it is often necessary to simultaneously deploy multiple types of materials to enhance the diversity and richness of the materials, thereby increasing user stickiness.

[0069] However, considering that the current method of manually adjusting the weights of materials in material delivery cannot quickly adapt to the changing customer interests and business goals in real-world scenarios, and that manual weighting will impair the model's click-through rate prediction and result in a lack of personalization in material category weights, and that the relationship between multiple materials is not considered, i.e., users actually browse materials in a certain order, but the current single-material click-through rate prediction method does not take into account the correlation between materials, resulting in low accuracy of material delivery results, failing to meet the actual needs of users, and thus preventing the delivery of materials from being converted into actual benefits.

[0070] Therefore, in order to solve the above problems, this invention proposes a material delivery method, which aims to avoid the conflict caused by the combination of existing models and manual weighting methods in actual operation. This makes the entire material delivery method more consistent in actual scenarios, fully leverages the role of the model, and realizes the combination recommendation of delivered materials, which is more in line with the actual situation of the user's overall browsing and can effectively improve the user experience.

[0071] Reference Figure 2 , Figure 2 This is a flowchart illustrating the first embodiment of the material delivery method of the present invention.

[0072] This invention provides an embodiment of a material dispensing method. It should be noted that although the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than that shown here.

[0073] To ensure that the material delivery model aligns with actual user browsing behavior, and that materials are delivered, exposed, and clicked on the user's side according to user preferences, this embodiment uses reinforcement learning as a framework, considering the entire material recommendation process as a Markov Decision Process (MDP). The process comprises at least five components: state, action, state transition probability, reward, and discount coefficient, as detailed below:

[0074] State: In the current scenario, the state is a sequence of N historical clicked materials for a single user at time t. This sequence is ordered chronologically, with earlier clicks appearing first. The sequence is represented as follows:

[0075]

[0076] Action: In the current scenario, the action can be: recommending K items to the user at once at time t. This action is represented as follows:

[0077]

[0078] Reward: In the current scenario, rewards can include two types: user feedback (r) tu and material abundance r ti Among them, user feedback r tu This refers to whether a user clicked on any of the K items. Item richness refers to the number of product categories among the K items. In practice, users should receive more reward for interacting with items ranked higher than those ranked lower, as this indicates that the recommendation scheme meets the user's needs and improves the user experience. Therefore, the reward for user feedback is:

[0079]

[0080] Where Γ k-1 Let α be the location discount coefficient, and U be the user's feedback on the delivered material (i.e., whether they clicked it). Based on this, the above action α is performed at time t. t The reward received is:

[0081]

[0082] Transition Probability: The state transition probability in the current state can be expressed as: the probability of choosing action α at time t. tAfter that, the current state s t From the state s of the next moment t+1 The transition probability, denoted as p(s) t+1 |s t ,α t ), where if the user's current action α t No feedback, meaning no click was made on any of the K materials deployed, then s t+1 =s t Otherwise, the state s in the next moment t+1 It will be updated.

[0083] Discount factor: This factor γ∈[0,1] represents the discount on future rewards. The reward received at each subsequent time step should be discounted an additional time, i.e., r t ,γr t+1 , For example, when γ = 0, the material delivery model selects action α at time t. t Only the reward r of the current action will be considered. t When γ = 1, the material delivery model will fully consider the rewards brought by subsequent actions.

[0084] Based on the above description of the components of the MDP process, the personalized recommendation problem of delivering K materials in a single run is transformed into finding a strategy to maximize cumulative rewards. The problem.

[0085] Therefore, in this embodiment, the entire material delivery module is divided into two parts: action selection and action value judgment. The material delivery algorithm adopts the Actor-Critic method, in which the Actor is used to solve the problem of action selection, and the Critic is used to judge the action value. Together, they can achieve accurate delivery of multiple types of materials, meet actual needs, and improve material conversion rate, i.e., user click-to-purchase rate, etc.

[0086] Based on the above description, the material delivery method in this embodiment specifically includes the following steps:

[0087] Step S10: At a preset time, obtain the current user's historical clicked material sequence;

[0088] Based on the above explanation, the terminal device uses the Actor model to obtain the current user's historical clicked material sequence s at a preset time t. t .

[0089] In this embodiment, the aforementioned historical click material sequence s tIt is a rule-based sorting sequence of materials, meaning that materials clicked earlier by the user are arranged first, and materials clicked later are arranged last. This arrangement reflects user characteristics and provides a basis for accurate delivery of subsequent materials.

[0090] Step S20: Based on the historical clicked material sequence, determine the optimal action sequence for material delivery;

[0091] The terminal device obtains the current user's historical clicked material sequence s t Then, the Actor model will be used to further analyze the historical clicked material sequence s. t Determine the optimal action sequence α for material delivery. t The optimal action sequence α for material delivery t It includes multiple materials to be delivered. Therefore, this embodiment can deliver multiple materials to the user at the same time t, and the delivered materials are determined by the user based on their historical clicks. Thus, while increasing the richness of the delivered materials, it also meets the user's needs and preferences as much as possible.

[0092] Step S30: Deploy multiple materials to be deployed in the optimal action sequence for material deployment, and evaluate the material deployment value of the multiple materials to be deployed after deployment.

[0093] The terminal equipment determines the optimal sequence of actions α for material delivery. t Then, using the Critic model described above, the optimal action sequence α for material delivery is determined. t Multiple materials to be deployed are deployed, and the deployment value is evaluated after deployment to determine the benefits generated. Therefore, in this embodiment, in addition to recommending multiple materials to the user simultaneously at time t, the deployment results can also be evaluated to optimize the deployment plan.

[0094] In this embodiment, the terminal device uses the Actor model to obtain the current user's historical clicked material sequence s at a preset time t. t The terminal device obtains the current user's historical clicked material sequence s t Then, the Actor model will be used to further analyze the historical clicked material sequence s. t The terminal device identifies multiple materials to be deployed. After identifying these materials, it uses the Critic model described above to evaluate the deployment value of each material and determine the revenue generated from this deployment.

[0095] As can be seen, compared to the existing single-material delivery method that combines model and manual intervention, this invention can directly determine multiple materials to be delivered based on the user's historical click sequence, and then deliver these multiple materials, evaluating the value of each delivery afterward. Therefore, this invention improves delivery accuracy by delivering materials based on the user's historical click history. Furthermore, it can deliver multiple materials to the user, increasing the richness of the delivery and making it more aligned with the user's actual browsing and usage needs, effectively enhancing the user experience. In addition, this invention eliminates the need for manual intervention in the weighting of each material, thus improving both delivery efficiency and accuracy.

[0096] Furthermore, based on the first embodiment of the material delivery of the present invention, a second embodiment of the material delivery of the present invention is proposed.

[0097] In this embodiment, step S20 may include:

[0098] Step S201: Obtain the user representation vector corresponding to the historical clicked material sequence;

[0099] Step S202: Based on the material vectors in the preset material space and combined with the user representation vectors corresponding to the historical clicked material sequences, obtain an action sequence containing multiple materials;

[0100] Step S203: Select the preset number of materials at the beginning of the action sequence as the materials to be delivered, and determine the optimal action sequence for material delivery based on the materials to be delivered.

[0101] The terminal device obtains the current user's historical clicked material sequence s t Then, the historical clicked material sequence s will be obtained. t The corresponding user representation vector. Furthermore, this is combined with the material vector e in the preset material space. i This yields an action sequence containing multiple materials. At this point, the first K materials in the action sequence are selected as the materials to be delivered. In this embodiment, the value of K is not specifically limited; the user can pre-set the value of K. The resulting K materials to be delivered will then be displayed on the user's interface.

[0102] As can be seen, in this embodiment, the Actor model enables precise and rich delivery of materials to each user without the need for manual adjustment of the weights between materials, thus avoiding damage to the model output and interference with the accuracy of the model estimation results. This makes the determined materials to be delivered more in line with user needs, thereby improving the material conversion rate.

[0103] Furthermore, before step S201, "obtaining the user representation vector corresponding to the historical clicked material sequence", the following may also be included:

[0104] Step S204: Construct user profile vectors for material delivery targets;

[0105] Therefore, step S201 above includes:

[0106] Step S2011: Combine the user profile vector with the historical click material sequence to obtain the user representation vector.

[0107] Terminal devices need to pre-build user profile vector p of the current material delivery target. u .

[0108] Based on this, the terminal device uses a deep learning model to generate the user profile vector p u With the above historical click material sequence s t By combining them, we obtain a user representation vector of dimension k. The user profile vector mentioned above is a merging of the user's basic features, including age, gender, and region. The deep learning model described above can be specifically a neural network composed of a feature domain embedding layer and a multi-head self-attention layer, represented as: Where, θ π These are the parameters of the model, such as the parameters of the feature domain embedding layer and the parameters of the multi-head self-attention layer.

[0109] Specifically, for example, the user profile vector p mentioned above u With historical click material sequence s t The specific method of combination is as follows: the user profile vector p u and historical clicked material sequences t The concatenated layers are then fed into a neural network composed of multiple MLP layers to obtain the output.

[0110] Furthermore, before step S202 above, "obtaining a material sequence containing multiple materials based on the material vector in the preset material space and in combination with the historical clicked material sequence", the following may also be included:

[0111] Step S205: Construct the material space;

[0112] Step S206: Obtain the material vector of each material in the material space.

[0113] The terminal device needs to pre-construct a material space I, and then obtain the material vector e of each material in the material space I. i .

[0114] In one embodiment, the material vector e of each of the above materials i It is generated using the deep learning model AutoInt based on the material's basic information, multimodal features (such as text, images, videos, and audio), and popularity information (user click-through rate).

[0115] Based on this, the terminal equipment can then determine the material vector e of the material. i By combining this with the user representation vectors corresponding to the historical clicked material sequences, a material sequence containing multiple materials is obtained. This method yields an action sequence containing multiple materials.

[0116] Furthermore, in step S202 above, "based on the material vectors in the preset material space and combined with the historical clicked material sequence, obtain an action sequence containing multiple materials" may include:

[0117] Step S2021, repeat the following steps according to a preset number of times: multiply the material vector of each material in the material space with the user representation vector corresponding to the historical clicked material sequence to obtain multiple multiplication results; obtain the material in the material space corresponding to the maximum value among the multiple multiplication results, place the material at the end of the preset initial action sequence, and delete the material from the material space to obtain the action sequence containing multiple materials.

[0118] It should be noted that, in this embodiment, it can be understood that the aforementioned preset number of times is the same as the value of the quantity K of the materials to be delivered. Thus, after K cycles, the first K materials in the obtained action sequence are taken as the materials to be delivered.

[0119] Specifically, for example:

[0120] 1) The terminal equipment will assign the material vector e to each material in the material space I. i User representation vectors of dimension k corresponding to the historical clicked material sequences. Multiply the results to calculate the cosine similarity; this will result in multiple multiplication results.

[0121] 2) Obtain the maximum value from the above multiplication results, and obtain the material in material space I corresponding to the maximum value;

[0122] 3) Place the above material at the end of the initial action sequence, wherein, in this embodiment, the initial action sequence is an action sequence that initializes an empty sequence.

[0123] 4) Remove the material in material space I corresponding to the above maximum value from material space I;

[0124] 5) Repeat steps 1) to 4) above K times, and obtain the action sequence after repeating K times. In this action sequence, there are multiple materials, and the order of each material in the action sequence reflects the user's click rate on the material and the correlation between the materials.

[0125] Based on this, the terminal device can use the first K materials in the above action sequence as materials to be delivered.

[0126] Furthermore, in step S30 above, "evaluating the material deployment value of the plurality of materials to be deployed" may include:

[0127] Step S301: Based on the multi-dimensional vectors corresponding to the multiple materials to be delivered, and combined with the user profile vector and the historical click material sequence, obtain the material delivery value for this material delivery.

[0128] After obtaining the optimal action sequence for material delivery and delivering multiple materials from the optimal action sequence, the terminal device will use the Critic model to evaluate the benefits brought by this delivery, based on the multi-dimensional vectors corresponding to the multiple materials to be delivered (in this embodiment, the multi-dimensional vectors are k-dimensional vectors), and combined with the user profile vector p. u and historical clicked material sequences t To obtain the material placement value Q of this material placement.

[0129] It should be noted that, in this embodiment, the goal of the Critic model is to accurately learn the action value function Q(s). t ,α t The function Q(s) t ,α t This is used to measure the benefits brought about by the output action of the above Actor model (that is, to deliver multiple materials to be delivered in the optimal action sequence of material delivery).

[0130] Specifically, for example, according to the Bellman equation:

[0131]

[0132] By using a multi-layered deep learning MLP model, the k-dimensional vector corresponding to the material to be delivered and the user profile vector p in the optimal action sequence for material delivery output by the Actor model are obtained. u and historical clicked material sequences t After merging, the result is transformed into a one-dimensional numerical output, yielding the material placement value for this material placement. The aforementioned multi-layer MLP model can be represented as:

[0133]

[0134] Where, θ μ These are the model parameters for a multi-layer MLP model.

[0135] Furthermore, before step S10, "obtaining the current user's historical clicked material sequence", the following may also be included:

[0136] Step S40: Construct a material delivery model;

[0137] Step S50: Train the material delivery model using a preset strategy gradient algorithm, and execute the step of the current user's historical clicked material sequence after training is completed.

[0138] Based on the above description, in this embodiment, the material delivery model can specifically be an Actor-Critic model, wherein the Actor model is used to output the optimal action sequence for material delivery, and the Critic model is used to evaluate the value of the optimal action sequence for material delivery.

[0139] Based on this, the terminal device uses the Policy Gradient Programming (DDPG) algorithm to train the Actor-Critic model, where the loss function of the Critic model is:

[0140]

[0141] in, θ μ Here, Q' represents the model parameters, and Q' represents the model parameters of the shadow network of the Critic model in the Policy Gradient Algorithm (DDPG).

[0142] In addition, the loss function of the Actor model is:

[0143]

[0144] Based on the loss functions corresponding to the Critic and Actor models mentioned above, the Actor-Critic model is trained, and material placement and value assessment are performed after training.

[0145] In this embodiment, the terminal device needs to pre-construct the user profile vector p of the current material delivery target. u The terminal device uses the deep learning model AutoInt to generate the user profile vector p. u With the above historical click material sequence s t By combining them, we obtain a user representation vector of dimension k. Then, repeat the process a preset number of times: change the material vector e of each material in material space I. i User representation vectors of dimension k corresponding to the historical clicked material sequences. Multiplication is performed, resulting in multiple multiplication results. The maximum value is selected from these results, and the material corresponding to this maximum value in material space I is retrieved. This material is placed at the end of the initial action sequence. The material corresponding to the maximum value in material space I is then removed from material space I. Finally, the first K materials in the action sequence are selected as the materials to be delivered. The terminal device will utilize the Critic model, based on the multidimensional vectors corresponding to the multiple materials to be delivered (in this embodiment, the multidimensional vectors are k-dimensional vectors), and combined with the user profile vector p. u and historical clicked material sequences t The material delivery value Q is obtained for this material delivery. The terminal device trains the Actor-Critic model using the loss functions corresponding to the Critic and Actor models, and performs material delivery and value evaluation after training.

[0146] As can be seen, in this invention, the optimal action sequence for material delivery at time t is determined through the Actor-Critic model, which includes multiple sequences to be delivered. This invention simultaneously improves user click-through rate and material richness, avoiding conflicts that can arise during actual operation due to manual weight adjustments. This makes the entire technology more consistent in real-world scenarios and fully leverages the model's capabilities. Furthermore, after material delivery is completed, this invention can evaluate the delivery's effectiveness, using the evaluation results to refine the Actor-Critic model, thereby improving delivery accuracy and ultimately enhancing the user experience.

[0147] Furthermore, based on the first and second embodiments of the material dispensing method of the present invention, a third embodiment of the material dispensing method of the present invention is proposed.

[0148] In this embodiment, to implement the various embodiments of the above material delivery method, it is divided into two parts: online production environment model service and offline model training and update. Specifically:

[0149] 1) Online model service:

[0150] Offline feature storage and loading: These features utilize the cluster's computing resources to perform logical calculations daily, including user features and material features. These features are then overwritten and stored in key-value storage resources for quick server retrieval. Material features use the material ID as the key, and user features use the user ID as the key. When in use, the corresponding feature is loaded based on the corresponding ID, with a time interval of O(1), to meet the timeliness requirements of online services.

[0151] Real-time feature computation, storage, and loading: These features need to be updated based on the latest events within a short time window. Therefore, a set of event data for the current time window is obtained through streaming data (such as Flink), then Flink SQL performs the latest feature computation, and finally the latest result is overwritten and stored in key-value storage resources. The loading method is the same as that for offline features.

[0152] Online feature engineering: using the AvaitorScript framework to evaluate features obtained from key-value stores;

[0153] Model inference service: The Actor-Critic model is deployed using Tensorflow Serving and invoked via a REST API. The RESTful API interacts using the standard HTTP POST method, with both requests and responses being JSON objects. Therefore, the request includes information such as the user ID and candidate material ID, allowing the Actor-Critic model to obtain the required canonical input (e.g., historical clicked material sequences) by completing the aforementioned steps. t User profile vector p u The system then processes the material space (I) and finally returns the inference result (i.e., the optimal sequence of actions for material placement) to the request.

[0154] 2) Offline model training

[0155] Training data collection: Online logs from the server are cleaned and collected into the large-scale computing cluster Spark. Training data calculation and generation are completed using methods such as Spark SQL.

[0156] Model training and update: Use the Tensorflow framework to train the model on the training data exported from the cluster and save the trained model file.

[0157] 3) Offline models are updated to online model services.

[0158] Alternatively, the model update action can be manually updated to each server using the latest trained model file, or the model update action can be encapsulated as an API so that the model file can be updated to each server via an HTTP POST request after the offline model file is generated.

[0159] In this way, the above operations enabled the construction of an online production environment model and the training and updating of an offline model, and further achieved accurate, efficient and rich material delivery.

[0160] Furthermore, embodiments of the present invention also propose a material dispensing device, referring to... Figure 3 The material dispensing device includes:

[0161] Module 10 is used to obtain the historical clicked material sequence of the current user;

[0162] The determining module 20 is used to determine the optimal action sequence for material delivery based on the historical clicked material sequence;

[0163] The delivery module 30 is used to deliver multiple materials to be delivered in the optimal action sequence of material delivery, and after the delivery is completed, to evaluate the material delivery value of the multiple materials to be delivered.

[0164] Furthermore, the determining module 20 includes:

[0165] The user representation vector acquisition unit is used to acquire the user representation vector corresponding to the historical clicked material sequence.

[0166] An action sequence acquisition unit is used to acquire an action sequence containing multiple materials based on material vectors in a preset material space and combined with user representation vectors corresponding to the historical clicked material sequences.

[0167] The optimal action sequence determination unit for material delivery is used to select a preset number of materials at the beginning of the action sequence as materials to be delivered, and to determine the optimal action sequence for material delivery based on the materials to be delivered.

[0168] Furthermore, the determining module 20 includes:

[0169] The first building unit is used to construct the user profile vector of the material delivery target;

[0170] The user representation vector acquisition unit includes:

[0171] The user representation vector acquisition subunit is used to combine the user profile vector with the historical click material sequence to obtain the user representation vector.

[0172] Furthermore, the determining module 20 includes:

[0173] The second building unit is used to build the material space;

[0174] The material vector acquisition unit is used to acquire the material vector of each material in the material space.

[0175] Furthermore, the action sequence acquisition unit includes:

[0176] An execution subunit is used to repeatedly execute the following steps according to a preset number of times: multiplying the material vector of each material in the material space with the user representation vector corresponding to the historical clicked material sequence to obtain multiple multiplication results; obtaining the material in the material space corresponding to the maximum value among the multiple multiplication results, placing the material at the end of the preset initial action sequence, and deleting the material from the material space to obtain the action sequence containing multiple materials.

[0177] Furthermore, the delivery module includes:

[0178] The material delivery value acquisition unit is used to obtain the material delivery value of this material delivery based on the multi-dimensional vectors corresponding to multiple materials to be delivered, combined with the user profile vector and the historical click material sequence.

[0179] Furthermore, the material dispensing device further includes:

[0180] The building module is used to build the material delivery model;

[0181] The training module is used to train the material delivery model using a preset strategy gradient algorithm, and to execute the steps of the current user's historical clicked material sequence after the training is completed.

[0182] The extended content of the specific implementation of the material dispensing device of the present invention is basically the same as the various embodiments of the material dispensing method described above, and will not be repeated here.

[0183] Furthermore, embodiments of the present invention also propose a computer-readable storage medium storing a material dispensing program, wherein the material dispensing program, when executed by a processor, implements the steps of the material dispensing method described below.

[0184] The various embodiments of the material dispensing device and computer-readable storage medium of the present invention can be referred to the various embodiments of the material dispensing method of the present invention, and will not be repeated here.

[0185] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.

[0186] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0187] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a wearable device, locator, smartphone, tablet computer, etc.) to execute the methods described in the various embodiments of the present invention.

[0188] The above are merely preferred embodiments of the present invention and do not limit the scope of the patent. Any equivalent structural or procedural transformations made based on the description and drawings of the present invention, or direct or indirect applications in other related technical fields, are similarly included within the scope of patent protection of the present invention.

Claims

1. A method for dispensing materials, characterized in that, The material delivery method includes: Get the current user's historical clicked material sequence; Based on the historical click material sequence, the optimal action sequence for material delivery is determined, wherein: the user representation vector corresponding to the historical click material sequence is obtained; based on the material vectors in the preset material space, and combined with the user representation vectors corresponding to the historical click material sequence, an action sequence containing multiple materials is obtained; a preset number of materials at the beginning of the action sequence are selected as materials to be delivered, and the optimal action sequence for material delivery is determined based on the materials to be delivered, wherein: the user representation vector is generated by concatenating the user profile vector with the historical click material sequence and inputting it into a multi-layer MLP layer through a neural network composed of a feature domain embedding layer and a multi-head self-attention layer; the neural network is used to deeply mine the correlation between user interests and materials; based on the material vectors in the preset material space, and combined with the user representation vectors corresponding to the historical click material sequence, an action sequence containing multiple materials is obtained, wherein the material vectors in the material space are processed by the deep learning model AutoInt. The generation process integrates basic information, multimodal features, and popularity information of the materials. The iterative filtering method for the action sequence is as follows: Repeatedly execute the following steps a preset number of times: Multiply the material vector of each material in the material space with the user representation vector corresponding to the historical clicked material sequence to obtain multiple multiplication results; Obtain the material in the material space corresponding to the maximum value among the multiple multiplication results, place the material at the end of the preset initial action sequence, and delete the material from the material space to obtain an action sequence containing multiple materials; Select the preset number of materials at the beginning of the action sequence as materials to be deployed, and determine the optimal action sequence for material deployment based on the materials to be deployed. The multiple materials to be deployed in the optimal action sequence of material deployment are deployed, and after deployment is completed, the material deployment value of the multiple materials to be deployed is evaluated. In this process, according to the Bellman equation, the k-dimensional vectors, user profile vectors and historical click material sequences corresponding to the materials to be deployed in the optimal action sequence of material deployment output by the Actor model are merged and transformed into a one-dimensional numerical output to obtain the material deployment value. Before the step of obtaining the user representation vector corresponding to the historical clicked material sequence, the method further includes: Construct user profile vectors for material delivery targets; Build a material delivery model; The material delivery model is trained using a preset strategy gradient algorithm, and after training, the step of obtaining the current user's historical clicked material sequence is executed through the material delivery model.

2. The material dispensing method as described in claim 1, characterized in that, The step of obtaining the user representation vector corresponding to the historical clicked material sequence includes: The user profile vector is combined with the historical click material sequence to obtain the user representation vector.

3. The material dispensing method as described in claim 2, characterized in that, Before the step of obtaining an action sequence containing multiple materials based on the material vectors in the preset material space and the user representation vectors corresponding to the historical clicked material sequences, the method further includes: Construct the material space; Obtain the material vector of each material in the material space.

4. The material dispensing method as described in claim 2, characterized in that, The step of evaluating the material deployment value of the plurality of materials to be deployed includes: Based on the multidimensional vectors corresponding to the multiple materials to be delivered, and combined with the user profile vector and the historical click material sequence, the material delivery value of this delivery is obtained.

5. A material dispensing device, characterized in that, The material dispensing device includes: The acquisition module is used to acquire the historical clicked material sequence of the current user and construct a material delivery model; the material delivery model is trained by a preset strategy gradient algorithm, and after the training is completed, the step of acquiring the historical clicked material sequence of the current user is executed through the material delivery model; The determination module is used to determine the optimal action sequence for material delivery based on the historical click material sequence. This includes: obtaining the user representation vector corresponding to the historical click material sequence; obtaining an action sequence containing multiple materials based on material vectors in a preset material space and the user representation vectors corresponding to the historical click material sequence; selecting a preset number of materials at the beginning of the action sequence as materials to be delivered, and determining the optimal action sequence for material delivery based on these materials. The user representation vectors are generated by concatenating the user profile vector with the historical click material sequence and inputting it into a multi-layer MLP layer using a neural network composed of a feature domain embedding layer and a multi-head self-attention layer. This neural network is used to deeply mine the correlation between user interests and materials. The action sequence containing multiple materials is obtained based on material vectors in a preset material space and the user representation vectors corresponding to the historical click material sequence. The material vectors in the material space are processed by the deep learning model AutoInt. The generation process integrates basic information, multimodal features, and popularity information of the materials. The iterative filtering method for the action sequence is as follows: Repeatedly execute the following steps a preset number of times: Multiply the material vector of each material in the material space with the user representation vector corresponding to the historical clicked material sequence to obtain multiple multiplication results; Obtain the material in the material space corresponding to the maximum value among the multiple multiplication results, place the material at the end of the preset initial action sequence, and delete the material from the material space to obtain an action sequence containing multiple materials; Select the preset number of materials at the beginning of the action sequence as materials to be deployed, and determine the optimal action sequence for material deployment based on the materials to be deployed. The delivery module is used to deliver multiple materials to be delivered in the optimal action sequence of material delivery, and after the delivery is completed, evaluate the delivery value of the multiple materials to be delivered. In this module, according to the Bellman equation, the k-dimensional vectors, user profile vectors and historical click material sequences corresponding to the materials to be delivered in the optimal action sequence of material delivery output by the Actor model are merged and transformed into a one-dimensional numerical output to obtain the delivery value of the materials. The building module is used to construct user profile vectors for material delivery targets.

6. A terminal device, characterized in that, The terminal device includes a memory, a processor, and a basic material dispensing program stored in the memory and executable on the processor. When the material dispensing program is executed by the processor, it implements the steps of the material dispensing method as described in any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a material dispensing program, which, when executed by a processor, implements the steps of the material dispensing method as described in any one of claims 1 to 4.

Images