Resource recommendation model training method, resource recommendation method, device and equipment

Abstract

The invention provides a resource recommendation model training method which can be applied to the technical field of artificial intelligence. The method comprises the steps that training samples and candidate resource information are input into a resource recommendation model for resource recommendation processing, and a plurality of resource recommendation sequences for the training samples are generated; comparing the respective sequence scores of the plurality of resource recommendation sequences to obtain respective relative advantage values of the plurality of resource recommendation sequences; for each resource recommendation sequence in the plurality of resource recommendation sequences, determining a resource advantage value of a recommendation resource in the resource recommendation sequence according to a relative advantage value of the resource recommendation sequence and a respective position weight of the at least one sorting position; and according to the resource advantage value of the recommended resource, training the resource recommendation model in a reinforcement learning mode to obtain a trained resource recommendation model. The invention further provides a resource recommendation method and device, electronic equipment, a storage medium and a program product.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of artificial intelligence, and more particularly, to a resource recommendation model training method, a resource recommendation method, an apparatus and a device. BACKGROUND

[0002] With the rapid development of deep learning and natural language processing technology, large language models (LLM) have shown excellent capabilities in text generation, dialogue, search recommendation and other fields. For example, in the e-commerce recommendation scenario, user search terms, user portraits and candidate product information are usually used as inputs to generate product recommendation sequences that meet user interests using LLM. Although this recommendation method based on generative models can generate recommendation results combined with context information, there are still many challenges in the recommendation effect in this specific e-commerce recommendation scenario. SUMMARY

[0003] Therefore, the present disclosure provides a resource recommendation model training method, a resource recommendation method, an apparatus, a device, a storage medium and a program product.

[0004] One aspect of the present disclosure provides a resource recommendation model training method, comprising:

[0005] inputting the training sample and the candidate resource information into the resource recommendation model for resource recommendation processing to generate a plurality of resource recommendation sequences for the training sample, wherein the training sample includes query information, attribute features and historical behavior sequences of a sample object, and the resource recommendation sequence includes at least one recommended resource corresponding to each sorting position;

[0006] comparing the sequence scores of the plurality of resource recommendation sequences to obtain a relative advantage value of each of the plurality of resource recommendation sequences;

[0007] for each resource recommendation sequence in the plurality of resource recommendation sequences, determining a resource advantage value of the recommended resource in the resource recommendation sequence according to the relative advantage value of the resource recommendation sequence and the position weight of each sorting position; and

[0008] training the resource recommendation model in a reinforcement learning manner according to the resource advantage value of the recommended resource to obtain a trained resource recommendation model.

[0009] According to an embodiment of the present disclosure, the method further comprises:

[0010] determine a position weight adjustment strategy for the sample object according to the historical behavior sequence, and adjust the position weight of each of the at least one ranking position by using the position weight adjustment strategy to obtain an adjusted position weight of each of the at least one ranking position;

[0011] determine a resource advantage value of the recommended resource in the resource recommendation sequence according to the relative advantage value of the resource recommendation sequence and the position weight of each of the at least one ranking position;

[0012] determine a resource advantage value of the recommended resource according to the relative advantage value of the resource recommendation sequence and the adjusted position weight of each of the at least one ranking position.

[0013] According to an embodiment of the present disclosure, determining a position weight adjustment strategy for the sample object according to the historical behavior sequence comprises:

[0014] determine position preference information of the sample object according to the historical behavior sequence, wherein the position preference information comprises an attention degree of the sample object to each of the at least one ranking position;

[0015] determine the position weight adjustment strategy according to the position preference information.

[0016] According to an embodiment of the present disclosure, determining a position weight adjustment strategy for the sample object according to the position preference information comprises:

[0017] determine at least one to-be-adjusted ranking position in the at least one ranking position according to the attention degree of each of the at least one ranking position;

[0018] determine the position weight adjustment strategy according to the attention degree of each of the at least one to-be-adjusted ranking position.

[0019] According to an embodiment of the present disclosure, determining at least one to-be-adjusted ranking position in the at least one ranking position according to the attention degree of each of the at least one ranking position comprises:

[0020] for each ranking position in the at least one ranking position, in response to the attention degree of the ranking position being in a preset attention degree range, determine that the ranking position is the to-be-adjusted ranking position.

[0021] According to an embodiment of the present disclosure, the preset attention degree range comprises a first attention degree sub-range and a second attention degree sub-range;

[0022] determine the position weight adjustment strategy according to the attention degree of each of the at least one to-be-adjusted ranking position.

[0023] For each of the at least one to-be-adjusted ranking position, in response to the attention degree of the to-be-adjusted ranking position being located in the first attention degree sub-range, a sub-strategy of increasing the position weight is generated;

[0024] In response to the attention degree of the to-be-adjusted ranking position being located in the second attention degree sub-range, a sub-strategy of reducing the position weight is generated;

[0025] According to the respective sub-strategy of the at least one to-be-adjusted ranking position, the position weight adjustment strategy is determined.

[0026] According to an embodiment of the present disclosure, the method further comprises:

[0027] For each of the plurality of resource recommendation sequences, the target recommended resource in the resource recommendation sequence is masked according to the historical behavior sequence, to obtain a masked resource recommendation sequence;

[0028] The determination of the resource advantage value of the recommended resource in the resource recommendation sequence comprises: determining the resource advantage value of the recommended resource in the masked resource recommendation sequence.

[0029] According to an embodiment of the present disclosure, the masking of the target recommended resource in the resource recommendation sequence according to the historical behavior sequence comprises:

[0030] According to the feedback behavior of the sample object determined by the historical behavior sequence, a target ranking position in at least one ranking position of the resource recommendation sequence is determined.

[0031] According to the target ranking position, the target recommended resource in the resource recommendation sequence is determined and masked.

[0032] According to an embodiment of the present disclosure, the feedback behavior comprises at least one of the following: click behavior, transaction behavior, and collection behavior.

[0033] According to an embodiment of the present disclosure, the method further comprises:

[0034] For each of the plurality of resource recommendation sequences, the attribute feature, the historical behavior sequence, and the resource recommendation sequence are input into the trained reward model to obtain a multi-dimensional resource reward of each of at least one recommended resource in the resource recommendation sequence;

[0035] The multi-dimensional resource rewards of the at least one recommended resource are fused to obtain the sequence score.

[0036] Another aspect of the present disclosure provides a resource recommendation method, comprising:

[0037] In response to the resource recommendation request, query information, target attribute information of a target object, a target historical behavior sequence, and candidate resource information contained in the resource recommendation request are obtained.

[0038] The query information, the target attribute information, the target historical behavior sequence, and the candidate resource information are input into a resource recommendation model for resource recommendation processing to generate a target resource recommendation result for the target object, where the resource recommendation model is obtained based on the training method.

[0039] Another aspect of the present disclosure provides a training device of a resource recommendation model, comprising:

[0040] A sequence generation module is configured to input a training sample and candidate resource information into a resource recommendation model for resource recommendation processing to generate a plurality of resource recommendation sequences for the training sample, where the training sample includes attribute features and a historical behavior sequence of a sample object, and the resource recommendation sequence includes at least one recommended resource corresponding to a respective ranking position;

[0041] A relative advantage value determination module is configured to compare sequence scores of the plurality of resource recommendation sequences to obtain a relative advantage value of each of the plurality of resource recommendation sequences.

[0042] A resource advantage value determination module is configured to, for each of the plurality of resource recommendation sequences, determine a resource advantage value of a recommended resource in the resource recommendation sequence according to the relative advantage value of the resource recommendation sequence and a position weight of the respective ranking position.

[0043] A training module is configured to train the resource recommendation model in a reinforcement learning manner according to the resource advantage value of the recommended resource to obtain a trained resource recommendation model.

[0044] Another aspect of the present disclosure provides a resource recommendation device, comprising:

[0045] An obtaining module is configured to, in response to a resource recommendation request, obtain query information, target attribute information of a target object, a target historical behavior sequence, and candidate resource information contained in the resource recommendation request.

[0046] A generation module is configured to input the query information, the target attribute information, the target historical behavior sequence, and the candidate resource information into a resource recommendation model for resource recommendation processing to generate a target resource recommendation result for the target object, where the resource recommendation model is obtained based on the training device.

[0047] Another aspect of the present disclosure provides an electronic device, comprising:

[0048] One or more processors;

[0049] a memory storing one or more programs,

[0050] wherein the one or more programs, when executed by the one or more processors, cause the one or more processors to implement the method described above.

[0051] Another aspect of the present disclosure provides a computer-readable storage medium storing computer-executable instructions for implementing the method described above.

[0052] Another aspect of the present disclosure provides a computer program product comprising computer-executable instructions for implementing the method described above.

[0053] According to the embodiments of the present disclosure, because the training sample and the candidate resource information are input into the resource recommendation model for resource recommendation processing, a plurality of resource recommendation sequences are generated, and the sequence scores of the plurality of resource recommendation sequences are compared to obtain the relative advantage values of the plurality of resource recommendation sequences; then, according to the relative advantage values of the resource recommendation sequences and the position weights of the at least one ranking position, the resource advantage values of the recommended resources in the resource recommendation sequences are determined; and then, according to the resource advantage values of the recommended resources, the resource recommendation model is trained in a reinforcement learning manner to obtain the trained resource recommendation model. The technical means fully considers the influence of the difference in recommended positions in the e-commerce recommendation scenario on the recommendation effect, and by combining the position weights of different ranking positions with different updating intensities for the recommended resources in different ranking positions after determining the relative advantage values of the resource recommendation sequences, the matching degree of the front-row resources with the user is higher, which can effectively improve the quality of the front-row resources in the resource recommendation sequence, and further improve the accuracy of resource recommendation and the user experience. BRIEF DESCRIPTION OF DRAWINGS

[0054] The above and other objects, features and advantages of the present disclosure will become more apparent from the following description of embodiments of the present disclosure taken in conjunction with the accompanying drawings, in which:

[0055] Figure 1 An exemplary system architecture to which the training method of the resource recommendation model according to the embodiments of the present disclosure can be applied is schematically shown;

[0056] Figure 2 A flowchart of the training method of the resource recommendation model according to the embodiments of the present disclosure is schematically shown;

[0057] Figure 3 A schematic diagram of a method for determining a position weight adjustment strategy according to the embodiments of the present disclosure is schematically shown;

[0058] Figure 4A schematic diagram illustrating a mask processing according to an embodiment of the present disclosure is shown;

[0059] Figure 5 A flowchart illustrating a method of training a resource recommendation model according to an embodiment of the present disclosure is shown;

[0060] Figure 6 A flowchart illustrating a resource recommendation method according to an embodiment of the present disclosure is shown;

[0061] Figure 7 A block diagram illustrating a training apparatus of a resource recommendation model according to an embodiment of the present disclosure is shown;

[0062] Figure 8 A block diagram illustrating a resource recommendation apparatus according to an embodiment of the present disclosure is shown; and

[0063] Figure 9 A block diagram of an electronic device suitable for implementing the above-described methods according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0064] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. It is to be understood, however, that the description is merely illustrative of the present disclosure and in no way limits the scope of the present disclosure. In the following detailed description of embodiments of the present disclosure, numerous specific details are set forth in order to provide a thorough understanding of the present disclosure. However, it will be apparent to one skilled in the art that one or more embodiments of the present disclosure can be practiced without these specific details. In other instances, well-known structures and functions have been omitted to avoid obscuring the present disclosure in unnecessary detail.

[0065] The terms used herein are merely used to describe specific embodiments and are not intended to limit the present disclosure. The terms "include" and "have" and the like used herein indicate the presence of the described features, steps, operations, and / or components, but do not preclude the presence or addition of one or more other features, steps, operations, or components.

[0066] All terms used herein, including technical and scientific terms, have the same meanings as those generally understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein should be interpreted as having meanings consistent with the context of the present description, and should not be interpreted in an idealized or overly formal manner.

[0067] In the case of using expressions similar to "at least one of A, B, and C, etc.", it is generally to be interpreted as including one or more of the items enumerated in the list (e.g., "a system having at least one of A, B, and C" should be interpreted as including a system having A alone, a system having B alone, a system having C alone, a system having both A and B together, a system having both A and C together, a system having both B and C together, and / or a system having all of A, B, and C together, etc.).

[0068] In the embodiments of the present disclosure, the collection, updating, analysis, processing, use, transmission, provision, disclosure, storage, etc. of the data involved (for example, including but not limited to user personal information) comply with the relevant legal regulations, are used for legal purposes, and do not violate public order and good customs. In particular, necessary measures are taken for user personal information to prevent illegal access to user personal information data and to maintain user personal information security and network security.

[0069] In the embodiments of the present disclosure, the authorization or consent of the user is obtained before the user personal information is acquired or collected.

[0070] Term explanation:

[0071] Token: refers to a basic unit used for model training or generation in natural language processing, which can represent a word, a token or a symbol, etc.

[0072] Mask: refers to the operation of selectively masking or "not updating" certain sequence positions (or tokens) in the training update, so as to achieve more accurate or efficient training of the key part.

[0073] With the rapid development of deep learning and natural language processing technology, LLM has shown excellent ability in text generation, dialogue, search recommendation and other fields. When training LLM, reinforcement learning (Reinforcement Learning) method is often used to continuously optimize the performance of the model in downstream tasks (such as ranking, dialogue quality, etc.).

[0074] In some related examples, the commonly used reinforcement learning method such as proximal policy optimization (Proximal Policy Optimization, PPO) needs a separate value evaluation model to score the model output, and the policy update is guided by the value function or advantage function. However, such methods often have high computational cost when training LLM, and the value model needs to be constantly fine-tuned, which is relatively complex.

[0075] To solve the above problems, in some other related examples, a GRPO (Group Relative Policy Optimization) algorithm is used for model training. The core idea is: generate multiple candidate responses (i.e., a "group") for the same input query, and then compare each response in the group to calculate the relative advantage. In this way, there is no need to train a value model separately, and an effective reward signal can be obtained through group comparison. When calculating the relative advantage, the group average score μ and the standard deviation σ are usually normalized: Aᵢ = (rᵢ - μ) / σ. The reward value determined by the group comparison determines the advantage value of the entire sequence or token, and the strategy is updated. This method does not require a value evaluation model to score the output, and the cost is lower during training.

[0076] However, in the e-commerce search recommendation scenario, users often pay more attention to the products ranked at the top, and the actual business side has more stringent requirements or needs more exposure opportunities for the selection and presentation of the top products. The GRPO in related examples implements a "one-size-fits-all" reward mechanism for all tokens (or all recommended products) in the same sequence, which has a low degree of adaptation to the e-commerce search recommendation scenario, resulting in poor recommendation results.

[0077] In addition, the GRPO in related examples cannot make fine-grained or multi-grained updates for different business needs, and simply updates all sequences or all tokens uniformly, which is difficult to meet the multi-scenario needs of e-commerce businesses. For example: business A only needs to improve the click-through rate of the top 3 products (focusing on the main promoted products); business B needs to focus on the overall conversion rate of the top 5 or top 10 products.

[0078] To address the above technical problems, embodiments of the present disclosure provide a training method of a resource recommendation model, including: inputting a training sample and candidate resource information into a resource recommendation model for resource recommendation processing to generate multiple resource recommendation sequences for the training sample; comparing sequence scores of the multiple resource recommendation sequences to obtain relative advantage values of the multiple resource recommendation sequences; for each resource recommendation sequence in the multiple resource recommendation sequences, determining resource advantage values of recommended resources in the resource recommendation sequence according to the relative advantage value of the resource recommendation sequence and position weights of the at least one ranking position; and training the resource recommendation model in a reinforcement learning manner according to the resource advantage values of the recommended resources to obtain a trained resource recommendation model. By assigning different update intensities to recommended resources at different ranking positions according to the position weights, the front row resources of the resource recommendation sequence can be focused on, thereby improving the adaptation to the e-commerce search recommendation scenario and improving the recommendation results.

[0079] Figure 1An exemplary system architecture 100 to which the training method of the resource recommendation model according to an embodiment of the present disclosure can be applied is schematically shown. It should be noted that Figure 1 The system architecture shown is merely an example of the system architecture to which the embodiments of the present disclosure can be applied, to help those skilled in the art understand the technical content of the present disclosure, but does not mean that the embodiments of the present disclosure cannot be used in other devices, systems, environments or scenarios.

[0080] As Figure 1 The system architecture 100 according to this embodiment can include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104 and a server 105, as shown. The network 104 is a medium for providing a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103 and the server 105. The network 104 can include various connection types, such as wired and / or wireless communication links, etc.

[0081] The user can use the first terminal device 101, the second terminal device 102, the third terminal device 103 to interact with the server 105 through the network 104 to receive or send messages, etc. Various communication client applications can be installed on the first terminal device 101, the second terminal device 102, the third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients and / or social platform software, etc. (only as examples).

[0082] The first terminal device 101, the second terminal device 102, the third terminal device 103 can be various electronic devices with display screens and supporting web browsing, including but not limited to smartphones, tablet computers, laptop computers and desktop computers, etc.

[0083] The server 105 can be a server providing various services, such as a background management server providing support for websites browsed by users using the first terminal device 101, the second terminal device 102, the third terminal device 103 (only as an example). The background management server can analyze and process received user requests and other data, and feed back the processing results (such as web pages, information or data generated according to user requests, etc.) to the terminal device.

[0084] It should be noted that the training method of the resource recommendation model provided in the embodiments of the present disclosure can generally be executed by the server 105. Correspondingly, the training apparatus of the resource recommendation model provided in the embodiments of the present disclosure can generally be arranged in the server 105. The training method of the resource recommendation model provided in the embodiments of the present disclosure can also be executed by a server or a server cluster different from the server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105. Correspondingly, the training apparatus of the resource recommendation model provided in the embodiments of the present disclosure can also be arranged in a server or a server cluster different from the server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105. Alternatively, the training method of the resource recommendation model provided in the embodiments of the present disclosure can also be executed by the first terminal device 101, the second terminal device 102 or the third terminal device 103, or by other terminal devices different from the first terminal device 101, the second terminal device 102 or the third terminal device 103. Correspondingly, the training apparatus of the resource recommendation model provided in the embodiments of the present disclosure can also be arranged in the first terminal device 101, the second terminal device 102 or the third terminal device 103, or in other terminal devices different from the first terminal device 101, the second terminal device 102 or the third terminal device 103.

[0085] For example, the training sample and the candidate resource information can originally be stored in any one of the first terminal device 101, the second terminal device 102 or the third terminal device 103 (for example, the first terminal device 101, but not limited thereto), or on an external storage device and can be imported into the first terminal device 101. Then, the first terminal device 101 can execute the training method of the resource recommendation model provided in the embodiments of the present disclosure locally, or send the training sample and the candidate resource information to other terminal devices, servers or server clusters, and execute the training method of the resource recommendation model provided in the embodiments of the present disclosure by other terminal devices, servers or server clusters receiving the training sample and the candidate resource information.

[0086] It should be understood that Figure 1 The number of terminal devices, networks and servers in the system shown in FIG. 1 is only illustrative. According to the needs of implementation, there can be any number of terminal devices, networks and servers.

[0087] Figure 2 A flowchart of the training method of the resource recommendation model according to the embodiments of the present disclosure is schematically shown.

[0088] As Figure 2 shown, the method includes operations S210-S240.

[0089] In operation S210, the training sample and the candidate resource information are input into the resource recommendation model for resource recommendation processing, to generate multiple resource recommendation sequences for the training sample.

[0090] The training sample is a data sample for training the resource recommendation model, and can include query information, attribute features, and a historical behavior sequence of the sample object. The query information of the sample object can be a search keyword or query intent actively input by the sample user, such as query content for a certain type of product or topic. The attribute features can be basic features for describing the sample object, such as region, age, consumption preference, etc. The historical behavior sequence can be a time-ordered user interaction record, which can reflect the behavior habits and preference changes of the sample object. The historical behavior sequence can include, for example, clicks, adds to cart, orders, dwell time, etc.

[0091] It should be noted that the query information, attribute features, and historical behavior sequence of the sample object in the training of the embodiments of the present application can be provided by the user and authorized for use, or obtained from a public data set, or obtained through other means that comply with relevant laws and regulations.

[0092] The candidate resource information can be a set of alternative resources (such as N products or contents) filtered by a recall system or a preliminary retrieval model, serving as an input candidate pool for the resource recommendation model.

[0093] The multiple resource recommendation sequences are multiple lists of recommended resources for the same training sample (such as the same user). Different resource recommendation sequences can be generated based on different recommendation strategies (such as value-for-money priority, style matching priority, etc.), and differ from each other.

[0094] The resource recommendation sequence includes recommended resources (such as products, contents, etc.) sorted in a certain logic, i.e., the resource recommendation sequence includes multiple sorting positions, each of which corresponds to a recommended resource. The sorting position refers to the order of the recommended resource in the resource recommendation sequence, and the earlier the position, the higher the matching degree of the resource to the user demand. The recommended resource refers to a specific resource (such as a certain product, an article, etc.) selected from the candidate resource information and finally recommended to the sample object.

[0095] The resource recommendation model can be a generative model, such as a large language model.

[0096] In some examples, inputting the training sample and the candidate resource information into the resource recommendation model for resource recommendation processing to generate multiple resource recommendation sequences for the training sample can include: inputting the training sample and the candidate resource information into a large language model, and requiring multiple resource recommendation sequences to be generated in the prompt. For example, requiring 3 resource recommendation sequences to be generated in the prompt: value ascending order recommendation, user historical preference matching recommendation, and product popularity recommendation.

[0097] It should be noted that the candidate resource information in the training model in the embodiments of the present application can be provided by an e-commerce platform and authorized to be used, or obtained from a public data set, or obtained through other means in accordance with relevant laws and regulations.

[0098] In operation S220, the sequence scores of the plurality of resource recommendation sequences are compared to obtain the relative advantage values of the plurality of resource recommendation sequences.

[0099] The sequence score, i.e., the sequence reward, represents a numerical value for quantitatively evaluating the overall quality or effectiveness of a single resource recommendation sequence. The basis for the score usually includes the matching degree of the sequence to the user's demand (such as whether it meets the user's query information, attribute characteristics), the relevance of the resources within the sequence (such as whether there is a logical association between resources), the user's potential satisfaction (such as the click probability or order probability predicted based on historical behavior sequences), etc. For example, resource recommendation sequence A can get 80 points because it highly matches the consumption preferences of the sample object, and resource recommendation sequence B can get 60 points because of insufficient diversity.

[0100] In some embodiments, a reward model can be used to score the plurality of resource recommendation sequences. The reward model is a scoring or scoring model for evaluating the quality of generated sequences, which guides the policy update in reinforcement learning by assigning a corresponding score to each sequence.

[0101] The relative advantage value can be the degree of advantage exhibited by a certain resource recommendation sequence relative to other resource recommendation sequences after comparing the sequence scores of the plurality of resource recommendation sequences.

[0102] In some embodiments, the relative advantage values of the plurality of resource recommendation sequences can be calculated using the GRPO algorithm, and the average value μ and the standard deviation σ of the plurality of resource recommendation sequences can be calculated using the following formula (1), and the relative advantage value A can be calculated using formula (2).

[0103] (1);

[0104] (2);

[0105] Wherein, n is the number of resource recommendation sequences, r i is the sequence score of the i-th resource recommendation sequence, and Ai is the relative advantage value of the i-th resource recommendation sequence.

[0106] At operation S230, for each resource recommendation sequence in the plurality of resource recommendation sequences, a resource advantage value of a recommended resource in the resource recommendation sequence is determined according to the relative advantage value of the resource recommendation sequence and a position weight of each of the at least one ranking position.

[0107] The position weight is a weight coefficient set for each ranking position according to factors such as the degree of influence of the ranking position on the user, and is used to adjust the advantage value of the recommended resource, and reflects the importance degree that the ranking position should be given in the update or advantage calculation. The earlier the ranking position is, the greater the weight is, for example, the weight of the first position is set to 1.0, and the weight of the second position is set to 0.8.

[0108] Determining the resource advantage value of the recommended resource in the resource recommendation sequence according to the relative advantage value of the resource recommendation sequence and the position weight of each of the at least one ranking position can include multiplying the relative advantage value of the resource recommendation sequence by the position weight of each of the at least one ranking position to obtain the resource advantage value of the recommended resource in the resource recommendation sequence.

[0109] In some examples, the resource advantage value can be calculated according to the following formula (3).

[0110] Aᵢ(token j ) = Aᵢ × w(j) (3);

[0111] Wherein, Aᵢ represents the i-th resource recommendation sequence, Aᵢ(token j ) represents the resource advantage value of the recommended resource in the j-th ranking position in the i-th resource recommendation sequence, and w(j) represents the position weight of the j-th ranking position.

[0112] By weighting the relative advantage value of the resource recommendation sequence to the specific recommended resource according to the position importance, a more accurate advantage quantification result of the single recommended resource in the recommendation system is obtained to assist the optimization of the resource recommendation strategy of the recommendation system.

[0113] At operation S240, the resource recommendation model is trained in a reinforcement learning manner according to the resource advantage value of the recommended resource to obtain a trained resource recommendation model.

[0114] The resource advantage value is applied to a policy gradient or a similar optimization method as an immediate reward or advantage estimate, and the parameters of the resource recommendation model in generating the resource recommendation sequence are updated.

[0115] For example, the resource recommendation sequence includes resource A and resource B, and the resource advantage values of resource A and resource B are 0.9 and 0.64 respectively. If the sample object clicks resource A, the model updates through policy gradient, and strengthens the ranking of resource A in similar context. If the sample object skips resource A but clicks resource B, the ranking weight of resource A is reduced, and the parameters of the model are adjusted.

[0116] According to the technical means of the embodiment of the present disclosure, the training sample and the candidate resource information are input into the resource recommendation model for resource recommendation processing, a plurality of resource recommendation sequences are generated, and the sequence scores of the plurality of resource recommendation sequences are compared to obtain the relative advantage values of the plurality of resource recommendation sequences. Then, according to the relative advantage values of the resource recommendation sequences and the position weights of the at least one ranking position, the resource advantage values of the recommended resources in the resource recommendation sequences are determined. Then, according to the resource advantage values of the recommended resources, the resource recommendation model is trained in a reinforcement learning manner to obtain the trained resource recommendation model. The technical means fully considers the influence of the difference in the recommended position in the e-commerce recommendation scene on the recommendation effect. By combining the position weights of different ranking positions with different updating intensities of the recommended resources in different ranking positions after determining the relative advantage values of the resource recommendation sequences, the matching degree of the front row resources and the user is higher, which can effectively improve the quality of the front row resources in the resource recommendation sequence, and further improve the accuracy of the resource recommendation and the user experience.

[0117] According to the embodiment of the present disclosure, the above method further comprises: for each resource recommendation sequence in the plurality of resource recommendation sequences, inputting the attribute feature, the historical behavior sequence and the resource recommendation sequence into the trained reward model to obtain the multi-dimensional resource reward of each of the at least one recommended resource in the resource recommendation sequence; and fusing the multi-dimensional resource rewards of the at least one recommended resource to obtain the sequence score.

[0118] The trained reward model is used as a judge in reinforcement learning to quantify the quality of the recommended resources.

[0119] The multi-dimensional resource reward includes the revenue of the recommended resources evaluated from different dimensions, for example, the business dimension: click probability, add-to-cart probability, transaction probability, conversion probability; the user experience dimension: satisfaction estimation, novelty, diversity; and the platform dimension: resource cold start intensity, brand weight. For example, for a recommended resource A, the corresponding multi-dimensional resource reward can be [click reward: 0.7, add-to-cart reward: 0.3, transaction reward: 0.5].

[0120] In some embodiments, fusing the multi-dimensional resource rewards of the at least one recommended resource can include using weighted linear fusion. Specifically, a weight coefficient W click, W cart, W order is assigned to each of the three dimensions of click probability, cart probability, and transaction probability, where W click, W cart, W order > 0 and W click + W cart + W order = 1; for each recommended resource in the resource recommendation sequence, the scores of the resource in the three dimensions are multiplied by the corresponding weight coefficients and summed to obtain a sequence score of the resource recommendation sequence, denoted as S total sample.

[0121] In some embodiments, after determining S total sample, a difference or ratio can be calculated based on S total sample and the reward S total expo of the real exposure sequence to determine the final sequence score.

[0122] According to embodiments of the present disclosure, before inputting the attribute features, the historical behavior sequence, and the resource recommendation sequence into the trained reward model to output the multi-dimensional resource rewards of each of the at least one recommended resource in the resource recommendation sequence, the method further comprises: constructing a reward model based on a multi-task supervised learning framework, inputting the training samples and candidate resource information into the reward model, outputting the click probability, cart probability, and transaction probability of each candidate resource, and training the reward model based on multi-granularity cross-entropy loss (click probability cross-entropy loss, cart probability cross-entropy loss, and transaction probability cross-entropy loss) and joint loss (e.g., weighted sum of click probability cross-entropy loss, cart probability cross-entropy loss, and transaction probability cross-entropy loss) to obtain the trained reward model.

[0123] According to embodiments of the present disclosure, the method further comprises: determining a position weight adjustment strategy for the sample object based on the historical behavior sequence, and adjusting the position weight of each of the at least one ranking position using the position weight adjustment strategy to obtain an adjusted position weight of each of the at least one ranking position; and determining the resource advantage value of the recommended resource in the resource recommendation sequence based on the relative advantage value of the resource recommendation sequence and the position weight of each of the at least one ranking position, which comprises: determining the resource advantage value of the recommended resource based on the relative advantage value of the resource recommendation sequence and the adjusted position weight of each of the at least one ranking position.

[0124] The position weight adjustment strategy can include a weight adjustment strategy for all ranking positions in the at least one ranking position, or a weight adjustment strategy for part of the ranking positions in the at least one ranking position.

[0125] For example, the ranking positions include three, and the position weight adjustment strategy can include a weight adjustment strategy for the first position and a weight adjustment strategy for the third position. It can also include a weight adjustment strategy for the first position, a weight adjustment strategy for the second position, and a weight adjustment strategy for the third position.

[0126] The weight adjustment strategy of each ranking position can include at least one of a weight adjustment coefficient, a weight adjustment range, and a weight adjustment target value. For example, the weight adjustment strategy for the first ranking position can include a position weight reduction of 0.1, can also include a position weight coefficient of 0.9, and can also include a position weight adjustment of 0.6.

[0127] By dynamically adjusting the weights of different ranking positions in the resource recommendation sequence according to the historical behavior sequence (such as increasing the weight of the position frequently clicked by the user and reducing the weight of the ignored position), the historical behavior preferences of the user are integrated into the position decay mechanism, so that the finally generated resource advantage value takes into account both the quality of the recommended resource itself and the position sensitivity, thereby improving the adaptability to the e-commerce search recommendation scenario and improving the recommendation effect.

[0128] According to an embodiment of the present disclosure, determining a position weight adjustment strategy for a sample object according to a historical behavior sequence includes: determining position preference information of the sample object according to the historical behavior sequence, wherein the position preference information includes respective attention degrees of the sample object to at least one ranking position; and determining the position weight adjustment strategy according to the position preference information.

[0129] The position preference information is the preference and attention degree of the sample object to different ranking positions in the resource recommendation sequence based on analysis of the historical behavior sequence, and reflects the attention tendency of the sample object to different positions when facing ordered recommended content. For example, by analyzing the historical behavior sequence, it is found that the sample object often clicks resources in the first and second positions of the resource recommendation sequence, and rarely pays attention to resources after the fifth position, and therefore the following position preference information can be summarized: the sample object has high attention to the first position and low attention to the fifth and subsequent positions.

[0130] The attention degree is a specific embodiment of the position preference information, and the method for determining the attention degree can include a basic statistical method and a machine learning method. The basic statistical method can include at least one of the following: a click rate method, a dwell time weighting method, and a comprehensive behavior score method.

[0131] In some embodiments, the click rate method can be used to determine the attention degree, and the following formula (3) can be used to determine the attention degree.

[0132] (3).

[0133] In some embodiments, the dwell time weighting method can be used to determine the attention, and the following formula (4) can be used to determine the attention.

[0134] (4) ;

[0135] wherein a is a normalization coefficient, which can be 1 / max(time length) for example.

[0136] By analyzing the historical behavior sequence, the attention of the sample to different ranking positions in the resource recommendation sequence is dynamically quantified, and a personalized position weight adjustment strategy is generated based on this, so as to overcome the mechanical deviation of the fixed position decay mechanism. At the same time, through data-driven dynamic weight adjustment, the relevance of the recommendation result is maintained, and the accuracy of personalized experience and resource distribution can be improved.

[0137] According to an embodiment of the present disclosure, determining the position weight adjustment strategy according to the position preference information comprises: determining at least one to-be-adjusted ranking position in the at least one ranking position according to the attention of each ranking position; and determining the position weight adjustment strategy according to the attention of each to-be-adjusted ranking position.

[0138] The to-be-adjusted ranking position represents a ranking position that needs to adjust the position weight.

[0139] In some embodiments, the to-be-adjusted ranking position can be determined according to a preset threshold. For example, if the attention of a ranking position is significantly greater than the preset threshold, the ranking position is determined to be a to-be-adjusted ranking position. For example, if the attention of a ranking position is significantly less than the preset threshold, the ranking position is determined to be a to-be-adjusted ranking position. The preset threshold can be determined according to the average attention of the group, for example, the average click rate.

[0140] In other embodiments, the to-be-adjusted ranking position can be determined according to a preset attention range. Specifically, for each ranking position in the at least one ranking position, in response to the attention of the ranking position being in the preset attention range, the ranking position is determined to be a to-be-adjusted ranking position.

[0141] The preset attention range can include a first attention sub-range and a second attention sub-range. The first attention sub-range can be a high attention range, for example, the first attention sub-range can be [0.7, 1.0]. The second attention sub-range can be a low attention range, for example, the second attention sub-range can be [0, 0.3].

[0142] In response to the attention of the ranking position being in the preset attention range, determining the ranking position to be a to-be-adjusted ranking position comprises: in response to the attention of the ranking position being in at least one of the first attention sub-range and the second attention sub-range, determining the ranking position to be a to-be-adjusted ranking position.

[0143] The to-be-adjusted ranking positions include to-be-adjusted ranking positions of increasing position weights and to-be-adjusted ranking positions of decreasing position weights. If the attention degree of a ranking position is located in the first attention degree sub-range, the ranking position is determined as a to-be-adjusted ranking position of increasing weight; if the attention degree of a ranking position is located in the second attention degree sub-range, the ranking position is determined as a to-be-adjusted ranking position of decreasing weight.

[0144] It should be noted that the position weight of a ranking position not in the attention degree range is not adjusted.

[0145] Exemplarily, the position weight adjustment strategy is determined according to the attention degrees of the at least one to-be-adjusted ranking position, including: for each to-be-adjusted ranking position in the at least one to-be-adjusted ranking position, in response to the attention degree of the to-be-adjusted ranking position being located in the first attention degree sub-range, generating a sub-strategy of increasing the position weight; in response to the attention degree of the to-be-adjusted ranking position being located in the second attention degree sub-range, generating a sub-strategy of decreasing the position weight; and determining the position weight adjustment strategy according to the respective sub-strategies of the at least one to-be-adjusted ranking position.

[0146] Figure 3 A schematic diagram of a method of determining a position weight adjustment strategy according to an embodiment of the present disclosure is schematically shown.

[0147] As shown in Figure 3 the method of determining a position weight adjustment strategy of this embodiment includes: determining position preference information 320 according to a historical behavior sequence 310, where the position preference information 320 includes the attention degrees of 10 recommended positions, such as the attention degree of position 1, the attention degree of position 2, …, and the attention degree of position 10. Then, to-be-adjusted recommended positions 340 in the 10 recommended positions are determined according to the position preference information 320 and a preset attention degree range 330. In this embodiment, the to-be-adjusted recommended positions 340 include, for example, position 1, position 2, position 3, and position 4. Subsequently, the sub-strategy of each position is determined according to the attention degree of each of position 1, position 2, position 3, and position 4 and the first attention degree sub-range and the second attention degree sub-range. For example, the attention degrees of position 1 and position 2 are located in the first attention degree sub-range, so the sub-strategy of position 1 and position 2 is determined as a sub-strategy 350 of increasing the position weight; the attention degrees of position 3 and position 4 are located in the second attention degree sub-range, so the sub-strategy of position 3 and position 4 is determined as a sub-strategy 360 of decreasing the position weight; and finally, the position weight adjustment strategy 370 is determined according to the sub-strategy 350 of increasing the position weight and the sub-strategy 360 of decreasing the position weight.

[0148] By determining the to-be-adjusted ranking position according to the preset attention range, and implementing different strategies for different attention sub-ranges, and finally synthesizing the position weight adjustment strategy, the ranking position that needs to be intervened can be accurately positioned, and excessive intervention on normal positions can be avoided.

[0149] According to the disclosed embodiment, the above method can further include: making dynamic adjustment on the position weight according to time, festival activities, business strategy, etc.

[0150] For example, an external event signal and a corresponding weight configuration template are pre-configured. The external event signal can include time (such as a specific date, a specific period), festival activity identification, business strategy instructions (such as “promote new products” and “improve exposure of long-tail products”). The weight configuration template can be a pre-defined position weight table associated with a specific event or strategy. For example: promotion template: w(1)=1.0, w(2)=0.9, w(3)=0.7, … (the first few weights are higher, emphasizing the head effect); exploration template: w(1)=0.8, w(2)=0.8, w(3)=0.8, … (the weight is more gentle, encouraging the model to pay attention to more positions); long tail template: w(1)=0.7, w(2)=0.7, w(3)=0.9, w(4)=1.0, … (reduce the head weight and increase the middle and rear position weight).

[0151] When the external event signal is detected, the weight of different ranking positions is adjusted using the weight configuration template corresponding to the external event signal. For example, the external event signal is a festival activity identifier, and the corresponding promotion template can be used to adjust the position weight.

[0152] In the e-commerce scenario, it may only be necessary to optimize the front row of products. For example, the first three resources in the resource recommendation sequence are important during a certain promotion period, and the products behind have limited contribution to conversion. If all ranking positions are updated, it will consume too many computing resources and disperse the optimization focus. Therefore, before adjusting the position weight of each of the at least one ranking position using the position weight adjustment strategy, the recommended resource sequence can be dynamically Masked, that is, according to different business objectives, different update granularities (Top3, Top5, Top7, Top10…) are used to filter the update range, and the update range can be further determined in combination with the behavior feedback of the sample object (such as whether the sample object actually orders the product located in the first three positions, if so, only the first three positions are updated).

[0153] According to an embodiment of the present disclosure, the masking processing on the recommended resource sequence can include: for each of the plurality of resource recommendation sequences, performing masking processing on a target recommended resource in the resource recommendation sequence according to the historical behavior sequence to obtain a masked resource recommendation sequence; and determining the resource advantage value of the recommended resource in the resource recommendation sequence includes: determining the resource advantage value of the recommended resource in the masked resource recommendation sequence.

[0154] The number of recommended resources in the masked resource recommendation sequence can be less than the number of recommended resources in the resource recommendation sequence before masking. For example, the resource recommendation sequence A includes 10 recommended resources in order, and the masked resource recommendation sequence A includes the first 5 recommended resources.

[0155] Figure 4 An illustrative diagram of the masking processing according to an embodiment of the present disclosure is schematically shown.

[0156] As shown in Figure 4 , for the resource recommendation sequence 440, the following method is used for masking processing: first, the feedback behavior 420 is determined according to the historical behavior sequence 410 of the sample object, and then the target recommended position 430 for the resource recommendation sequence 440 is determined according to the feedback behavior 420; then the target recommended resource 450 in the resource recommendation sequence 440 is determined according to the target recommended position 430, and the target recommended resource 450 is masked to obtain the masked resource recommendation sequence 460.

[0157] The feedback behavior includes at least one of the following: click behavior, transaction behavior, and collection behavior. The click behavior represents the user's active click operation on the recommended resource. The transaction behavior represents the user's conversion behavior such as completing an order or payment. The collection behavior represents the user's active collection of products or content, such as adding to a wish list.

[0158] In some examples, according to the feedback behavior of the sample object determined by the historical behavior sequence, determining the target ranking position in the at least one ranking position of the resource recommendation sequence can include: the feedback behavior represents that the product ordered by the sample object is located in the first three recommended positions, and the update granularity is Top3, that is, only the recommended resources in the first three recommended positions are updated, the third recommended position is the target ranking position, and the recommended resources in the fourth and subsequent positions are masked, at this time, the recommended resources in the fourth and subsequent positions are the target recommended resources.

[0159] In some examples, the feedback behavior represents that the product ordered by the sample object is not in the third recommended position, but in the fifth recommended position, and the granularity is switched to Top5, that is, only the recommended resources in the first five recommended positions are updated, and therefore, the recommended resources in the sixth and subsequent positions are masked.

[0160] In some embodiments, whether being clicked (or staying) can also be used as a criterion for determining the recommended position of the target. For example, if the sample object clicks but does not order the product located in the top three recommended positions, it can be determined that the update granularity is Top3.

[0161] In some examples, if there are different requirements for system resources or training efficiency, the Top3 granularity, Top5 granularity, Top7 granularity, and Top10 granularity can be dynamically adjusted automatically or manually.

[0162] By analyzing the feedback behavior to determine the update granularity (such as Top3, Top5, and Top10) of the resource recommendation sequence, and performing mask processing on the subsequent position resources that exceed the current granularity, the key ranking area can be adaptively focused, and the model resources can be concentrated to optimize the high-value positions. In addition, through the dynamic mask mechanism driven by historical behavior, the balance between recommendation accuracy and computing efficiency is achieved, which helps to reduce the training time. In addition, this method supports flexible configuration of multiple granularities, which can adapt to the needs of different business scenarios and dynamically adjust the computing scale according to the system load, thereby improving the system efficiency while ensuring the recommendation effect.

[0163] Figure 5 A flowchart of a training method of a resource recommendation model according to an embodiment of the present disclosure is schematically shown.

[0164] As shown in Figure 5 The training method of the resource recommendation model of this embodiment includes: inputting the training sample 511, such as the query information, attribute features, historical behavior sequence, and candidate resource information 512 of the sample object, into the resource recommendation model 520, and outputting a plurality of resource recommendation sequences 530 for the sample object. Then, the plurality of resource recommendation sequences 530 are input into the reward model 540 for scoring to obtain the sequence reward 550 of each resource recommendation sequence. Then, the relative advantage value 556 of each resource recommendation sequence is obtained by comparing within the group according to the sequence reward 550 of each resource recommendation sequence. Then, for each resource recommendation sequence, the mask processing is performed using the multi-granularity mask mechanism to obtain a plurality of mask-processed resource recommendation sequences 570. Then, for each resource recommendation sequence, the resource advantage value 590 of the recommended resource at each recommended position is determined using the position weight corresponding to the resource recommendation sequence. Finally, the resource advantage value 590 is applied to the policy gradient or similar optimization method to update the parameters of the resource recommendation model 520, and the above operations are iteratively performed until the stopping condition is reached, thereby obtaining the trained resource recommendation model.

[0165] By first using dynamic mask processing to discretely select the recommended resources, and then using the position weight for weighting, the method can flexibly adapt to different business scenarios.

[0166] In some embodiments, applying the resource advantage value 590 to a policy gradient or similar optimization method, the parameter update of the resource recommendation model 520 can include the following calculation of the policy gradient and parameter update process.

[0167] (1) Policy Gradient Calculation: The goal is to minimize the negative expected reward of the policy, or equivalently, to maximize the expected reward. The policy gradient theorem shows that the gradient of the expected reward with respect to the parameters can be estimated as follows in equation (5):

[0168] (5);

[0169] where N represents the number of resource recommendation sequences included in a batch; represents the number of recommended resources in the i-th resource recommendation sequence that are not masked, i.e., the number of recommended resources after masking the resource recommendations; represents the recommended resource in the i-th resource recommendation sequence at the j-th ranking position; represents the context relied on when generating , usually the Token sequence before the recommendation position and possibly other features; represents the policy function of the resource recommendation model, i.e., given the context , the model selects the probability (determined by the current parameters ) of generating ; represents the gradient of the log probability of the selected action with respect to the model parameters , which can be obtained through forward calculation of the model and automatic differentiation; Aᵢ(token j ) is the resource advantage value of the j-th ranking position in the i-th resource recommendation sequence after masking processing (determining whether to participate in updating) and position weighting (determining the updating intensity w(j).

[0170] (2) Parameter Update: Using the calculated policy gradient , combined with an optimizer (such as Adam), the parameters of the resource recommendation model are updated, as follows in equation (6):

[0171] (6);

[0172] where represents the learning rate, controlled by the optimizer; represents the parameters before updating; represents the parameters after updating.

[0173] Optimizers (such as Adam) usually compute the moving average of the gradient (first moment) and uncentered variance (second moment) to make more stable and efficient updates, but the core step is still based on the above gradient estimates.

[0174] After multiple rounds of iteration using the above method, the resource recommendation model obtained can give resources that match the user's preferences more in the front row position; a small amount of updating (or direct non-updating) is performed on unimportant or low point display ratio and conversion rate positions to save resources; and the attention to the front row position and the mask range can be continuously self-adaptively fine-tuned according to actual user behavior data, so as to achieve optimal sorting quality.

[0175] Figure 6 A flowchart of a resource recommendation method according to an embodiment of the present disclosure is schematically shown.

[0176] As shown in Figure 6 , the method includes operation S610 to operation S620.

[0177] In operation S610, in response to a resource recommendation request, query information contained in the resource recommendation request, target attribute information of a target object, a target historical behavior sequence, and candidate resource information are obtained.

[0178] The query information is a search keyword or query intent actively input by the target object. The target attribute information is used to describe the basic characteristics of the target object, such as region, age, consumption preference, etc. The target historical behavior sequence is the same as the format of the historical behavior sequence of the sample object, which will not be repeated here.

[0179] In operation S620, the query information, the target attribute information, the target historical behavior sequence, and the candidate resource information are input into a resource recommendation model for resource recommendation processing to generate a target resource recommendation result for the target object, wherein the resource recommendation model is obtained based on the training method described above.

[0180] Figure 7 A block diagram of a training device of a resource recommendation model according to an embodiment of the present disclosure is schematically shown.

[0181] As shown in Figure 7 , the training device 700 of the resource recommendation model of this embodiment includes a sequence generation module 710, a relative advantage value determination module 720, a resource advantage value determination module 730, and a training module 740.

[0182] The sequence generation module 710 is configured to input the training sample and candidate resource information into a resource recommendation model for resource recommendation processing, and generate a plurality of resource recommendation sequences for the training sample, wherein the training sample includes attribute features and a historical behavior sequence of a sample object, and the resource recommendation sequence includes at least one recommended resource corresponding to a respective ranking position.

[0183] The relative advantage value determination module 720 is configured to compare sequence scores of the plurality of resource recommendation sequences to obtain a relative advantage value of each of the plurality of resource recommendation sequences.

[0184] The resource advantage value determination module 730 is configured to determine, for each of the plurality of resource recommendation sequences, a resource advantage value of a recommended resource in the resource recommendation sequence according to the relative advantage value of the resource recommendation sequence and a position weight of the respective ranking position.

[0185] The training module 740 is configured to train the resource recommendation model in a reinforcement learning manner according to the resource advantage value of the recommended resource, to obtain a trained resource recommendation model.

[0186] According to an embodiment of the present disclosure, the apparatus further includes an adjustment module.

[0187] The adjustment module is configured to determine a position weight adjustment strategy for the sample object according to the historical behavior sequence, and adjust the position weight of the respective ranking position by using the position weight adjustment strategy to obtain an adjusted position weight of the respective ranking position.

[0188] According to an embodiment of the present disclosure, the resource advantage value determination module is further configured to determine the resource advantage value of the recommended resource according to the relative advantage value of the resource recommendation sequence and the adjusted position weight of the respective ranking position.

[0189] According to an embodiment of the present disclosure, the apparatus further includes a first determination module and a second determination module.

[0190] The first determination module is configured to determine position preference information of the sample object according to the historical behavior sequence, wherein the position preference information includes an attention degree of the sample object to the respective ranking position.

[0191] The second determination module is configured to determine the position weight adjustment strategy according to the position preference information.

[0192] According to an embodiment of the present disclosure, the second determination module includes a first determination submodule and a second determination submodule.

[0193] The first determining sub-module is configured to determine at least one to-be-adjusted ranking position from the at least one ranking position according to the attention degree of each of the at least one ranking position.

[0194] The second determining sub-module is configured to determine the position weight adjustment strategy according to the attention degree of each of the at least one to-be-adjusted ranking position.

[0195] According to an embodiment of the present disclosure, the first determining sub-module comprises a first determining unit.

[0196] The first determining unit is configured to determine, for each ranking position in the at least one ranking position, the ranking position as the to-be-adjusted ranking position in response to the attention degree of the ranking position being within a preset attention degree range.

[0197] According to an embodiment of the present disclosure, the preset attention degree range comprises a first attention degree sub-range and a second attention degree sub-range.

[0198] According to an embodiment of the present disclosure, the second determining sub-module comprises a first generating unit, a second generating unit and a second determining unit.

[0199] The first generating unit is configured to generate, for each to-be-adjusted ranking position in the at least one to-be-adjusted ranking position, a sub-strategy of increasing the position weight in response to the attention degree of the to-be-adjusted ranking position being within the first attention degree sub-range.

[0200] The second generating unit is configured to generate a sub-strategy of decreasing the position weight in response to the attention degree of the to-be-adjusted ranking position being within the second attention degree sub-range.

[0201] The second determining unit is configured to determine the position weight adjustment strategy according to the sub-strategy of each of the at least one to-be-adjusted ranking position.

[0202] According to an embodiment of the present disclosure, the device further comprises a mask processing module.

[0203] The mask processing module is configured to perform mask processing on a target recommended resource in each resource recommendation sequence in the plurality of resource recommendation sequences according to the historical behavior sequence to obtain a masked resource recommendation sequence.

[0204] According to an embodiment of the present disclosure, the resource advantage value determining module is further configured to determine a resource advantage value of a recommended resource in the masked resource recommendation sequence.

[0205] According to an embodiment of the present disclosure, the mask processing module comprises a third determining sub-module and a mask processing sub-module.

[0206] The third determining sub-module is configured to determine a target ranking position in at least one ranking position of the resource recommendation sequence according to the feedback behavior of the sample object determined by the historical behavior sequence.

[0207] The mask processing sub-module is configured to determine a target recommended resource in the resource recommendation sequence according to the target ranking position, and perform mask processing on the target recommended resource.

[0208] According to an embodiment of the present disclosure, the feedback behavior includes at least one of a click behavior, a transaction behavior, and a collection behavior.

[0209] According to an embodiment of the present disclosure, the device further includes a resource reward determining module and a fusion module.

[0210] The resource reward determining module is configured to input the attribute feature, the historical behavior sequence, and the resource recommendation sequence into a trained reward model for each resource recommendation sequence in the plurality of resource recommendation sequences, to obtain a multi-dimensional resource reward of each of at least one recommended resource in the resource recommendation sequence.

[0211] The fusion module is configured to fuse the multi-dimensional resource rewards of the at least one recommended resource to obtain the sequence score.

[0212] Any one or more of the modules, sub-modules, units, and sub-units according to the embodiments of the present disclosure, or at least part of the functions of any one or more of the modules, sub-modules, units, and sub-units, can be implemented in one module. Any one or more of the modules, sub-modules, units, and sub-units according to the embodiments of the present disclosure can be split into multiple modules for implementation. Any one or more of the modules, sub-modules, units, and sub-units according to the embodiments of the present disclosure can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on chip, a system on a substrate, a system in a package, an application specific integrated circuit (ASIC), or any other reasonable hardware or firmware that can be integrated or packaged with a circuit, or can be implemented in any one of software, hardware, and firmware, or in an appropriate combination of any one or more of the three. Alternatively, one or more of the modules, sub-modules, units, and sub-units according to the embodiments of the present disclosure can be at least partially implemented as computer program modules that can perform corresponding functions when the computer program modules are run.

[0213] For example, any of the sequence generation module 710, the relative advantage value determination module 720, the resource advantage value determination module 730, and the training module 740 can be combined in one module / unit / subunit for implementation, or any of the modules / units / subunits can be split into multiple modules / units / subunits. Alternatively, at least part of the functions of one or more of the modules / units / subunits can be combined with at least part of the functions of other modules / units / subunits and implemented in one module / unit / subunit. According to an embodiment of the present disclosure, at least one of the sequence generation module 710, the relative advantage value determination module 720, the resource advantage value determination module 730, and the training module 740 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on chip, a system on board, a system on package, an application specific integrated circuit (ASIC), or any other reasonable manner that can be integrated or packaged by a circuit, etc. hardware or firmware, or any one of software, hardware, and firmware or a suitable combination of any of them. Alternatively, at least one of the sequence generation module 710, the relative advantage value determination module 720, the resource advantage value determination module 730, and the training module 740 can be at least partially implemented as a computer program module that can perform corresponding functions when executed.

[0214] It should be noted that the training device part of the resource recommendation model in the embodiments of the present disclosure corresponds to the training method part of the resource recommendation model in the embodiments of the present disclosure, and the description of the training device part of the resource recommendation model is specifically referred to the training method part of the resource recommendation model, which will not be repeated here

[0215] Figure 8 A block diagram of a resource recommendation device according to an embodiment of the present disclosure is schematically shown.

[0216] As Figure 8 shown, the resource recommendation device 800 of this embodiment includes an acquisition module 810 and a generation module 820.

[0217] The acquisition module 810 is configured to, in response to a resource recommendation request, acquire query information, target attribute information of a target object, a target historical behavior sequence, and candidate resource information contained in the resource recommendation request.

[0218] The generation module 820 is configured to input the query information, the target attribute information, the target historical behavior sequence, and the candidate resource information into a resource recommendation model for resource recommendation processing to generate a target resource recommendation result for the target object, wherein the resource recommendation model is obtained based on the training device.

[0219] Any of the modules, sub-modules, units, sub-units, or at least part of any of them according to the embodiments of the present disclosure can be implemented in one module. Any of the modules, sub-modules, units, sub-units according to the embodiments of the present disclosure can be split into multiple modules for implementation. Any of the modules, sub-modules, units, sub-units according to the embodiments of the present disclosure can be implemented at least in part as a hardware circuit, for example, a field-programmable gate array (FPGA), a programmable logic array (PLA), a system on chip, a system on board, a system on package, an application-specific integrated circuit (ASIC), or any other reasonable manner of hardware or firmware through integration or packaging of circuits, or in any one of software, hardware, and firmware, or in a proper combination of any of them. Alternatively, any of the modules, sub-modules, units, sub-units according to the embodiments of the present disclosure can be implemented at least in part as computer program modules, which can perform corresponding functions when executed.

[0220] For example, any of the modules 810 and 820 can be combined in one module / unit / sub-unit for implementation, or any of the modules / unit / sub-unit can be split into multiple modules / unit / sub-units. Alternatively, at least part of the functions of one or more of the modules / unit / sub-units can be combined with at least part of the functions of other modules / unit / sub-units, and implemented in one module / unit / sub-unit. According to the embodiments of the present disclosure, at least one of the modules 810 and 820 can be implemented at least in part as a hardware circuit, for example, a field-programmable gate array (FPGA), a programmable logic array (PLA), a system on chip, a system on board, a system on package, an application-specific integrated circuit (ASIC), or any other reasonable manner of hardware or firmware through integration or packaging of circuits, or in any one of software, hardware, and firmware, or in a proper combination of any of them. Alternatively, at least one of the modules 810 and 820 can be implemented at least in part as computer program modules, which can perform corresponding functions when executed.

[0221] It should be noted that the resource recommendation device part in the embodiments of the present disclosure corresponds to the resource recommendation method part in the embodiments of the present disclosure, and the description of the resource recommendation device part is specifically referred to the resource recommendation method part, which will not be repeated here.

[0222] Figure 9 A block diagram of an electronic device suitable for implementing the above-described method according to the embodiments of the present disclosure is schematically shown. Figure 9The electronic device shown is merely an example and should not impose any limitation on the functions and usage range of the embodiments of the present disclosure.

[0223] As Figure 9 shown, the electronic device 900 according to the embodiments of the present disclosure includes a processor 901, which can perform various appropriate actions and processes according to programs stored in a read-only memory (ROM) 902 or loaded from a storage section 908 into a random access memory (RAM) 903. The processor 901 can include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor, and / or a related chipset, and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), and so on. The processor 901 can also include an on-board memory for cache use. The processor 901 can include a single processing unit or a plurality of processing units for performing different actions of the method processes according to the embodiments of the present disclosure.

[0224] In the RAM 903, various programs and data required for the operation of the electronic device 900 are stored. The processor 901, the ROM 902, and the RAM 903 are connected to each other via a bus 904. The processor 901 performs various operations of the method processes according to the embodiments of the present disclosure by executing the programs in the ROM 902 and / or the RAM 903. Note that the programs can also be stored in one or more memories other than the ROM 902 and the RAM 903. The processor 901 can also perform various operations of the method processes according to the embodiments of the present disclosure by executing the programs stored in the one or more memories.

[0225] According to the embodiments of the present disclosure, the electronic device 900 can also include an input / output (I / O) interface 905, which is also connected to the bus 904. The electronic device 900 can also include one or more of the following components connected to the input / output (I / O) interface 905: an input section 906 including a keyboard, a mouse, etc.; an output section 907 including a display such as a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 908 including a hard disk, etc.; and a communication section 909 including a network interface card such as a LAN card, a modem, etc. The communication section 909 performs communication processing via a network such as the Internet. A drive 910 is also connected to the input / output (I / O) interface 905 as necessary. A removable medium 911 such as a magnetic disk, a magneto-optical disk, a semiconductor memory, etc. is attached to the drive 910 as necessary, so that a computer program read out therefrom is installed into the storage section 908 as necessary.

[0226] According to an embodiment of the present disclosure, the method flow according to the embodiments of the present disclosure can be implemented as a computer software program. For example, the embodiments of the present disclosure include a computer program product comprising a computer program carrying out the program codes for executing the method shown in the flow chart. In such an embodiment, the computer program can be downloaded and installed from a network through the communication part 909, and / or installed from the detachable medium 911. When the computer program is executed by the processor 901, the above-mentioned functions defined in the system / apparatus according to the embodiments of the present disclosure are executed. According to an embodiment of the present disclosure, the system, device, apparatus, module, unit, etc. described above can be implemented by computer program modules.

[0227] The present disclosure also provides a computer readable storage medium, which can be included in the device / apparatus / system described in the above embodiments, or exist separately without being assembled into the device / apparatus / system. The above computer readable storage medium carries one or more programs, which when executed, implement the method according to the embodiments of the present disclosure.

[0228] According to an embodiment of the present disclosure, the computer readable storage medium can be a non-volatile computer readable storage medium. For example, it can include but not limited to portable computer diskette, hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disc read-only memory (CD-ROM), optical storage device, magnetic storage device, or any appropriate combination thereof. In the present disclosure, the computer readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus or device.

[0229] For example, according to an embodiment of the present disclosure, the computer readable storage medium can include the ROM 902 and / or the RAM 903 described above and / or one or more memories other than the ROM 902 and the RAM 903.

[0230] The embodiments of the present disclosure also include a computer program product comprising a computer program containing program codes for executing the method provided by the embodiments of the present disclosure, which are used to make the electronic device implement the above-mentioned method provided by the embodiments of the present disclosure when the computer program product is running on the electronic device.

[0231] When the computer program is executed by the processor 901, the above-mentioned functions defined in the system / apparatus according to the embodiments of the present disclosure are executed. According to an embodiment of the present disclosure, the system, device, apparatus, module, unit, etc. described above can be implemented by computer program modules.

[0232] In one embodiment, the computer program can be tangibly embodied in a non-transitory computer readable medium, such as the optical storage device, the magnetic storage device, etc. In another embodiment, the computer program can be tangibly embodied in a signal, such as a download singal, and be distributed over the networks, such as the Internet, and be downloaded into the computer system 900 via the communication interface 909, and / or the removable storage device 911. The computer program embodied in the computer program can be transmitted in any suitable form including, but not limited to, radio frequency, electrical, magnetic, optical, electromagnetic, infrared or other suitable transmission mediums. The computer program embodied in the computer program may, for example, be transmitted over communication network 905, including the Internet, Digital Subscriber Line (DSL), Cable Modem over Hybrid Fibre-Coaxial (HFC) networks, and / or Institute of Electrical and Electronics Engineers (IEEE) network.

[0233] According to an embodiment of the present disclosure, program code of a computer program for performing the methods of embodiments of the present disclosure can be written in any combination of one or more programming languages, including a high-level procedural or object-oriented programming language, and / or an assembly or machine language. Programming languages include, but are not limited to, Java, C++, python, "C", or the like. Program code can execute entirely on the user's computing device, partly on the user's device, as a stand-alone software package, partly on a remote computing device, or entirely on the remote computing device or server. In the latter scenario, the remote computing device can be connected to the user's computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computing device, such as through the Internet using an Internet Service Provider (ISP).

[0234] The flow diagrams and the block diagrams in the drawings are illustrations of architectures, functionalities, and operations of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flow diagrams or block diagrams can represent a module, a procedure, or a portion of code, which comprises one or more executable instructions for implementing the specific logical functions specified for the block. It should also be noted that in some alternative implementations, the functions noted in the blocks can occur out of the order noted in the block diagrams. For example, two blocks shown in succession may, in fact, be executed substantially concurrently or in the reverse order, depending on the functionality involved. It is also noted that each block in the block diagrams or flow diagrams, and combinations of blocks in the block diagrams or flow diagrams, can be implemented by dedicated hardware-based systems that perform the specified functions or operations, or can be implemented by a combination of dedicated hardware and computer instructions. Those skilled in the art will recognize that the features recited in the various embodiments of the present disclosure can be combined and / or integrated in various combinations, even if such combinations are not expressly disclosed in the present disclosure. In particular, the features recited in the various embodiments of the present disclosure can be combined and / or integrated in any combination, without departing from the scope of the present disclosure.

[0235] The above described embodiments of the present disclosure. However, these embodiments are merely for illustrative purposes, and are not intended to limit the scope of the present disclosure. Although each embodiment is described above separately, this does not mean that the measures in each embodiment cannot be used advantageously in combination. Without departing from the scope of the present disclosure, those skilled in the art can make various substitutions and modifications, which should all fall within the scope of the present disclosure.

Claims

1. A method for training a resource recommendation model, comprising: inputting a training sample and candidate resource information into a resource recommendation model for resource recommendation processing to generate a plurality of resource recommendation sequences for the training sample, wherein the training sample comprises query information, attribute features and a historical behavior sequence of a sample object, and each of the resource recommendation sequences comprises at least one recommended resource corresponding to a respective ranking position; comparing sequence scores of the plurality of resource recommendation sequences to obtain a respective relative advantage value of each of the resource recommendation sequences; for each of the plurality of resource recommendation sequences, determining a resource advantage value of the recommended resource in the resource recommendation sequence according to the relative advantage value of the resource recommendation sequence and a respective position weight of the at least one ranking position; and training the resource recommendation model in a reinforcement learning manner according to the resource advantage values of the recommended resources to obtain a trained resource recommendation model.

2. The method of claim 1, further comprising: determining a position weight adjustment strategy for the sample object according to the historical behavior sequence, and adjusting the respective position weight of the at least one ranking position by using the position weight adjustment strategy to obtain an adjusted position weight of each of the at least one ranking position; the determining of the resource advantage value of the recommended resource in the resource recommendation sequence according to the relative advantage value of the resource recommendation sequence and the respective position weight of the at least one ranking position comprises: determining the resource advantage value of the recommended resource according to the relative advantage value of the resource recommendation sequence and the adjusted position weight of each of the at least one ranking position.

3. The method of claim 2, wherein, the determining of the position weight adjustment strategy for the sample object according to the historical behavior sequence comprises: determining position preference information of the sample object according to the historical behavior sequence, wherein the position preference information comprises a respective attention degree of the sample object to the at least one ranking position; determining the position weight adjustment strategy according to the position preference information.

4. The method of claim 3, wherein, the determining of the position weight adjustment strategy according to the position preference information comprises: determining at least one to-be-adjusted ranking position in the at least one ranking position according to the respective attention degree of the at least one ranking position; determining the position weight adjustment strategy according to the respective attention degree of the at least one to-be-adjusted ranking position.

5. The method of claim 4, wherein, the determining of the at least one to-be-adjusted ranking position in the at least one ranking position according to the respective attention degree of the at least one ranking position comprises: for each of the at least one ranking position, in response to that the attention degree of the ranking position is within a preset attention degree range, determining the ranking position as the to-be-adjusted ranking position.

6. The method of claim 5, wherein, the preset attention degree range comprises a first attention degree sub-range and a second attention degree sub-range. the determining of the position weight adjustment strategy according to the respective attention degree of the at least one to-be-adjusted ranking position comprises: for each of the at least one to-be-adjusted ranking position, in response to that the attention degree of the to-be-adjusted ranking position is within the first attention degree sub-range, generating a sub-strategy of increasing the position weight. in response to the attention degree of the to-be-adjusted ranking position being located in the second attention degree sub-range, generating a sub-strategy of reducing the position weight; determining the position weight adjustment strategy according to the respective sub-strategy of each of the at least one to-be-adjusted ranking position.

7. The method of claim 1, further comprising: for each of the plurality of resource recommendation sequences, performing mask processing on a target recommended resource in the resource recommendation sequence according to the historical behavior sequence to obtain a masked resource recommendation sequence; the determining of the resource advantage value of the recommended resource in the resource recommendation sequence comprises: determining the resource advantage value of the recommended resource in the masked resource recommendation sequence.

8. The method of claim 7, wherein, the mask processing on the target recommended resource in the resource recommendation sequence according to the historical behavior sequence comprises: determining a target ranking position in at least one ranking position of the resource recommendation sequence according to the feedback behavior of the sample object determined by the historical behavior sequence; determining the target recommended resource in the resource recommendation sequence according to the target ranking position and performing mask processing on the target recommended resource.

9. The method of claim 8, wherein, The feedback behavior includes at least one of the following: click behavior, transaction behavior, and collection behavior.

10. The method of claim 1, further comprising: for each of the plurality of resource recommendation sequences, inputting the attribute feature, historical behavior sequence and resource recommendation sequence into a trained reward model to obtain a multi-dimensional resource reward of each of at least one recommended resource in the resource recommendation sequence; fusing the multi-dimensional resource reward of each of the at least one recommended resource to obtain the sequence score.

11. A resource recommendation method, comprising: in response to a resource recommendation request, obtaining query information, target attribute information of a target object, a target historical behavior sequence and candidate resource information contained in the resource recommendation request; inputting the query information, target attribute information, target historical behavior sequence and candidate resource information into a resource recommendation model for resource recommendation processing to generate a target resource recommendation result for the target object, wherein the resource recommendation model is obtained based on the training method of any one of claims 1-9.

12. A training device of a resource recommendation model, comprising: a sequence generation module configured to input a training sample and candidate resource information into a resource recommendation model for resource recommendation processing to generate a plurality of resource recommendation sequences for the training sample, wherein the training sample comprises attribute features and historical behavior sequences of a sample object, and the resource recommendation sequence comprises at least one recommended resource corresponding to each ranking position; a relative advantage value determination module configured to compare sequence scores of the plurality of resource recommendation sequences to obtain relative advantage values of the plurality of resource recommendation sequences. a resource advantage value determination module configured to determine, for each of the plurality of resource recommendation sequences, a resource advantage value of a recommended resource in the resource recommendation sequence according to the relative advantage value of the resource recommendation sequence and the position weight of the respective position of the at least one ranking position; and a training module configured to train the resource recommendation model in a reinforcement learning manner according to the resource advantage values of the recommended resources to obtain a trained resource recommendation model.

13. A resource recommendation apparatus, comprising: an acquisition module configured to, in response to a resource recommendation request, acquire query information, target attribute information of a target object, a target historical behavior sequence, and candidate resource information included in the resource recommendation request; a generation module configured to input the query information, the target attribute information, the target historical behavior sequence, and the candidate resource information into a resource recommendation model for resource recommendation processing to generate a target resource recommendation result for the target object, wherein the resource recommendation model is obtained based on the training apparatus of claim 12.

14. An electronic device, comprising: one or more processors; a memory configured to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors are caused to implement the method of any one of claims 1 to 11.

15. A computer-readable storage medium having stored thereon executable instructions that, when executed by a processor, cause the processor to implement the method of any one of claims 1 to 11.

16. A computer program product comprising a computer program that, when executed by a processor, implements the method of any one of claims 1 to 11.

Images