Object recommendation method, training method of object recommendation model and electronic equipment

Abstract

The invention provides an object recommendation method, a training method of an object recommendation model and electronic equipment, and relates to the field of computer technology and artificial intelligence. The method comprises the following steps: acquiring historical interaction information of a user; the historical interaction information is input into an object recommendation model, an object recommendation list is generated by means of the object recommendation model, the object recommendation list comprises at least one target object generated by means of the object recommendation model, and the object recommendation list is obtained by conducting sorting according to the matching degree corresponding to the at least one target object; the matching degree is used for representing the matching degree between the corresponding target object and the user, and the object recommendation model is obtained by utilizing reinforcement learning training; and pushing at least one target object to the user based on the object recommendation list. According to the method and the device, the technical problem of relatively low accuracy and personalization of content recommendation for the user in related technologies is solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer technology and artificial intelligence, in particular to an object recommendation method, a training method of an object recommendation model, and an electronic device. BACKGROUND

[0002] A recommendation system has gradually become an important part of online service platforms such as e-commerce and social media, aiming to provide personalized content recommendations to users and improve user satisfaction and experience. However, the recommendation method of the related art has technical problems of insufficient accuracy and shallow understanding of user preferences when dealing with large-scale and high-dimensional user and item data. Model pre-training technology is widely used in recommendation systems. However, due to the low proportion of recommendation data in the model pre-training stage, it is often difficult to accurately capture user preferences and effectively recommend content when directly applying the pre-trained model to the recommendation scenario, resulting in low accuracy and personalization of content recommendation for users in the related art.

[0003] At present, there is no effective solution to the above problems. SUMMARY

[0004] The embodiments of the present application provide an object recommendation method, a training method of an object recommendation model, and an electronic device to at least solve the technical problem of low accuracy and personalization of content recommendation for users in the related art.

[0005] According to an aspect of an embodiment of the present application, an object recommendation method is provided, which includes: obtaining historical interaction information of a user, wherein the historical interaction information is used to represent object information corresponding to a plurality of interaction objects that the user has interacted with, and information generated in the process of the user interacting with the plurality of interaction objects; inputting the historical interaction information into an object recommendation model, and generating an object recommendation list using the object recommendation model, wherein the object recommendation list contains at least one target object generated using the object recommendation model, the object recommendation list is sorted according to matching degrees of the at least one target object respectively, the matching degree is used to represent the matching degree between the corresponding target object and the user, and the object recommendation model is trained using reinforcement learning; and pushing the at least one target object to the user based on the object recommendation list.

[0006] According to another aspect of the embodiments of the present application, a method for training an object recommendation model is also provided. The method comprises: obtaining historical training interaction information of a training user, wherein the historical training interaction information is used to represent object information corresponding to a plurality of interaction objects interacted by the training user, and information generated in the process of the training user interacting with the plurality of interaction objects; inputting the historical training interaction information into an initial recommendation model, and generating a training object recommendation list by using the initial recommendation model, wherein the training object recommendation list contains at least one target training object generated by using the object recommendation model, and the training object recommendation list is sorted according to training matching degrees of the at least one target training object, wherein the training matching degrees are used to represent matching degrees between the corresponding target training objects and the training user; performing reinforcement learning on the initial recommendation model based on the at least one interaction object and the training object recommendation list, and obtaining the object recommendation model, wherein the object recommendation model is used to execute the method described above.

[0007] According to another aspect of the embodiments of the present application, an object recommendation device is also provided. The device comprises: an obtaining module configured to obtain historical interaction information of a user, wherein the historical interaction information is used to represent object information corresponding to a plurality of interaction objects interacted by the user, and information generated in the process of the user interacting with the plurality of interaction objects; a generating module configured to input the historical interaction information into an object recommendation model, and generate an object recommendation list by using the object recommendation model, wherein the object recommendation list contains at least one target object generated by using the object recommendation model, and the object recommendation list is sorted according to matching degrees of the at least one target object, wherein the matching degrees are used to represent matching degrees between the corresponding target objects and the user, and the object recommendation model is obtained by using reinforcement learning; and a pushing module configured to push the at least one target object to the user based on the object recommendation list.

[0008] According to another aspect of the embodiments of the present application, a device for training an object recommendation model is also provided. The device comprises: an obtaining module configured to obtain historical training interaction information of a training user, wherein the historical training interaction information is used to represent object information corresponding to a plurality of interaction objects interacted by the training user, and information generated in the process of the training user interacting with the plurality of interaction objects; a generating module configured to input the historical training interaction information into an initial recommendation model, and generate a training object recommendation list by using the initial recommendation model, wherein the training object recommendation list contains at least one target training object generated by using the object recommendation model, and the training object recommendation list is sorted according to training matching degrees of the at least one target training object respectively, wherein the training matching degree is used to represent a matching degree between the corresponding target training object and the training user; and a reinforcement learning module configured to perform reinforcement learning on the initial recommendation model based on the at least one interaction object and the training object recommendation list, to obtain the object recommendation model, wherein the object recommendation model is used to execute the method described above.

[0009] According to another aspect of the embodiments of the present application, an electronic device is also provided. The electronic device comprises: a memory storing an executable program; and a processor connected with the memory through a bus, configured to run the program, wherein the program is used to execute the method in each embodiment of the present application when running.

[0010] According to another aspect of the embodiments of the present application, a computer readable storage medium is also provided. The computer readable storage medium comprises a stored executable program, wherein the computer readable storage medium is used to execute the method in each embodiment of the present application when the executable program runs.

[0011] According to another aspect of the embodiments of the present application, a computer program product is also provided. The computer program product comprises a computer program, and the computer program is used to implement the method in each embodiment of the present application when executed by a processor.

[0012] According to another aspect of the embodiments of the present application, a computer program product is also provided. The computer program product comprises a non-volatile computer readable storage medium, and the non-volatile computer readable storage medium stores a computer program. The computer program is used to implement the method in each embodiment of the present application when executed by a processor.

[0013] According to another aspect of the embodiments of the present application, a computer program is also provided. The computer program is used to implement the method in each embodiment of the present application when executed by a processor.

[0014] In the embodiment of the present application, first, the historical interaction information of the user is acquired, the historical interaction information is used to represent the object information corresponding to the plurality of interaction objects that the user has interacted with, and the information generated in the interaction process between the user and the plurality of interaction objects; then, the historical interaction information is input into the object recommendation model, and the object recommendation list is generated by using the object recommendation model, the object recommendation list contains at least one target object generated by using the object recommendation model, the object recommendation list is obtained by sorting the matching degrees corresponding to the at least one target object, the matching degree is used to represent the matching degree between the corresponding target object and the user, and the object recommendation model is obtained by using reinforcement learning; finally, at least one target object is pushed to the user based on the object recommendation list. The object recommendation list is generated by acquiring the historical interaction information of the user and using the object recommendation model trained by reinforcement learning. By using the historical interaction information of the user, it can be ensured that the object recommendation model can learn based on the preferences and behavior patterns of the user, and the data basis for personalized recommendation of the object recommendation model is provided. The object recommendation model trained by reinforcement learning can understand the user preferences. The introduction of reinforcement learning enables the object recommendation model to continuously learn and improve in real interaction with the user. The object recommendation model also learns how to sort the recommended objects according to the user preferences through the reinforcement learning process, generates the object recommendation list, and determines the target objects pushed to the user based on the object recommendation list. The accuracy and personalization of content recommendation for the user are improved, thereby solving the technical problem of low accuracy and personalization of content recommendation for the user in the related art.

[0015] The above general description and the following detailed description are used to illustrate and explain the present application, and do not constitute a limitation on the present application. BRIEF DESCRIPTION OF DRAWINGS

[0016] The drawings described herein are used to provide further understanding of the present application, and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application, and do not constitute an improper limitation on the present application. In the drawings:

[0017] Figure 1 is a schematic diagram of an application scenario of an object recommendation method according to an embodiment of the present application;

[0018] Figure 2 is a flowchart of an object recommendation method according to an embodiment of the present application;

[0019] Figure 3 is a flowchart of a training method of an object recommendation model according to an embodiment of the present application;

[0020] Figure 4 is a schematic diagram of an optional training process of an object recommendation model according to an embodiment of the present application;

[0021] Figure 5 FIG. 1 is a schematic diagram of an object recommendation device according to an embodiment of the present application;

[0022] Figure 6 FIG. 2 is a schematic diagram of a training device of an object recommendation model according to an embodiment of the present application;

[0023] Figure 7 FIG. 3 is a structural block diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0024] In order to enable persons skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the following described embodiments are only part of the embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by persons skilled in the art without creative labor should be within the scope of protection of the present application.

[0025] The terms "first", "second", and the like in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily indicate a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in other orders. Among them, the other order refers to an order other than those illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device that includes a series of steps or units, and is not necessarily limited to those steps or units clearly listed, but can include other steps or units that are not clearly listed or inherent to the process, method, product or device.

[0026] First, some of the nouns or terms that appear in the description of the embodiments of the present application are applicable to the following explanations:

[0027] Group Relative Policy Optimization (GRPO): It can be an improved strategy in the field of reinforcement learning, which can be applied to tasks with multiple decision sequences. It is more effective in handling continuous action space or tasks that require the generation of multiple sequence responses, and can establish relative advantages between multiple decisions within a sequence, thereby more finely adjusting the strategy.

[0028] According to an embodiment of this application, an object recommendation method is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0029] The object recommendation method provided in this application embodiment can be applied to, for example, Figure 1 The application scenarios shown are not limited to these. In, for example... Figure 1 In the application scenario shown, server 10 can connect to one or more client devices 20 via a local area network (LAN), wide area network (WAN), internet connection, or other types of data network. These client devices 20 may include, but are not limited to, smartphones, tablets, laptops, PDAs, personal computers, smart home devices, and in-vehicle devices. Client devices 20 can interact with users through a graphical user interface to implement the methods provided in this application embodiment.

[0030] In this embodiment, the system consisting of a client device and a server can perform the following steps: The client device can interact with the server. The server can obtain the user's historical interaction information; input the historical interaction information into the object recommendation model, use the object recommendation model to filter multiple objects to be recommended, and generate an object recommendation list; push at least one target object to the user based on the object recommendation list.

[0031] It should be noted that with the rapid development of high-performance computing units, the methods provided in this application embodiment can also be applied to model-in-machine systems in other application scenarios. In one optional embodiment, the model-in-machine system has multiple built-in models. Users can select one model to adjust as needed to obtain their own model. The high-performance computing unit built into the model-in-machine system can then directly call the adjusted model to execute the methods provided in this application embodiment. In another optional embodiment, the model-in-machine system has a pre-trained model built-in. Therefore, the high-performance computing unit built into the model-in-machine system can directly call this model to execute the methods provided in this application embodiment.

[0032] Furthermore, when users need to train their own models, they can upload their own datasets via the client. These datasets are then sent to the server, allowing the server to adjust the pre-trained model using the dataset to obtain the user's customized model, which can then be deployed to the production environment. To facilitate users' model adjustment needs, the server provides complete adjustment tools, development frameworks, and processes, supporting multiple adjustment strategies. This allows the adjusted model to better adapt to different application domains and achieve a high degree of customization.

[0033] Under the aforementioned operating environment, this application provides the following: Figure 2 The recommended method for the object shown. Figure 2 This is a flowchart of an object recommendation method according to an embodiment of this application. For example... Figure 2 As shown, the method may include the following steps:

[0034] Step S202: Obtain the user's historical interaction information.

[0035] Among them, historical interaction information is used to represent the object information corresponding to multiple interaction objects that the user has interacted with, as well as the information generated during the user's interaction with multiple interaction objects.

[0036] The aforementioned historical interaction information refers to information generated when users interacted with the recommendation platform or service in the past. Historical interaction information helps understand user preferences, interests, and behavioral patterns. This information may include, but is not limited to, user behavior data such as clicks, browsing, searches, favorites, and purchases, as well as more detailed quantitative information such as timestamps, frequency, and duration of these behaviors. In recommendation algorithms, historical interaction information can be used to customize recommended content to better match users' individual preferences.

[0037] The aforementioned object information can refer to descriptive information about entities that users have historically interacted with, such as products, articles, and videos. This information can include the object's title, description, category, price, sales volume, and user ratings. Object information can be textualized so that object recommendation models can process it. For example, a product's title and description can form the product's textual information, which is then input into the object recommendation model for feature extraction and matching.

[0038] In one optional embodiment, historical user interaction information can be obtained. Specifically, data can be collected from various user behavior logs, including front-end applications, databases, log services, and real-time data streams. For e-commerce, social media, and video platforms, this can include user interaction records such as product clicks, reviews, search keywords, and video viewing duration. The collected data can also be cleaned and formatted to remove invalid or duplicate information, ensuring data quality. Preprocessing can include converting unstructured data such as text and images into structured data for easier processing by the object recommendation model. The processed data can also be integrated to construct user profiles and item feature sets. User profiles can include basic user attributes such as age, gender, and region, and behavioral characteristics such as purchasing preferences and browsing habits. Item features can cover item categories, attributes, and descriptive text. Historical interaction information can be organized into a data format suitable for input into the object recommendation model, such as sequence data, for feature extraction and subsequent personalized recommendations.

[0039] In the process described above, by analyzing users' historical interaction information, the recommendation system can more accurately understand user preferences, thereby providing more personalized recommendations, increasing user satisfaction, and improving click-through rates and conversion rates. Utilizing historical interaction information helps predict users' potential future behavior; for example, based on a user's past frequent purchases of a certain type of product, it's possible to predict that the user will be more interested in that type of product in the future, thus making more accurate recommendations.

[0040] Step S204: Input the historical interaction information into the object recommendation model and use the object recommendation model to generate an object recommendation list.

[0041] The object recommendation list contains at least one target object generated using the object recommendation model. The object recommendation list is obtained by sorting the objects according to the matching degree corresponding to each of the at least one target object. The matching degree is used to characterize the degree of matching between the corresponding target object and the user. The object recommendation model is trained using reinforcement learning.

[0042] The aforementioned object recommendation model refers to a model trained using reinforcement learning for item matching and ranking in recommendation scenarios. This model receives historical interaction information as input, predicts the user's interest in items, and generates a list of recommended items. Reinforcement learning training ensures that the model learns user preferences and ranks items based on those preferences when outputting them, placing items that better match the user's preferences at the top.

[0043] The aforementioned object recommendation list refers to a list of items generated based on user preferences during the output phase of the recommendation algorithm. Items in the object recommendation list are selected from available items, deemed more likely to attract user attention or meet user needs. The items in the object recommendation list can be sorted based on their match with the user; the higher the match, the higher the item's position in the list.

[0044] In one alternative embodiment, an object recommendation model can be used to generate an object recommendation list. Specifically, historical interaction information can be passed as input to an object recommendation model that has been trained through reinforcement learning. Internally, the object recommendation model can calculate the user's matching degree with each object based on the object information using a neural network structure. Then, the object recommendation model can sort the objects according to these matching degrees to generate an object recommendation list.

[0045] In the above process, the object recommendation model trained using reinforcement learning can effectively learn user preferences and generate a highly personalized object recommendation list based on those preferences, thus improving the accuracy of the recommendations. The presentation order of objects in the object recommendation list can affect user experience and satisfaction. The matching degree-based ranking mechanism ensures that more attractive items in the recommendation list are shown to the user first, enhancing the efficiency and effectiveness of the recommendation system. The reinforcement learning training process continuously adjusts the output of the object recommendation model through a reward mechanism, enabling the model to learn from positive feedback as well as from user disinterest or negative feedback. This avoids excessive concentration of recommendations for the same type of items and enhances the diversity and breadth of recommendations.

[0046] Step S206: Push at least one target object to the user based on the object recommendation list.

[0047] The aforementioned target objects can refer to items that are determined to be shown or recommended to users in a recommendation scenario. Target objects can be identified from a list of recommended items, have a high degree of relevance to the user, and are more likely to attract the user and meet their needs or interests.

[0048] In one alternative embodiment, at least one target object can be determined from an object recommendation list. Specifically, this can be based on a generated object recommendation list that has been sorted according to the matching degree between the user and each object. Determining the target object from the object recommendation list can be achieved in several ways. The top N items in the object recommendation list can be selected as the target objects. The value of N can be adjusted according to actual needs and user experience. For example, if the goal is to improve user satisfaction, N can be set smaller to display a smaller subset of items that the user is more interested in; if the goal is to increase exposure, N can be set larger. A matching degree threshold can also be set. When the matching degree of an object is higher than this threshold, the object is considered a target object. This method ensures that the recommended target objects are those that the user has a high probability of being interested in, avoiding low-quality recommendations that might disturb the user. Furthermore, the recommendation strategy can be dynamically adjusted by combining the user's historical behavior and real-time feedback to more accurately determine the target objects. For example, if a user has frequently purchased a certain category of goods in the past, goods of this category can be prioritized in the object recommendation list.

[0049] Next, after identifying the target audience, customized recommendations can be pushed to users based on factors such as device type, screen size, and usage habits. For example, mobile users might see a simplified recommendation list, while web users might receive more detailed item descriptions. The push process can take into account the user's current activity context and time to improve the relevance and timeliness of the recommendations. For instance, relevant recommendations can be pushed instantly when a user browses a product page, or content on specific topics can be pushed at specific times of the day. User behavior after the push, such as whether they click or make a purchase, can be recorded and used for subsequent training and improvement of the object recommendation model, forming a closed loop of continuous learning and iteration to gradually improve the accuracy of the recommendations.

[0050] In the above process, by identifying highly matching items from the recommended list as target objects, user needs can be met more accurately, reducing user frustration when browsing recommendations and improving user satisfaction. Personalized strategies based on specific user interests and behaviors make the recommendation system more intelligent and personalized, providing a better recommendation experience for different users. Precisely pushing target objects can significantly increase user engagement with recommended content. Since recommendations are based on user preferences, it reduces the time cost for users to discover content of interest and improves the user experience.

[0051] In this embodiment, firstly, the user's historical interaction information is obtained. This historical interaction information is used to represent the object information corresponding to multiple interactive objects that the user has interacted with, as well as the information generated during the user's interaction with these multiple interactive objects. Next, the historical interaction information is input into an object recommendation model, which generates an object recommendation list. The object recommendation list contains at least one target object generated by the object recommendation model. The object recommendation list is sorted according to the matching degree corresponding to each of the at least one target object. The matching degree is used to represent the degree of matching between the corresponding target object and the user. The object recommendation model is trained using reinforcement learning. Finally, at least one target object is pushed to the user based on the object recommendation list. This application generates an object recommendation list by acquiring users' historical interaction information and using an object recommendation model trained with reinforcement learning. Utilizing users' historical interaction information ensures that the object recommendation model can learn based on users' preferences and behavioral patterns, providing a data foundation for personalized recommendations. The object recommendation model trained with reinforcement learning can understand user preferences. The introduction of reinforcement learning allows the object recommendation model to continuously learn and improve through real-world interactions with users. Furthermore, through the reinforcement learning process, it learns how to sort recommended objects according to user preferences, generate an object recommendation list, and determine the target objects to push to users based on this list. This improves the accuracy and personalization of content recommendations, thereby solving the technical problem of low accuracy and personalization in content recommendations in related technologies.

[0052] In the above embodiments of this application, inputting historical interaction information into an object recommendation model and generating an object recommendation list using the object recommendation model includes: inputting historical interaction information into an object recommendation model; generating at least one target object using the object recommendation model, wherein the matching degree between the target object and the user is greater than a preset degree; matching the at least one target object with the user using the object recommendation model to determine the matching degree corresponding to each of the at least one target object; and sorting the at least one target object based on the matching degree corresponding to each of the at least one target object using the object recommendation model to generate an object recommendation list.

[0053] The matching degree mentioned above can be considered a quantitative representation of a user's interest in a specific object, calculated by the object recommendation model based on the input's historical interaction information. A higher matching degree indicates a greater degree of match between the user and the object, and a greater user interest or need for that object.

[0054] The aforementioned preset level refers to a pre-defined threshold used to judge the matching degree calculated by the object recommendation model. The preset level can be determined based on application goals, user behavior analysis, or system performance requirements, which helps to find a balance between recommendation diversity and accuracy.

[0055] In one optional embodiment, the object recommendation model can extract features from historical interaction information, converting the object information in the historical interaction information into a high-dimensional feature vector. Then, based on the obtained feature vector, at least one target object can be generated. Next, the at least one target object can be matched with a user, quantifying the user's interest in the target object to obtain the matching degree corresponding to each of the at least one target object. Then, based on their respective matching degrees, the target objects can be sorted; the higher the matching degree of a target object, the higher its ranking in the object recommendation list, thereby generating an object recommendation list arranged in descending order of matching degree.

[0056] In the above process, by accurately calculating the matching degree between users and each object, it is ensured that the recommended objects are highly personalized and can meet the specific needs and interests of users. Filtering based on preset levels and sorting based on matching scores ensures that the recommendation list includes objects that users are more interested in, while maintaining a certain degree of diversity. This avoids over-recommending a single type of object and provides users with a wider range of choices.

[0057] In the above embodiments of this application, the method further includes: outputting an object recommendation list; responding to receiving a modified recommendation list from user feedback, evaluating the modified recommendation list and the object recommendation list based on a preset reward rule to obtain a total reward value, wherein the modified recommendation list is obtained by modifying the sorting order of at least one target object and / or at least one target object; and updating the model parameters of the object recommendation model based on the total reward value.

[0058] The aforementioned modified recommendation list refers to a list that a user manually modifies based on personal preferences after receiving the initial recommendation list. User adjustments may include removing uninteresting items, adding new suggestions, or reordering items in the recommendation list to better suit the user's current needs or interests.

[0059] The aforementioned pre-defined reward rules can refer to standard reward rules designed to evaluate the quality of recommendation results. These rules can be based on direct feedback from user behavior, such as clicks or purchases, or indirect metrics measuring how well the recommendation results match user expectations. Pre-defined reward rules provide a metric for the recommendation system, enabling the object recommendation model to learn and adjust itself based on user feedback.

[0060] The aforementioned total reward value refers to a comprehensive score derived from evaluating the recommendation results and the revised recommendation list based on user feedback. It reflects the overall performance of the recommendation system. The total reward value combines the matching degree between recommended items and user preferences, the rationality of the recommendation list's ranking, and other influencing factors such as diversity and novelty. It can serve as data for updating the parameters of the recommendation model, and the total reward value can be maximized through parameter adjustments to improve recommendation quality and user experience.

[0061] In one optional embodiment, an object recommendation list can be generated based on an object recommendation model. This list may contain multiple objects identified as potentially interesting to the user. After viewing the list, the user can modify the recommendations according to their preferences, such as adjusting the position of certain items or removing items they are not interested in. Upon receiving the modified recommendation list from the user, the recommendation system evaluates the list before and after modification according to preset reward rules. These rules consider changes in user item preferences and the rationality of the ranking, assigning corresponding reward values ​​to each recommendation. Rewards can be positive numbers indicating encouragement or negative numbers indicating punishment, reflecting the degree to which the recommendation list matches the user's preferences. After evaluating the recommendations with rewards, the parameters of the object recommendation model can be adjusted based on the calculated total reward value. The total reward value guides the learning direction of the object recommendation model, enabling it to gradually improve its recommendation strategy and more accurately predict and meet user needs.

[0062] The process described above allows users to modify their object recommendation list and feed this modified list back to the recommendation system. This makes the recommendations more closely reflect the user's actual needs, thereby improving user satisfaction. Through user modifications and reward mechanisms, the object recommendation model can learn more refined user preferences, thus improving the accuracy and personalization of recommendations. This user feedback-driven model update mechanism creates a cycle of continuous learning and improvement. User feedback provides the object recommendation model with learning opportunities, enabling the recommendation system to become more intelligent and efficient as user data accumulates.

[0063] In the above embodiments of this application, the preset reward rules include: a recommendation reward rule and a ranking reward rule; the reward evaluation of the modified recommendation list and the object recommendation list based on the preset reward rules to obtain a total reward value includes: evaluating the rewards of at least one target object in the object recommendation list and at least one modified object in the modified recommendation list based on the recommendation reward rule to obtain a recommendation reward value; evaluating the rewards of at least one negative sample object in the object recommendation list based on the ranking reward rule to obtain a ranking reward value, wherein the negative sample object is a target object in the object recommendation list that fails to match at least one modified object; and summing the recommendation reward value and the ranking reward value to obtain a total reward value.

[0064] The aforementioned recommendation reward rules refer to rules used to quantify the degree of matching between the recommendations given by the recommendation system and the user's actual interests or behaviors. In practical applications, recommendation reward rules can determine the reward value of each item or candidate item in the recommendation list based on user feedback to the recommendation results, such as clicks, purchases, and browsing time.

[0065] The aforementioned ranking reward rule refers to a rule used to evaluate whether the order of objects in the recommendation list matches user preferences. The reward value can be calculated based on the user's sorting actions in the recommendation list, such as moving objects up or down, or deleting them. For example, if a user moves an item they are not interested in to the bottom of the list, the ranking reward rule can give this action a low or negative reward value, thereby encouraging the object ranking model to adjust its ranking strategy to better reflect the user's true preference order.

[0066] The aforementioned referral reward value refers to a reward value calculated according to referral reward rules, reflecting the degree of match between a specific object in the recommendation list and the user's actual preferences. For example, if a user clicks on or purchases an object, then according to the referral reward rules, that object can receive a higher referral reward value.

[0067] The aforementioned ranking reward value can refer to a reward value evaluated based on ranking reward rules, used to reflect whether the relative ranking of negative sample objects in the recommendation list is consistent with the user's expected preferences. For example, in user feedback, if a certain type of object is frequently moved to the end of the list, it may reflect that the user tends not to want to see this type of object, and a higher ranking reward value can be assigned to the ranking that does not match the user's preferences.

[0068] In one optional embodiment, a preliminary recommendation list can be generated, containing a set of target objects predicted based on user historical interaction information. After viewing the recommendation list, users can make modifications, such as moving an item to the front of the list, deleting items they are not interested in, or adjusting the overall sorting of the list. Based on these modifications, a modified recommendation list can be formed. For the recommendation reward value, the target objects in the recommendation list before and after the modification can be evaluated according to recommendation reward rules, such as user actions on a modified object (clicks, purchases, etc.). The higher the consistency between the target object and the user's actions, the higher the recommendation reward value can be. For the sorting reward value, the changes in the relative positions of items in the list before and after the modification can be analyzed. For example, if a user moves a less relevant item to the end, it reflects a decrease in the relative importance of the item, and the sorting reward value of that item can be reduced accordingly. Items moved to the front will receive higher sorting reward values. Finally, the recommendation reward value and sorting reward value of each object can be summed to obtain the total reward value.

[0069] Introducing recommendation and ranking rewards in the above process allows for a more refined learning process for the object ranking model. It enables the system to focus on the recommendation effect of individual items as well as the relative ranking between items, thus improving recommendation quality and user experience. Through direct user feedback, the recommendation system can adjust the recommendation list in real time, making the recommended content more aligned with users' actual needs and preferences, thereby increasing user satisfaction and interaction rates.

[0070] In the above embodiments of this application, based on the recommendation reward rules, a reward evaluation is performed on at least one target object in the object recommendation list and at least one modified object in the modification recommendation list to obtain a recommendation reward value. This includes: for any target object in the object recommendation list, matching the target object with at least one modified object in the modification recommendation list to obtain a matching result corresponding to the target object; determining a first reward value corresponding to the target object based on the matching result corresponding to the target object; and summarizing the first reward values ​​corresponding to at least one target object to determine the recommendation reward value.

[0071] The aforementioned first reward value can refer to the instantaneous score for changes in the importance of each target object in the recommendation list after user feedback. It can be determined by comparing the target object's position in the original recommendation list with its position in the modified recommendation list, or by using direct user feedback on the target object, such as clicks, favorites, or purchases. It can quantify the degree of match between the target object and the user's current preferences, as well as the change in the target object's relative importance in the recommendation list.

[0072] In one optional embodiment, each target object in the object recommendation list can be matched with at least one modified object in the user-modified recommendation list to identify changes in the user's preference for the target objects. The matching process can be based on user actions, such as moving objects, adding new objects, deleting objects, etc., or on more complex user behavior analysis. Next, based on the matching results, a first reward value can be determined for each target object. If the target object's position in the modified recommendation list improves, or the user directly interacts with it positively, the first reward value can be positive; if the target object's position declines, or the user shows disinterest, the first reward value can be negative, reflecting the increase or decrease in the degree to which the target object matches the user's preferences. Finally, the first reward values ​​of each target object can be summed to obtain a comprehensive recommendation reward value.

[0073] In the above process, the calculation of the first reward value enables the recommendation system to quickly adjust its recommendation strategy based on changes in user behavior, achieving more personalized recommendations and improving user satisfaction and system efficiency. By converting the matching result between the target object and the modified object into a reward signal, the system can accurately capture users' preferences for recommendation ranking, thereby improving the ranking process and making the recommendation list more in line with user expectations.

[0074] In the above embodiments of this application, the reward evaluation of at least one negative sample object in the object recommendation list based on the ranking reward rule to obtain a ranking reward value includes: evaluating the matching degree of the negative sample object based on the ranking reward rule to obtain a second reward value corresponding to the negative sample object; and summing up the second reward values ​​corresponding to at least one negative sample object to obtain a ranking reward value.

[0075] The aforementioned second reward value can refer to the reward value obtained by evaluating negative sample objects under the ranking reward rule. Negative sample objects here can be those items in the recommended list that failed to create a valid match with the user in the modified recommended list. The calculation of the second reward value can be based on the change in the item's predicted position before the user modified the list and its relative position after the user's actual action. The second reward value, by quantifying the position of items that should not appear or should be lower in the recommended list, provides feedback to the recommendation system, helping it understand and adjust its recommendation strategy to more accurately reflect user preferences.

[0076] In one optional embodiment, after a user provides feedback to modify the recommendation list, items that have not received positive user interaction can be identified as negative sample objects. These can include items that the user did not click, purchase, or provide positive feedback on in the object recommendation list, as well as items that appeared in the object recommendation list but were removed or significantly demoted in the modified recommendation list. For each negative sample object, the difference between its actual position and its ideal position in the modified recommendation list—that is, the position the user does not want to see or desires to be lower—can be calculated. This difference is used as a basis to evaluate the matching degree. The lower the matching degree, the less the item's position in the list matches the user's expectations, and the more negative the corresponding second reward value will be, thus guiding the object recommendation model to reduce erroneous ranking in future recommendations. Finally, the second reward values ​​of the negative sample objects can be summarized to obtain the ranking reward value.

[0077] In the above process, by calculating the second reward value and summing them to obtain the ranking reward value, targeted improvements can be made to the object recommendation model to address the ranking issues raised by users. This reduces the discrepancy between the position of items in the recommendation list and the user's expectations, thereby improving the personalization of recommendations and user satisfaction. By dynamically generating negative sample objects and assigning them a second reward value, the accurate understanding and effective utilization of negative sample objects during the object recommendation model's learning process is ensured, avoiding learning bias caused by inconsistencies between the distribution of negative samples and the output distribution of the object recommendation model.

[0078] In the above embodiments of this application, updating the model parameters of the object recommendation model based on the total reward value includes: determining the advantage function value corresponding to the object recommendation model based on the total reward value; generating the objective function value corresponding to the object recommendation model based on the advantage function value and the matching degree corresponding to at least one target object; and updating the model parameters of the object recommendation model based on the objective function value.

[0079] The aforementioned advantage function value can refer to the measure of the superiority of an action's expected reward in a specific state compared to the average expected reward of possible actions in that state. When guiding policy improvement, it helps the recommendation model learn which actions to take in different states—that is, which items to recommend or which to adjust the order of the recommendation list—to achieve better results than average.

[0080] The objective function value mentioned above can refer to a function value calculated based on the current model parameters, the advantage function value, and the matching degree of each target object, used to guide the update direction of the object recommendation model parameters. The objective function can aim to minimize the loss or maximize the gain, depending on the improvement strategy adopted.

[0081] In one optional embodiment, the relative advantage of each action—that is, the selection and ranking of items in the recommendation list—can be calculated based on the total reward value. Specifically, the reward values ​​of possible actions can be averaged, and then the average value can be subtracted from the reward value of each specific action to obtain the advantage function value. The advantage function value reflects the superiority or inferiority of a particular recommendation or ranking action compared to other possible actions, and can serve as data for the object recommendation model to learn and adjust its strategy. Next, the advantage function value and the matching degree of the target object can be combined to generate a target function value. The matching degree reflects the degree of fit between the recommended items and the user's actual preferences. Calculating the target function value ensures that the object recommendation model focuses on the accuracy of recommendations and also emphasizes the personalized ranking of recommended content to maximize user satisfaction. This process can also include quantifying the difference between the model's predicted probability and the actual reward, prompting the object recommendation model to prioritize recommending items that can obtain higher rewards, while improving the ranking of items to match the user's preferred order. Finally, based on the calculated target function value, the parameters of the object recommendation model can be updated. This can be achieved through algorithmic improvements such as gradient descent, with the improvement goal being to minimize the target function value, i.e., to improve recommendation performance and user satisfaction. In each training iteration, the parameters of the object recommendation model can be adjusted based on the feedback from the objective function value, gradually approaching a better recommendation strategy.

[0082] In the above process, the calculation of the dominance function value provides the object recommendation model with clear feedback on the merits of actions, guiding the model to learn how to make better recommendation decisions under specific user preferences, thus improving the efficiency and effectiveness of the model's learning. The generation of the objective function value ensures that the object recommendation model considers both the item's matching degree and the user's preference for the order of the recommendation list during recommendations. This setting allows the recommendation system to be more personalized and better meet user needs. Through the calculation of the dominance and objective function values, the update direction and magnitude of the object recommendation model's parameters are more accurately guided, accelerating the model's convergence speed and improving the overall performance of the recommendation system.

[0083] According to an embodiment of this application, a training method for an object recommendation model is also provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in an order not used here.

[0084] Figure 3 This is a flowchart of a training method for an object recommendation model according to an embodiment of this application, such as... Figure 3 As shown, the specific steps may include the following:

[0085] Step S302: Obtain the historical training interaction information of the training user.

[0086] Among them, historical training interaction information is used to characterize the object information corresponding to multiple interaction objects that the training user has interacted with, as well as the information generated during the interaction between the training user and multiple interaction objects.

[0087] The aforementioned historical object interaction information refers to information generated when users interact with the recommendation system in the past. This can include user interactions with different objects, such as products, articles, and videos, including actions like clicking, saving, purchasing, and commenting. Historical object interaction information serves as an important data source for the recommendation system to learn user preferences and make personalized recommendations.

[0088] Step S304: Input the historical training interaction information into the initial recommendation model, and use the initial recommendation model to generate a training object recommendation list.

[0089] The training object recommendation list contains at least one target training object generated using the object recommendation model. The training object recommendation list is sorted according to the training matching degree corresponding to each of the at least one target training object. The training matching degree is used to characterize the degree of matching between the corresponding target training object and the training user.

[0090] The initial recommendation model mentioned above refers to a model that has not yet undergone specific improvements for the recommendation task at the start of training. This initial model can be pre-trained based on general domains, such as language understanding or image recognition. However, it lacks an understanding of the specific needs of the recommendation domain and adaptability to users' personalized preferences.

[0091] Step S306: Based on at least one interactive object and a training object recommendation list, perform reinforcement learning on the initial recommendation model to obtain an object recommendation model.

[0092] The object recommendation model is used to execute the methods described above.

[0093] In one optional embodiment, historical object interaction information of the training user can be collected. This information may include the user's previous interactions with various objects and related feedback, reflecting which objects the user responded positively to and which objects the user might not be interested in or had negative feedback on. This historical object interaction information can then be used as input to an initial recommendation model. Based on this information, the initial recommendation model can filter and rank multiple objects to be recommended, generating a preliminary training object recommendation list. Next, reinforcement learning techniques can be applied to fine-tune the initial recommendation model based on the interaction between the training user and the generated training object recommendation list, as well as any modifications made by the user to the list. The initial recommendation model can calculate rewards based on user feedback, such as recommendation reward values ​​and ranking reward values, and then adjust its parameters based on these reward values ​​to improve the recommendation frequency and ranking accuracy of highly matched objects in future recommendations. This reinforcement learning process is then applied to the initial recommendation model to obtain a final object recommendation model.

[0094] In the above process, by fine-tuning using historical object interaction information, the initial recommendation model can gradually learn and understand the user's personalized preferences, thereby more accurately recommending objects that attract the user's attention in the recommendation list and improving the relevance and personalization of the recommended content. The initial recommendation model is based on general domain pre-training and has strong generalization ability. Through reinforcement learning fine-tuning, the resulting object recommendation model can maintain its generalization ability while improving its understanding and adaptation to recommendation domain-specific rules, achieving effective recommendations in a wider range of scenarios.

[0095] In the above embodiments of this application, reinforcement learning is performed on an initial recommendation model based on at least one interactive object and a training object recommendation list to obtain an object recommendation model, including: evaluating the rewards of at least one interactive object and the training object recommendation list based on a preset reward rule to obtain a total training reward value; and updating the model parameters of the initial recommendation model based on the total training reward value to obtain the object recommendation model.

[0096] In one optional embodiment, at least one interaction object in the user's training history and a recommended list of training objects generated by the initial recommendation model can be evaluated according to preset reward rules. The reward rules may include recommendation reward rules and ranking reward rules, designed to quantify the degree of matching between items in the recommended list and user preferences, as well as the relative ranking of items. For each interaction object and target training object in the recommended list, corresponding reward values ​​can be calculated, reflecting the accuracy of the initial model's recommendations and rankings, and user satisfaction.

[0097] Next, the obtained reward values ​​can be summarized to calculate the total training reward value. This total training reward value serves as a comprehensive feedback on the entire recommendation list and user interaction, taking into account both the quality of the recommendation list and the level of user engagement, thus guiding the updating of the initial recommendation model parameters. Finally, based on the calculated total training reward value, the parameters of the initial recommendation model can be updated using reinforcement learning mechanisms. The direction and magnitude of the initial model parameter updates can be determined by calculating the advantage function value and the objective function value. High reward values ​​enhance the initial recommendation model's tendency to recommend and rank corresponding items, while low reward values ​​prompt the initial recommendation model to adjust its strategy, reducing errors in recommendation and ranking adjustments.

[0098] In the above process, by evaluating rewards based on users' historical interactions, the recommendation strategy can be dynamically adjusted to ensure that the recommended content broadly covers users' interests and deeply aligns with their personalized needs, thereby improving the accuracy and satisfaction of the recommendations. Introducing a ranking reward mechanism enhances the sensitivity of the initial recommendation model to the order of the recommendation list, prompting the model to learn a more reasonable item ranking logic and ensuring that the order of the recommendation list largely conforms to the user's preferred priorities.

[0099] In the above embodiments of this application, the preset reward rules include: a recommendation reward rule and a ranking reward rule; the reward evaluation of at least one interactive object and a training object recommendation list based on the preset reward rules to obtain a total training reward value includes: evaluating the rewards of at least one target training object and at least one interactive object in the training object recommendation list based on the recommendation reward rule to obtain a training recommendation reward value; evaluating the rewards of at least one negative sample training object in the training object recommendation list based on the ranking reward rule to obtain a training ranking reward value, wherein at least one negative sample training object is a target training object in the training object recommendation list that fails to match at least one interactive object; and summing the training recommendation reward value and the training ranking reward value to obtain a total training reward value.

[0100] In one alternative embodiment, information about at least one interactive object can be extracted from the historical interaction data of the training user, along with a training object recommendation list generated using an initial recommendation model based on the training user's historical behavior. The training object recommendation list can contain multiple target training objects, each with a training match score, i.e., the degree to which the object matches objects the user has historically interacted with. Next, a recommendation reward rule can be applied to evaluate the match between each target training object and historically interacted objects. If a target object is ranked high in the recommendation list and the user has a history of positive interactions with this type of object, a higher recommendation reward value can be given to this target object. If the target object does not match the user's historical preferences, the reward value can be lower. The recommendation reward values ​​of all target objects are summed to obtain the training recommendation reward value. Then, a ranking reward rule can be used to evaluate those training objects that are in the recommendation list but have not been interacted with by the user, i.e., negative sample training objects. The ranking reward value can be calculated based on the relative position of the target object in the recommendation list. If a negative sample object is ranked high, a lower ranking reward value can be obtained. The ranking reward values ​​of the negative sample training objects are summed to obtain the training ranking reward value. Finally, the training recommendation reward value and the training ranking reward value can be summarized and the total training reward value can be obtained through weighted or fused methods.

[0101] In the above process, the recommendation reward rule allows for better measurement of the personalization and relevance of the content recommended by the initial recommendation model. This incentivizes the model to recommend items that are more likely to attract user attention, thereby increasing user satisfaction and engagement with the recommended content. The introduction of the ranking reward rule enables the initial recommendation model to learn more reasonable ranking logic, avoiding placing items that users are not interested in at the top of the recommendation list and ensuring that the ranking of the recommendation list largely reflects user preferences.

[0102] In the above embodiments of this application, updating the model parameters of the initial recommendation model based on the total training reward value to obtain the object recommendation model includes: determining the training advantage function value corresponding to the initial recommendation model based on the total training reward value; generating the training objective function value corresponding to the initial recommendation model based on the training advantage function value and the training matching degree corresponding to at least one target training object; and updating the model parameters of the initial recommendation model based on the training objective function value to obtain the object recommendation model.

[0103] In one alternative embodiment, the dominance function measures the superiority of taking a particular action relative to the average action in a given state. The training dominance function value can be calculated based on the total training reward value, quantifying the contribution of each recommendation action, such as recommending a specific product, to the overall training process. During calculation, the baseline portion of the reward—the basic value the reward should receive without considering the effect of the current action—can be removed to obtain the additional value of the current action, i.e., the training dominance function value. Next, a training objective function value can be generated based on the training dominance function value and the training matching degree of each target training object in the recommendation list. The training matching degree reflects the fit between the recommended object and the user's historical preferences. The calculation of the training objective function value combines the model output probability with the dominance function value, ensuring that the initial object recommendation model is improved by increasing the output probability of those highly advantageous recommended objects, while considering the matching degree between these recommended objects and the user. This aims to guide the initial object recommendation model to learn how to improve the accuracy and satisfaction of recommendations while maintaining personalization. Finally, the parameters of the initial recommendation model can be updated based on the training objective function value. The update process can follow the basic principles of reinforcement learning, adjusting the initial object recommendation model parameters by maximizing the objective function. This allows the initial object recommendation model to be more inclined to adopt recommendation strategies that performed well during training in future learning. Specific methods for parameter updating can include gradient ascent or other gradient-based improvement algorithms to gradually adjust the weights of the initial object recommendation model and improve its recommendation and ranking capabilities.

[0104] In the above process, by introducing a training advantage function and a training objective function, a balance can be found between personalized recommendations and recommendation accuracy. The advantage function helps the initial object recommendation model identify which recommendation strategies are more attractive to users, while the objective function guides the initial object recommendation model on how to adjust parameters to maximize the effectiveness of these strategies. The calculation of the training objective function value considers recommendation accuracy and incorporates ranking information, that is, the relative position of each object in the recommendation list. The resulting object recommendation model can correctly identify products that users are interested in and has also learned to rank them according to user preferences, thus improving user satisfaction with the recommendations.

[0105] In the above embodiments of this application, the initial recommendation model is used to match multiple training objects with users to generate a training object recommendation list, including: matching any training object with a user using the initial recommendation model to determine candidate training objects from multiple training objects; and adding the candidate training object to the training object recommendation list if there is no target training object in the training object recommendation list that is the same as the candidate training object.

[0106] In one optional embodiment, an initial recommendation model can be used to match any training object. This can include inputting historical user interaction data and information about the object to be recommended into the initial recommendation model. Through the inference process of the initial recommendation model, a matching score between each training object and the user's historical preferences is calculated. Based on these matching scores, a series of candidate training objects with high matching scores can be selected. If no target training object identical to a candidate training object exists in the training object recommendation list, the candidate object can be added to the training object recommendation list and sorted according to its matching score.

[0107] In the above process, by ensuring that there are no duplicate target training objects in the training object recommendation list, the diversity of the recommendation list can be effectively improved, and the initial object recommendation model can be prevented from ignoring other objects that would attract users due to overlearning some objects.

[0108] In the above embodiments of this application, the method further includes: if there is a target training object in the training object recommendation list that is the same as the candidate training object, re-matching any training object with the initial recommendation model, determining a new candidate training object from multiple training objects, until there is no target training object in the training object recommendation list that is the same as the new candidate training object.

[0109] In one optional embodiment, during training iterations, if a newly generated candidate training object is detected to be identical to an existing object in the training object recommendation list, a fallback mechanism can be triggered. The initial recommendation model can be invoked again to evaluate another training object not currently in the recommendation list, in order to find a new candidate training object that is different from the target training object in the list. When a new candidate training object is found, it can be added to the training object recommendation list, ensuring the diversity and non-redundancy of the training object recommendation list. The fallback mechanism avoids duplicate recommendations, ensuring the richness of the training data and the breadth of learning by the initial object recommendation model.

[0110] In the above process, avoiding duplicate objects in the training object recommendation list helps to cover a wider range of user interests. During training, forcing the model to process diverse negative samples can prevent the initial object recommendation model from relying too much on the preferences of certain popular or common objects. This helps the initial object recommendation model to better grasp the overall picture of user preferences and reduce the risk of overfitting.

[0111] The technical solution proposed in this application is described below with reference to an optional embodiment. This application proposes a recommendation large model preference alignment algorithm based on reinforcement learning fine-tuning. In general, the method proposed in this application can include three stages: model response acquisition, reward function calculation, and training objective calculation. This application is concise and effective, organically combining the characteristics of reinforcement learning fine-tuning and recommendation tasks.

[0112] This application provides prompts containing user interaction information, specifically the user's historical product interaction records. It first uses a restricted beam search to allow the model to output multiple distinct legal items. Then, it assigns reward values ​​to each item based on rule-based rewards and ranking rewards. Next, it calculates the reward function value and improvement objective using a group-relative policy improvement method. Finally, it updates the parameters based on the group-relative policy improvement objective. The aforementioned user interaction information can refer to the user's historical product records represented in text form. This application uses rule-based reward settings to improve evaluation metrics, hit rate, and normalized depreciation cumulative gain. Reinforcement learning based on thought processes, such as group-relative policy improvement, provides 0 / 1 rewards based on whether a hit occurs, making it difficult to distinguish between a large number of negative samples. This application introduces ranking rewards, allowing for differentiated penalties for negative samples during reinforcement learning, thus improving learning performance. This application ensures better sampling diversity through beam search, retaining the top B local candidates with high probabilities at each step and continuously expanding these candidates in subsequent steps to approximate the higher probability sequence of the entire sentence. The first k outputs of the beam search can be used as the sampling results. Meanwhile, in order to output legitimate samples to obtain rewards based on rules, this application uses constraint decoding to limit the output solution space of the model.

[0113] Figure 4 This is a schematic diagram illustrating the training process of an optional object recommendation model according to an embodiment of this application, as shown below. Figure 4 As shown, historical training interaction information is obtained, and a bundle search is performed using an initial object recommendation model to process multiple interaction objects, resulting in training recommendation reward values ​​and training ranking reward values. Based on these training recommendation reward values ​​and training ranking reward values, group relative policy improvement calculations are performed to obtain the total training reward value. This total training reward value can then be applied to the initial object recommendation model to train and improve it.

[0114] During the model response collection phase, prompts containing user interaction history are provided. This application employs a restricted beam search strategy during the sampling phase. Beam search ensures that for the same input, the sampled model responses are all different. Beam search introduces richer negative samples, expanding the number of items the model encounters during training, which helps improve the model's ranking ability.

[0115] During the reward function calculation phase, the recommended reward function assigns a reward value of 1 if and only if the item output by the model is the target item. This approach is simple and easy to implement, while ensuring that the reward feedback is consistent with the user's actual preferences, and is less prone to misalignment between the reward mechanism and recommendation performance.

[0116] This application adds a ranking reward function to the recommendation reward function. The ranking reward function is calculated as follows:

[0117] ;

[0118] ;

[0119] The ranking reward function formally references the calculation formula of the normalized depreciation cumulative gain metric. It assigns lower reward values ​​to negative examples with higher output probabilities—that is, negative examples that the model finds harder to distinguish—to encourage the model to further reduce the output probability. Subsequent normalization ensures that the advantage function value for negative examples is always negative. The final item reward can be the sum of the rule-based reward function and the ranking reward function.

[0120] ;

[0121] The reward function designed in this way takes into account both emphasizing positive samples (i.e., rule-based rewards) and distinguishing negative samples (i.e., ranking rewards). Ranking rewards improve the depth and density of ranking information, injecting more granular ranking information into the model.

[0122] During the training objective calculation phase, after obtaining the reward value, the advantage function and objective function can be derived by referring to the group-relative policy improvement. The advantage function of the group-relative policy improvement is directly derived from the normalized reward function, eliminating the need to train a separate value network and reducing training overhead. The formula for calculating the advantage function of the group-relative policy improvement is as follows:

[0123] ;

[0124] Here, represents the advantage function corresponding to the j-th token in the k-th item output by the model, and represents the vector composed of the reward values ​​of all items output by the model for the same cue word. mean and std can represent its mean and standard deviation, respectively. After obtaining the advantage value, the group relative policy improvement is maximized as follows: (Training objective to improve the model)

[0125] ;

[0126] Overall, the group-based strategy improvement aims to increase the output probability of high-yield responses and decrease the output probability of low-yield responses.

[0127] This application explicitly introduces ranking information by sampling multiple items at once, compensating for the lack of ranking information. The reinforcement learning paradigm of sampling and training simultaneously ensures that the distribution of negative samples is consistent with the distribution of model output, solving the problem of easy discrimination of random sampling of negative samples. The introduction of rule-based rewards and implicit reward mechanisms are more aligned with the recommendation performance of the model, while the introduction of ranking rewards further increases the density of ranking signals.

[0128] This application adapts reinforcement learning fine-tuning to the recommendation model domain, solving the problem of ranking signal injection and bringing significant performance improvements. Regarding the sampling strategy, this application improves upon the original reinforcement learning fine-tuning sampling strategy by introducing restricted bundle search to ensure the legitimacy and diversity of items, expanding the types of negative samples the model encounters. In terms of the reward mechanism, this application adds a ranking reward mechanism to the original rule-based reward, increasing the ranking information density of the original rule-based reward and allowing the model to learn more refined ranking information.

[0129] According to another aspect of the present invention, an object recommendation apparatus is also provided, which can execute the object recommendation method of the above embodiments. The specific implementation method and preferred application scenarios are the same as those of the above embodiments, and will not be described in detail here.

[0130] Figure 5 This is a schematic diagram of an object recommendation device according to an embodiment of this application, such as... Figure 5 As shown, the device includes the following: an acquisition module 502, a generation module 504, and a push module 506.

[0131] The system includes: an acquisition module 502 for acquiring the user's historical interaction information, which represents the object information corresponding to multiple interactive objects that the user has interacted with, as well as the information generated during the user's interaction with multiple interactive objects; a generation module 504 for inputting the historical interaction information into an object recommendation model, using the object recommendation model to filter multiple objects to be recommended, and generating an object recommendation list, wherein at least one target object in the object recommendation list is obtained by sorting according to the matching degree corresponding to at least one target object, and the matching degree represents the degree of matching between the corresponding target object and the user; and an object recommendation model is trained using reinforcement learning; and a push module 506 for pushing at least one target object to the user based on the object recommendation list.

[0132] The generation module is further used to match historical interaction information with the object information corresponding to any object to be recommended using an object recommendation model, thereby obtaining the matching degree of the object to be recommended; using the object recommendation model, based on the matching degrees corresponding to multiple objects to be recommended, at least one target object is selected from multiple objects to be recommended, wherein the matching degree between the target object and the user is greater than a preset degree; using the object recommendation model, based on the matching degrees corresponding to at least one target object, at least one target object is sorted to generate an object recommendation list.

[0133] The acquisition module is also used to output an object recommendation list; in response to receiving a modified recommendation list from user feedback, it evaluates the modified recommendation list and the object recommendation list based on a preset reward rule to obtain a total reward value, wherein the modified recommendation list is obtained by modifying at least one target object and / or the sorting order of at least one target object; and updates the model parameters of the object recommendation model based on the total reward value.

[0134] The preset reward rules include: recommendation reward rules and ranking reward rules; the acquisition module is also used to evaluate the rewards for at least one target object in the object recommendation list and at least one modified object in the modification recommendation list based on the recommendation reward rules, and obtain the recommendation reward value; to evaluate the rewards for at least one negative sample object in the object recommendation list based on the ranking reward rules, and obtain the ranking reward value, wherein the negative sample object is the target object in the object recommendation list that fails to match at least one modified object; and to summarize the recommendation reward value and the ranking reward value to obtain the total reward value.

[0135] The acquisition module is further configured to match any target object in the object recommendation list with at least one modified object in the modification recommendation list to obtain the matching result corresponding to the target object; determine the first reward value corresponding to the target object based on the matching result corresponding to the target object; and summarize the first reward values ​​corresponding to at least one target object to determine the recommendation reward value.

[0136] The acquisition module is also used to evaluate the matching degree of negative sample objects based on the ranking reward rules, and obtain the second reward value corresponding to the negative sample object; and to summarize the second reward values ​​corresponding to at least one negative sample object to obtain the ranking reward value.

[0137] The acquisition module is further used to determine the advantage function value corresponding to the object recommendation model based on the total reward value; generate the objective function value corresponding to the object recommendation model based on the advantage function value and the matching degree corresponding to at least one target object; and update the model parameters of the object recommendation model based on the objective function value.

[0138] According to another aspect of the present invention, a training apparatus for an object recommendation model is also provided. This apparatus can execute the training method of the object recommendation model in the above embodiments. The specific implementation method and preferred application scenarios are the same as those in the above embodiments, and will not be described in detail here.

[0139] Figure 6 This is a schematic diagram of a training apparatus for an object recommendation model according to an embodiment of this application, as shown below. Figure 6 As shown, the device includes the following: an acquisition module 602, a generation module 604, and a reinforcement learning module 606.

[0140] The system includes: an acquisition module 602, used to acquire historical training interaction information of the training user, wherein the historical training interaction information represents the object information corresponding to multiple interaction objects interacted with by the training user, as well as the information generated during the interaction between the training user and multiple interaction objects; a generation module 604, used to input the historical training interaction information into the initial recommendation model, and use the initial recommendation model to filter multiple training objects to generate a training object recommendation list, wherein at least one target training object in the training object recommendation list is sorted according to the training matching degree corresponding to at least one target training object, and the training matching degree represents the degree of matching between the corresponding target training object and the training user; and a reinforcement learning module 606, used to perform reinforcement learning on the initial recommendation model based on at least one interaction object and the training object recommendation list to obtain an object recommendation model, wherein the object recommendation model is used to execute the above methods.

[0141] The reinforcement learning module is also used to evaluate the rewards of at least one interactive object and the training object recommendation list based on a preset reward rule to obtain a total training reward value; and to update the model parameters of the initial recommendation model based on the total training reward value to obtain an object recommendation model.

[0142] The preset reward rules include: recommendation reward rules and ranking reward rules. The reinforcement learning module is also used to evaluate the rewards of at least one target training object and at least one interaction object in the training object recommendation list based on the recommendation reward rules, and obtain the training recommendation reward value; to evaluate the rewards of at least one negative sample training object in the training object recommendation list based on the ranking reward rules, and obtain the training ranking reward value, wherein at least one negative sample training object is a target training object in the training object recommendation list that fails to match at least one interaction object; and to summarize the training recommendation reward value and the training ranking reward value to obtain the total training reward value.

[0143] The reinforcement learning module is also used to determine the training advantage function value corresponding to the initial recommendation model based on the total training reward value; generate the training objective function value corresponding to the initial recommendation model based on the training advantage function value and the training matching degree corresponding to at least one target training object; and update the model parameters of the initial recommendation model based on the training objective function value to obtain the object recommendation model.

[0144] The generation module is also used to match any training object with the initial recommendation model to determine candidate training objects from multiple training objects; if there is no target training object in the training object recommendation list that is the same as the candidate training object, the candidate training object is added to the training object recommendation list.

[0145] The reinforcement learning module is also used to re-match any training object with the candidate training object when there is a target training object in the training object recommendation list that is the same as the candidate training object, and to determine a new candidate training object from multiple training objects until there is no target training object in the training object recommendation list that is the same as the new candidate training object.

[0146] Embodiments of this application may provide an electronic device. Figure 7 This is a structural block diagram of an electronic device according to an embodiment of this application, such as... Figure 7 As shown, the electronic device may include: an input / output device 702; a memory 704; and a processor 706, wherein the processor 706 is connected to the input / output device 702 and the memory 704 via a bus 708.

[0147] The memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the methods and apparatus in the embodiments of this application. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, thereby implementing the methods in the above embodiments. The memory may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include memory remotely located relative to the processor, and these remote memories can be connected to terminal A via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0148] The processor can invoke an executable program stored in memory via a transmission device to perform the following methods: obtaining user information and object information corresponding to multiple objects; inputting the user information and object information into an object recommendation model, using the object recommendation model to match multiple objects with the user respectively, generating an object recommendation list, wherein the object recommendation list contains at least one target object selected from multiple objects, and the at least one target object is sorted according to the matching degree corresponding to the at least one target object, the matching degree being used to characterize the degree of matching between the corresponding target object and the user, and the object recommendation model being trained using reinforcement learning; determining a target object from the object recommendation list; pushing the target object to the user; and executing the methods in the various embodiments of this application.

[0149] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0150] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0151] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, they can also be implemented by hardware. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause the processing unit to execute the methods in the various embodiments of this application.

[0152] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing the hardware related to the terminal device. The program can be stored in a computer-readable storage medium, which may include: flash drive, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0153] Embodiments of this application also provide a computer-readable storage medium. Optionally, in this embodiment, the computer-readable storage medium can be used to store program code executed by the method provided in the above embodiments.

[0154] Optionally, in this embodiment, the storage medium may be located in a computing device.

[0155] Optionally, in this embodiment, the computer-readable storage medium is configured to store an executable program, which, when the executable program is running, controls the device where the computer-readable storage medium is located to execute the method described in any of the above embodiments.

[0156] Embodiments of this application also provide a computer program product. Optionally, in this embodiment, the computer program product may include a computer program that, when executed by a processor, implements the methods provided in the embodiments described above.

[0157] The aforementioned computer program products can refer to software programs that have been written, tested, and released, and can run on computers or other devices. Computer program products can include application programs, operating systems, utility software, etc., used to achieve specific functions or solve specific problems.

[0158] Embodiments of this application also provide a computer program product. Optionally, the computer program product may include a non-volatile computer-readable storage medium, which can be used to store a computer program that, when executed by a processor, implements the method provided in the above embodiments.

[0159] The aforementioned non-volatile computer-readable storage medium can refer to a medium for storing data. Non-volatile computer-readable storage media can retain data without loss when power is off and can be used to store long-term data, such as operating systems, applications, and user files. Non-volatile storage media can include hard disk drives, solid-state drives, optical disks, and flash memory storage devices, etc.

[0160] Embodiments of this application also provide a computer program. Optionally, in this embodiment, when the computer program is executed by a processor, it implements the method provided in the above embodiments.

[0161] The aforementioned computer program can refer to a set of instructions used to tell the computer to perform specific tasks or operations. Computer programs can be written by programmers using specific programming languages ​​and can include algorithms, data structures, logic, and control flow. Computer programs can be used for a variety of purposes, including application software, operating systems, etc.

[0162] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0163] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0164] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0165] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0166] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0167] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. An object recommendation method, characterized in that, include: Obtain the user's historical interaction information, wherein the historical interaction information is used to characterize the object information corresponding to multiple interaction objects that the user has interacted with, as well as the information generated during the interaction between the user and the multiple interaction objects; The historical interaction information is input into the object recommendation model, and the object recommendation model is used to generate an object recommendation list. The object recommendation list contains at least one target object generated by the object recommendation model. The object recommendation list is obtained by sorting according to the matching degree corresponding to the at least one target object. The matching degree is used to characterize the degree of matching between the corresponding target object and the user. The object recommendation model is trained using reinforcement learning. The at least one target object is pushed to the user based on the object recommendation list.

2. The method according to claim 1, characterized in that, The step of inputting the historical interaction information into the object recommendation model and generating an object recommendation list using the object recommendation model includes: The historical interaction information is input into the object recommendation model, and the object recommendation model is used to generate the at least one target object, wherein the matching degree between the target object and the user is greater than a preset degree; The object recommendation model is used to match the at least one target object with the user, and the matching degree corresponding to the at least one target object is determined respectively; Using the object recommendation model, the at least one target object is sorted based on its matching degree to generate the object recommendation list.

3. The method according to claim 1 or 2, characterized in that, The method further includes: Output the recommended list of the objects; In response to receiving the modified recommendation list from the user, a reward evaluation is performed on the modified recommendation list and the object recommendation list based on a preset reward rule to obtain a total reward value, wherein the modified recommendation list is obtained by modifying the sorting order of the at least one target object and / or the at least one target object; The model parameters of the object recommendation model are updated based on the total reward value.

4. The method according to claim 3, characterized in that, The preset reward rules include: recommendation reward rules and ranking reward rules; the reward evaluation of the modified recommendation list and the object recommendation list based on the preset reward rules to obtain the total reward value includes: Based on the recommendation reward rules, a reward evaluation is performed on at least one target object in the object recommendation list and at least one modified object in the modified recommendation list to obtain a recommendation reward value; Based on the ranking reward rule, at least one negative sample object in the object recommendation list is evaluated for a reward to obtain a ranking reward value, wherein the negative sample object is a target object in the object recommendation list that fails to match the at least one modified object; The total reward value is obtained by summing the recommended reward value and the ranking reward value.

5. The method according to claim 4, characterized in that, The step of evaluating the rewards for at least one target object in the object recommendation list and at least one modified object in the modified recommendation list based on the recommendation reward rule to obtain a recommendation reward value includes: For any target object in the object recommendation list, match the target object with at least one modified object in the modification recommendation list to obtain the matching result corresponding to the target object; Based on the matching results corresponding to the target object, a first reward value corresponding to the target object is determined; The recommended reward value is determined by summing up the first reward values ​​corresponding to the at least one target object.

6. The method according to claim 4, characterized in that, The step of evaluating the ranking reward value for at least one negative sample object in the object recommendation list based on the ranking reward rule includes: Based on the ranking reward rule, the matching degree of the negative sample object is evaluated to obtain the second reward value corresponding to the negative sample object; The ranking reward value is obtained by summing the second reward values ​​corresponding to the at least one negative sample object.

7. The method according to claim 3, characterized in that, The step of updating the model parameters of the object recommendation model based on the total reward value includes: Based on the total reward value, determine the advantage function value corresponding to the object recommendation model; Based on the advantage function value and the matching degree corresponding to the at least one target object, the objective function value corresponding to the object recommendation model is generated. The model parameters of the object recommendation model are updated based on the objective function value.

8. A training method for an object recommendation model, characterized in that, include: Obtain historical training interaction information of the training user, wherein the historical training interaction information is used to characterize the object information corresponding to multiple interaction objects that the training user has interacted with, as well as the information generated during the interaction between the training user and the multiple interaction objects; The historical training interaction information is input into the initial recommendation model, and the initial recommendation model is used to generate a training object recommendation list. The training object recommendation list contains at least one target training object generated by the object recommendation model. The training object recommendation list is sorted according to the training matching degree corresponding to the at least one target training object. The training matching degree is used to characterize the degree of matching between the corresponding target training object and the training user. Based on the at least one interactive object and the training object recommendation list, reinforcement learning is performed on the initial recommendation model to obtain an object recommendation model, wherein the object recommendation model is used to perform the method described in any one of claims 1 to 7.

9. The method according to claim 8, characterized in that, The step of performing reinforcement learning on the initial recommendation model based on the at least one interactive object and the training object recommendation list to obtain an object recommendation model includes: Based on a preset reward rule, a reward evaluation is performed on the at least one interactive object and the recommended list of training objects to obtain a total training reward value; The model parameters of the initial recommendation model are updated based on the total training reward value to obtain the object recommendation model.

10. The method according to claim 9, characterized in that, The preset reward rules include: a recommendation reward rule and a ranking reward rule; the step of evaluating the rewards of the at least one interactive object and the training object recommendation list based on the preset reward rules to obtain a total training reward value includes: Based on the recommendation reward rule, a reward evaluation is performed on at least one target training object and at least one interaction object in the training object recommendation list to obtain a training recommendation reward value; Based on the ranking reward rule, at least one negative sample training object in the training object recommendation list is evaluated for reward to obtain a training ranking reward value, wherein the at least one negative sample training object is a target training object in the training object recommendation list that fails to match the at least one interaction object. The training recommendation reward value and the training ranking reward value are summed to obtain the total training reward value.

11. The method according to claim 9, characterized in that, The step of updating the model parameters of the initial recommendation model based on the total training reward value to obtain the object recommendation model includes: Based on the total training reward value, determine the training advantage function value corresponding to the initial recommendation model; Based on the training advantage function value and the training matching degree corresponding to the at least one target training object, the training objective function value corresponding to the initial recommendation model is generated. The model parameters of the initial recommendation model are updated based on the training objective function value to obtain the object recommendation model.

12. The method according to claim 8, characterized in that, The initial recommendation model is used to match the multiple training objects with the user to generate a training object recommendation list, including: The initial recommendation model is used to match any training object with the candidate training object, thereby determining the candidate training object from the plurality of training objects. If there is no target training object in the training object recommendation list that is the same as the candidate training object, the candidate training object is added to the training object recommendation list.

13. The method according to claim 12, characterized in that, The method further includes: If a target training object that is the same as the candidate training object exists in the training object recommendation list, the initial recommendation model is reused to match any training object with it, and a new candidate training object is determined from the multiple training objects, until no target training object that is the same as the new candidate training object exists in the training object recommendation list.

14. An electronic device, characterized in that, include: Memory, which stores executable programs; A processor, connected to a memory via a bus, is used to run the program, wherein the program, when running, performs the method described in any one of claims 1 to 13.

15. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored executable program, wherein, when the executable program is executed, it controls the device on which the computer-readable storage medium is located to perform the method according to any one of claims 1 to 13.

16. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method described in any one of claims 1 to 13.

Images