Object recommendation method, training method of object recommendation model and electronic equipment

Abstract

The invention provides an object recommendation method, a training method of an object recommendation model and electronic equipment, and relates to the field of artificial intelligence technology and large models. The method comprises the steps that preference information of a user and object information corresponding to a plurality of to-be-recommended objects are obtained, and the preference information is used for representing the preference of the user for the to-be-recommended objects; a plurality of reasoning tasks are executed on the preference information and object information corresponding to the to-be-recommended object through the object recommendation model, a reasoning track corresponding to the to-be-recommended object is generated, and the reasoning track comprises execution results corresponding to the reasoning tasks respectively; determining a target object from the plurality of to-be-recommended objects based on the reasoning trajectories corresponding to the plurality of to-be-recommended objects; and pushing the target object to the user. The technical problem that the model recommendation effect is poor in the prior art is solved.

Description

Technical Field

[0001] This application relates to artificial intelligence technology and the field of large models. Specifically, it relates to an object recommendation method, a training method for an object recommendation model, and an electronic device. Background Technology

[0002] Currently, recommender systems predict user preferences by analyzing historical user-item interactions, such as ratings, clicks, and purchases. Traditional recommender algorithms primarily rely on user-item interaction data (e.g., ratings, click records, and purchase history) for prediction. However, with the increasing diversification and personalization of user needs, the limitations of these methods are becoming increasingly apparent, especially in handling sophisticated reasoning and deep semantic understanding, resulting in poor recommendation performance for current models.

[0003] There is currently no effective solution to the above problems. Summary of the Invention

[0004] This application provides an object recommendation method, an object recommendation model training method, and an electronic device to at least solve the technical problem of poor model recommendation performance in related technologies.

[0005] According to one aspect of the embodiments of this application, an object recommendation method is provided, comprising: acquiring user preference information and object information corresponding to multiple objects to be recommended; performing multiple inference tasks on the preference information and the object information corresponding to the objects to be recommended using an object recommendation model to generate an inference trajectory corresponding to the objects to be recommended, wherein the inference trajectory includes: the execution results corresponding to the multiple inference tasks, the multiple inference tasks including: preference information extraction task, object information extraction task, matching degree evaluation task, and recommendation score prediction task, the object recommendation model being trained through reinforcement learning; determining a target object from the multiple objects to be recommended based on the inference trajectories corresponding to the multiple objects to be recommended; and pushing the target object to the user.

[0006] According to one aspect of the embodiments of this application, a method for training an object recommendation model is provided, comprising: acquiring training data, wherein the training data includes: training preference information, training object information, and preset recommendation scores; performing multiple inference tasks on the training preference information and training object information using an initial recommendation model to generate a training inference trajectory, wherein the training inference trajectory includes: training execution results corresponding to the multiple inference tasks, the multiple inference tasks including: preference information extraction task, object information extraction task, matching degree evaluation task, and recommendation score prediction task; and performing reinforcement learning on the initial recommendation model based on the training inference trajectory and the preset recommendation scores to obtain an object recommendation model, wherein the object recommendation model is used to execute the method of any one of the above embodiments.

[0007] According to another aspect of the embodiments of this application, a computing device is also provided, including: a memory storing an executable program; and a processor for running the program, wherein the program executes the methods in various embodiments of this application when it runs.

[0008] According to another aspect of the embodiments of this application, an electronic device is also provided, including: a memory storing an executable program; and a processor connected to the memory via a bus for running the program, wherein the program executes the methods in various embodiments of this application when it runs.

[0009] According to another aspect of the embodiments of this application, a computer-readable storage medium is also provided, the computer-readable storage medium including a stored executable program, wherein, when the executable program is running, it controls the device where the computer-readable storage medium is located to perform the methods of various embodiments of this application.

[0010] According to another aspect of the embodiments of this application, a computer program product is also provided, including a computer program that, when executed by a processor, implements the methods of various embodiments of this application.

[0011] According to another aspect of the embodiments of this application, a computer program product is also provided, including a non-volatile computer-readable storage medium storing a computer program, which, when executed by a processor, implements the methods in various embodiments of this application.

[0012] According to another aspect of the embodiments of this application, a computer program is also provided, which, when executed by a processor, implements the methods of the various embodiments of this application.

[0013] In this embodiment, user preference information and object information corresponding to multiple objects to be recommended are obtained. An object recommendation model is used to perform multiple inference tasks on the preference information and the object information corresponding to the objects to be recommended, generating inference trajectories for the objects to be recommended. These inference trajectories include the execution results of the multiple inference tasks, which include preference information extraction, object information extraction, matching degree evaluation, and recommendation score prediction. The object recommendation model is trained using reinforcement learning. Based on the inference trajectories corresponding to the multiple objects to be recommended, a target object is determined from among them. The target object is then pushed to the user. By comparing the inference trajectories of each object to be recommended, especially the execution results of its recommendation score prediction task, the model can identify objects with higher recommendation scores, which are the target objects. This decision-making process is based on the matching degree between user preferences and object characteristics reflected in the inference trajectories, as well as the deep understanding of the recommendation scenario and improvement strategies obtained by the model through reinforcement learning training. By pushing target objects to users, this application improves the accuracy of personalized recommendations. This is because the recommendation score prediction task directly reflects the user's potential rating tendency for each object to be recommended, while the matching degree evaluation task ensures that the recommended content is highly consistent with the user's preferences, thereby solving the technical problem of poor model recommendation performance in related technologies.

[0014] The above general description and the following detailed description are for illustrative and explanatory purposes only and do not constitute a limitation thereof. Attached Figure Description

[0015] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0016] Figure 1 This is a schematic diagram illustrating an application scenario of an object recommendation method according to an embodiment of this application;

[0017] Figure 2 This is a flowchart of an object recommendation method according to an embodiment of this application;

[0018] Figure 3 This is a schematic diagram of a recommended process according to an embodiment of this application;

[0019] Figure 4 This is a flowchart of a training method for an object recommendation model according to an embodiment of this application;

[0020] Figure 5 This is a schematic diagram of an object recommendation device according to an embodiment of this application;

[0021] Figure 6This is a schematic diagram of a training device for an object recommendation model according to an embodiment of this application;

[0022] Figure 7 This is a structural block diagram of a computing device according to an embodiment of this application;

[0023] Figure 8 This is a structural block diagram of an electronic device according to an embodiment of this application. Detailed Implementation

[0024] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the embodiments described below are only some, not all, of the embodiments of the present application. All other embodiments obtained by those skilled in the art based on the embodiments of the present application without creative effort should fall within the scope of protection of the present application.

[0025] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in other orders. Wherein, "other orders" refers to orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such as a process, method, system, product, or apparatus that comprises a series of steps or units, not necessarily limited to those explicitly listed, but may include other steps or units not explicitly listed, or inherent to such processes, methods, products, or apparatus.

[0026] The preference information (including but not limited to user device information, user interaction records, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation interfaces are provided for users to choose to authorize or refuse.

[0027] The technical solution provided in this application is mainly implemented using a deep learning model. Deep learning models can be widely applied in fields such as Natural Language Processing (NLP), computer vision, and speech processing. Specifically, they can be applied to computer vision tasks such as Visual Question Answering (VQA), Image Captioning (IC), and image generation, as well as to natural language processing tasks such as text-based sentiment classification, text summarization, and machine translation. Therefore, the main application scenarios of this application include, but are not limited to, digital assistants, intelligent robots, search, online education, office software, e-commerce, and intelligent design.

[0028] First, some nouns or terms that appear in the description of the embodiments of this application shall be interpreted as follows:

[0029] Mean Absolute Error (MAE) measures the average magnitude of the absolute difference between a predicted value and the actual value.

[0030] The root average squared error (RASE) is used to evaluate prediction accuracy.

[0031] According to embodiments of this application, an object recommendation method is provided. The steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowcharts, in some cases the steps shown or described may be performed in a different order than that shown here.

[0032] The technical solutions provided in this application can employ deep learning models with relatively large parameter scales, such as large models containing billions or even more model parameters. Here, "large model" is just one example; this application does not limit the number of model parameters supported by the deep learning model used, aiming to meet actual needs. The deep learning models involved in this application can be artificial intelligence-based language models (LM) or multimodal models (MM).

[0033] Considering the limited computing resources of mobile terminals, the methods described above in this application embodiment can be applied to, for example... Figure 1 The application scenarios shown. Figure 1 This is a schematic diagram illustrating an application scenario of an object recommendation method according to an embodiment of this application. Figure 1In the application scenario shown, the deep learning model is deployed on server 10. Server 10 can connect to one or more client devices 20 via a local area network (LAN), wide area network (WAN), internet connection, or other types of data network. Client devices 20 may include, but are not limited to, smartphones, tablets, laptops, PDAs, personal computers, smart home devices, and in-vehicle devices. Client devices 20 can interact with users through a graphical user interface to invoke the deep learning model, thereby implementing the method provided in this embodiment.

[0034] In this embodiment, the system consisting of a client device and a server can perform the following steps: The client device generates user preference information and object information corresponding to multiple objects to be recommended. The server obtains the user preference information and object information corresponding to multiple objects to be recommended; uses an object recommendation model to perform multiple inference tasks on the preference information and object information corresponding to the objects to be recommended, generating inference trajectories corresponding to the objects to be recommended; based on the inference trajectories corresponding to the multiple objects to be recommended, determines the target object from the multiple objects to be recommended; and pushes the target object to the user.

[0035] With the rapid development of high-performance computing units, the methods provided in this application embodiment can also be applied to model-in-the-loop machines in other application scenarios. In one optional embodiment, the model-in-the-loop machine has multiple built-in models. Users can select a model to adjust as needed to obtain their own model. The high-performance computing unit built into the model-in-the-loop machine can then directly call the adjusted model to execute the methods provided in this application embodiment. In another optional embodiment, the deep learning model-in-the-loop machine has a pre-trained model built-in. The high-performance computing unit built into the model-in-the-loop machine can then directly call this model to execute the methods provided in this application embodiment.

[0036] Furthermore, when users need to train their own models, they can upload their own datasets via the client. This dataset is sent from the client to the server. The server can then use this dataset to fine-tune the pre-trained model, resulting in the user's customized model, which can then be deployed to the production environment. To facilitate user adjustments, the server provides complete adjustment tools, development frameworks, and processes, supporting various adjustment strategies. This allows the adjusted model to better adapt to different application domains and achieve a high degree of customization.

[0037] Under the aforementioned operating environment, this application provides the following: Figure 2 The recommended method for the object shown. Figure 2 This is a flowchart of an object recommendation method according to an embodiment of this application. For example... Figure 2 As shown, the method may include the following steps:

[0038] Step S202: Obtain the user's preference information and the object information corresponding to the multiple objects to be recommended.

[0039] Among them, preference information is used to characterize a user's preferences for multiple objects to be recommended.

[0040] The aforementioned preference information refers to data showing a user's inclination towards specific content, functions, or interaction methods during the use of the system or service. This includes, but is not limited to, the user's search history, purchase records, evaluation feedback, browsing time, click preferences, and user reviews, used to build user profiles, accurately reflect user interests and needs, so that the system or model can provide more personalized recommendations and services.

[0041] The aforementioned objects to be recommended refer to the physical or virtual goods or content that the system intends to recommend to users, such as products, music, articles, and advertisements. Information about these objects includes descriptions, categories, attributes, and user feedback, forming the basis for the recommendation model's analysis and matching. The aim is to use algorithms to filter out items that match user preferences, thereby improving user satisfaction and recommendation effectiveness. Virtual goods can include digital collectibles, etc.

[0042] The object information mentioned above represents descriptive data about a specific entity or content, including but not limited to the characteristics, functions, appearance, and ratings of an item, used to characterize its attractiveness and value. In recommender systems, object information is a key input for models to understand item attributes, predict user preferences, and make personalized recommendations.

[0043] In one optional embodiment, user preference information and object information corresponding to multiple items to be recommended are obtained. The preference information covers the user's areas of interest, historical reviews, purchase history, etc., used to accurately depict the user's preferences for various items; the information of the items to be recommended includes item descriptions, attributes, categories, other user feedback, etc., helping the model understand the characteristics and attractiveness of the items. Obtaining this information is crucial for the model because it constitutes the basic input for the inference task, enabling the model to perform subsequent inference analysis based on the characteristics of the user and the items.

[0044] Step S204: Use the object recommendation model to perform multiple inference tasks on the preference information and the object information corresponding to the object to be recommended, and generate the inference trajectory corresponding to the object to be recommended.

[0045] The reasoning trajectory includes the execution results of multiple reasoning tasks, which include: preference information extraction, object information extraction, matching degree evaluation, and recommendation score prediction. The object recommendation model is trained through reinforcement learning. The aforementioned object information extraction task can be a task of summarizing key points of items.

[0046] The aforementioned Item Recommendation Model (IRM) is an algorithmic model trained through machine learning, particularly reinforcement learning (RL), to understand and predict a user's preference for specific objects. This model comprehensively considers the user's historical behavior, personal attributes, and object characteristics. Through extensive interaction and improvement, it learns to recommend content that matches the user's preferences, such as products, articles, or music.

[0047] In a recommender system, the aforementioned reasoning task involves the model analyzing and predicting based on existing data, aiming to reveal the potential relationship between users and the objects to be recommended. This includes stages such as preference information extraction, object information extraction, matching degree evaluation, and recommendation score prediction. Each task focuses on mining information from different dimensions to support the final recommendation decision.

[0048] In recommender systems, the aforementioned reasoning trajectories refer to the set of results generated by the model after performing a series of reasoning tasks, including all details from preference information extraction to recommendation score prediction. These trajectories not only reveal the model's thought process but can also be used to subsequently improve and understand the effectiveness of the recommendation process, making them key elements for achieving personalized recommendations and improving recommendation quality.

[0049] The aforementioned preference information extraction task is a type of inference task, specifically user interest extraction, which focuses on extracting key preference features from user behavioral data, historical reviews, and other available information. By analyzing users' interests, consumption habits, and interaction patterns, this task helps the model build user profiles, providing a foundation for personalized recommendations.

[0050] The object information extraction task described above is part of the reasoning task. It involves summarizing the key points of items and focusing on analyzing the detailed characteristics of the items to be recommended, such as descriptions, categories, attributes, and past user feedback. This task is crucial for understanding the unique selling points of items and helps improve the targeting and appeal of recommendations.

[0051] The purpose of the matching evaluation task described above is to measure the degree of fit between preference information and object information. By comparing user preferences and item features, the model can assess the likelihood of a match, providing an important basis for predicting recommendation scores.

[0052] The aforementioned recommendation score prediction task is the final step in the inference task. Based on the results of all previous tasks, the model predicts the user's potential rating or level of preference for a particular object. This prediction is the core of the recommendation decision, guiding the system to recommend appropriate content to the user.

[0053] In one optional embodiment, an object recommendation model is used to perform a series of reasoning tasks on the aforementioned acquired preference information and detailed information of the objects to be recommended, including preference information extraction, object information extraction, matching degree evaluation, and recommendation score prediction. The execution results of each reasoning task constitute a reasoning trajectory, which reflects how the model extracts key features from user and item information, evaluates the degree of matching between the two, and ultimately predicts the recommendation score. The object recommendation model, trained through reinforcement learning, can continuously improve its reasoning strategy through interaction with the environment, thereby enhancing the accuracy and efficiency of recommendations.

[0054] Step S206: Based on the reasoning trajectories corresponding to multiple objects to be recommended, determine the target object from the multiple objects to be recommended.

[0055] The aforementioned inference trajectory is a record of the inference process generated by the model when analyzing user and recommended object information, including the execution results of tasks such as preference information extraction, object information extraction, matching degree evaluation, and recommendation score prediction.

[0056] The target objects mentioned above are specific entities or content that meet user preferences and needs, determined by the model based on the evaluation results of the inference trajectory among multiple objects to be recommended, and are the final selections for recommendation decisions.

[0057] Based on the inference trajectories corresponding to multiple candidate objects, a target object is determined from these candidate objects. In this embodiment, this stage is a comprehensive analysis and evaluation process of the results of the preceding inference task. Its core lies in interpreting the inference trajectory, extracting the matching degree between each candidate object and user preferences, and ranking and selecting based on this matching degree. To determine the target object, the model needs to evaluate the degree of consistency between the preference information reflected in the inference trajectory and the characteristics of the candidate object, as well as the results of the recommendation score prediction task, thereby determining which candidate object is likely to be favored by the user. Intelligent recommendation decision-making is achieved through quantitative comparison, rather than relying solely on surface feature matching, ensuring that the recommended content is highly relevant and attractive based on a deep understanding of user needs.

[0058] This application's embodiments can accurately filter target objects from multiple potential recommendations. This process, based on a deep understanding of the inference trajectory, significantly improves the personalization and accuracy of recommendations. Because the inference trajectory includes multi-dimensional analysis of user preferences and item characteristics, the model can fully consider the user's historical behavior, interests, and the attractiveness of the item when determining the target object. This avoids blind recommendations, reduces the mismatch between recommended content and user needs, thereby improving the user experience and enhancing the effectiveness and user satisfaction of the recommendation system.

[0059] For example, taking the recommender system field in this application embodiment as an example, imagine a scenario where a user browses an e-commerce platform. The system collects the user's preference information, including the user's preferences for specific brands, categories, and price ranges, as well as detailed information on multiple objects to be recommended, such as product descriptions, user reviews, and sales data. Next, the model performs multiple inference tasks, generating an inference trajectory for each object to be recommended. These trajectories record in detail how the model analyzes user preferences, summarizes item characteristics, evaluates matching degree, and predicts recommendation scores. Based on these inference trajectories, the system performs intelligent analysis, identifies products that match the user's preferences as target objects, and pushes them to the user. This process not only relies on the user's historical behavior and item attributes but also deeply analyzes the inherent compatibility between the two, ensuring that the recommendation results not only match the user's tastes but also have high relevance and personalization, thereby significantly improving the efficiency of the recommender system.

[0060] The above steps ensure that recommendation decisions are based on a deep understanding of user preferences. By quantitatively comparing data in the inference trajectory, the system can accurately identify which recommended objects can meet the user's specific needs, ultimately achieving personalized matching of recommended content and improving user satisfaction and recommendation efficiency.

[0061] Based on the generated inference trajectory, the model will comprehensively analyze all objects to be recommended and determine those with higher recommendation scores as target objects. This decision is based on the model's deep understanding of user preferences and the analysis results obtained through matching degree evaluation and recommendation score prediction tasks, ensuring that the target objects meet the user's expectations.

[0062] Step S208: Push the target object to the user.

[0063] The above steps are the final stage of the recommendation system's workflow. Prior to this, the system has identified target objects that match user preferences through a series of reasoning tasks.

[0064] The aforementioned process of pushing targeted content to users can be viewed technically as personalized content delivery. This involves presenting carefully selected content or entities (such as products, music, or articles) directly to users based on their unique needs and preferences. This process typically involves updating the user interface (UI), push notifications, or recommendation lists to ensure users quickly access the targeted content that meets their expectations.

[0065] Within the framework of this application's embodiments, once a target object is identified, the system immediately takes action, visually displaying the target object to the user through a user interface or notification mechanism, aiming to stimulate user interest and promote user interaction with the platform. This push process includes not only the design of content display but also the selection of push strategies (such as push timing and display methods), as well as the real-time monitoring and analysis of user feedback by the backend system, in order to adjust the recommendation strategy subsequently.

[0066] In the steps described above, users receive content that matches their preferences in a timely manner, enhancing their trust in the recommendation system and increasing their engagement, thereby boosting user activity and satisfaction. This push mechanism effectively avoids content overload, ensuring that each recommendation is based on in-depth analysis and a precise match to the user's personalized needs, thus reducing the time cost for users to filter information and improving the overall user experience.

[0067] For example, suppose a user leaves a series of ratings and comments on the platform regarding their preferred music genres, artists, and albums. After a series of reasoning tasks, including preference information extraction, object information extraction, matching degree evaluation, and recommendation score prediction, the system identifies several songs that the user might like as target objects. Subsequently, in step S208, these target objects are organized into a personalized recommendation list and pushed directly to the user through the recommendation page in the user's mobile application or email notification. When the user opens the application or email, they can see music recommendations tailored to them. This direct and personalized push method not only meets the user's need to find new music but also reduces the time users spend blindly searching through a massive music library, significantly improving the user experience and increasing the platform's music playback volume and user retention rate.

[0068] The above steps play a crucial role in connecting user preferences with target audiences. Through precise push notifications, a seamless connection between recommended content and user expectations is achieved, enhancing the practical value of the recommendation system and user satisfaction.

[0069] The aforementioned target objects will be pushed to the user. By performing the above reasoning task, the embodiments of this application can significantly improve the accuracy of recommendations, reduce irrelevant recommendations, thereby improving the user experience in the recommendation system, increasing user satisfaction, and enhancing the platform's recommendation effectiveness.

[0070] For example, taking user behavior on a music recommendation platform as an example, the system first collects preference information, such as the types of songs the user has given high ratings to, artist preferences, and listening times, as well as metadata about the songs to be recommended, such as style, lyrical themes, and popularity. Then, the object recommendation model performs inference tasks based on this information, such as extracting the user's preferences for specific music styles and artists, summarizing the attributes of the songs to be recommended, evaluating the matching degree between user preferences and song features, and predicting a recommendation score for each song. Finally, songs with high recommendation scores are identified as target objects and pushed to the user, making the recommended music more closely aligned with the user's preferences, improving the user experience and the overall performance of the recommendation system.

[0071] This application embodiment significantly improves the accuracy and personalization of recommendation decisions by performing inference tasks and generating inference trajectories using an object recommendation model. Through a deep understanding of user preferences and the characteristics of the objects to be recommended, this application embodiment effectively enhances the relevance of recommended content, reduces invalid recommendations, and thus improves the user experience on the platform, enhancing the overall effectiveness of the recommendation system and user satisfaction.

[0072] This application's embodiments utilize a reinforcement learning-driven object recommendation model to accurately capture and satisfy users' personalized needs. First, the system collects preference information and data on objects to be recommended. Based on this information, the model executes a series of inference tasks, including preference information extraction, object information extraction, matching degree evaluation, and recommendation score prediction. The results of each task are integrated into an inference trajectory. Through in-depth analysis of user and object characteristics, the inference trajectory provides comprehensive and detailed decision-making basis, significantly improving the personalization and accuracy of recommendations. Next, the system uses the inference trajectory to identify target objects that closely match user preferences. This process not only considers the user's historical behavior and current interests but also deeply understands the object's attributes and attractiveness, ensuring that recommended content not only suits the user's tastes but also possesses high relevance and attractiveness, thereby improving user experience and the efficiency of the recommendation system. Finally, the target objects are pushed to the user. This push mechanism, based on a precise understanding of user preferences, ensures high-quality and highly relevant recommended content, reduces invalid recommendations, and increases user satisfaction and trust in the recommendation system, thereby promoting deep interaction and long-term user retention with the platform.

[0073] The steps in this embodiment introduce a reinforcement learning-trained object recommendation model to generate inference trajectories, and then determine and push target objects based on the trajectories. This achieves intelligent and personalized recommendation decisions, effectively improving the efficiency and quality of recommendations and providing users with a better recommendation experience that better meets their needs.

[0074] Through the above steps, user preference information and object information corresponding to multiple objects to be recommended are obtained. An object recommendation model is then used to perform multiple inference tasks on the preference information and the object information corresponding to the objects to be recommended, generating inference trajectories for the objects to be recommended. These inference trajectories include the execution results of multiple inference tasks, which include preference information extraction, object information extraction, matching degree evaluation, and recommendation score prediction. The object recommendation model is trained through reinforcement learning. Based on the inference trajectories corresponding to the multiple objects to be recommended, a target object is determined from among them. The target object is then pushed to the user. By comparing the inference trajectories of each object to be recommended, especially the execution results of its recommendation score prediction task, the model can identify objects with higher recommendation scores, which are then designated as the target objects. This decision-making process is based on the matching degree between user preferences and object characteristics reflected in the inference trajectories, as well as the deep understanding of the recommendation scenario and improvement strategies gained by the model through reinforcement learning training. By pushing target objects to users, this application improves the accuracy of personalized recommendations. This is because the recommendation score prediction task directly reflects the user's potential rating tendency for each object to be recommended, while the matching degree evaluation task ensures that the recommended content is highly consistent with the user's preferences, thereby solving the technical problem of poor model recommendation performance in related technologies.

[0075] In the above embodiments of this application, an object recommendation model is used to perform multiple inference tasks on preference information and object information corresponding to the object to be recommended, generating an inference trajectory corresponding to the object to be recommended. This includes: adding preference information and object information to a prompt word template to obtain a target prompt word, wherein the prompt word template contains multiple inference tasks and output labels corresponding to the multiple inference tasks; inputting the target prompt word into the object recommendation model, and using the object recommendation model to perform multiple inference tasks on preference information and object information to generate an inference trajectory, wherein the execution results corresponding to the inference tasks are located between the output labels corresponding to the inference tasks.

[0076] The aforementioned preference and object information serve as input to describe user preferences and the characteristics of the objects to be recommended. This input is the cornerstone of the recommender system, encompassing multiple aspects such as the user's historical behavior, preference settings, and object descriptions and classifications; it is a prerequisite for the model's reasoning.

[0077] The aforementioned Prompt Template (PT) is a structured guidance framework used to standardize the model's inference process and ensure that it generates inference trajectories according to preset steps. The Prompt Template clarifies the task boundaries and expected output by introducing specific tags (ST), and is a key component of the reinforcement learning fine-tuning strategy in this embodiment.

[0078] The first step in this embodiment is to integrate preference information and object information into a "cue word template" to form a "target cue word." This integration process is essentially information reorganization, placing user behavior data and object attributes in a structured context so that the model can process information in an orderly manner and proceed along a specified reasoning path. Next, the target cue word is input into the object recommendation model, which is trained through reinforcement learning and possesses the ability to independently complete reasoning tasks. After receiving the target cue word, the model executes the reasoning tasks one by one according to the multiple reasoning tasks and their corresponding output labels in the cue word template, generating an inference path. The inference path records the model's output when processing each task. For example, when the model encounters an output label extracted from user interests, it outputs a summary of user preferences; when it encounters a label summarizing item key points, it outputs a description of the object's key attributes. This process ensures the logical coherence and task orientation of the reasoning.

[0079] This application designs a global prompt word to guide the model in generating an inference trajectory beneficial to rating prediction. The inference process is broken down into three steps: user interest extraction, item key point summarization, and matching degree evaluation. In this embodiment, the global prompt word serves as a prompt word template, embedding instructions for multiple inference tasks and corresponding output labels to guide the model in sequentially executing inference tasks such as user preference analysis, item feature extraction, and compatibility evaluation. When the model encounters the global prompt word, it first identifies and extracts user preferences according to the preset label order, then summarizes the key attributes of the item, and finally evaluates the matching degree between the two, ultimately outputting a predicted rating. This design ensures the structured and goal-oriented nature of the inference process, helping to improve the accuracy of rating prediction and the depth of model inference.

[0080] By integrating preference and object information into prompt word templates and employing a reinforcement learning-trained object recommendation model to generate inference trajectories, this application effectively achieves fine-grained control over the inference process of the rating prediction task. This operational mode not only standardizes the model's inference behavior, ensuring the logic and completeness of the inference process, but also guides the model's output through specific labels, greatly improving the quality and practicality of the inference results. Compared to traditional supervised fine-tuning or unstructured inference frameworks, this solution significantly improves the accuracy of rating prediction while enhancing the model's understanding and response speed to personalized user needs.

[0081] For example, in a music recommendation scenario, preference information may include a user's past listening history, preferred genres, and artist reviews, while object information includes a song's genre, lyrical sentiment, and musician bio. This application's embodiments embed this information into a globally designed prompt template, which contains task descriptions and output labels for user interest extraction, music summary, and matching evaluation. When this template is input into the model, the model begins to perform structured reasoning, first identifying user preferences (e.g., a preference for relaxing music), then analyzing music features (confirming the song belongs to the relaxing genre), then evaluating the compatibility of both (determining the user might like the song), and finally... <rate>A predicted rating is generated under the tag. The entire process ensures the depth of reasoning and the accuracy of the results, and the final music list will be closer to the user's actual preferences, improving the personalization of recommendations and user satisfaction.

[0082] The step design of this application embodiment guides the model to generate high-quality inference trajectories through structured prompts and reinforcement learning training, which greatly improves the accuracy of the recommendation system in rating prediction, while strengthening the model's understanding and response to personalized needs, and realizing deep personalization and efficient matching of recommended content.

[0083] In the above embodiments of this application, an object recommendation model is used to perform multiple inference tasks on preference information and object information to generate an inference trajectory, including: using the object recommendation model to perform a preference information extraction task on the preference information to obtain preference features corresponding to the preference information extraction task; using the object recommendation model to perform an object information extraction task on the object information to obtain object features corresponding to the object information extraction task; using the object recommendation model to perform a matching degree estimation task on the preference features and object features to obtain a matching degree corresponding to the matching degree estimation task; and using the object recommendation model to perform a recommendation score prediction task on the matching degree to obtain a recommendation score corresponding to the recommendation score prediction task, wherein the recommendation score is used to characterize the degree of matching between the object to be recommended and the user.

[0084] The aforementioned preference information extraction task aims to extract preference features from user behavior data; in this embodiment, it specifically refers to user interests. This task requires the object recommendation model to analyze user historical interactions and identify user preferences for different attributes. This process relies on the model's semantic understanding and reasoning capabilities.

[0085] The aforementioned object information extraction task focuses on extracting object features from the metadata of the objects to be recommended. In this embodiment, the target item features cover key information such as item attributes, description, category, and user feedback, ensuring that the model has a comprehensive view of the items when evaluating the matching degree.

[0086] The matching assessment task described above is the process of evaluating the fit between preference features and object features. IRM compares user preferences with item attributes to determine the potential matching degree between users and items, providing a quantitative basis for recommendation decisions.

[0087] The aforementioned recommendation score prediction task is used to generate predictions of the user's possible ratings. This score directly reflects the degree of matching between the recommended object and the user, and is an important reference for the recommendation system to make recommendations.

[0088] The object recommendation model first deeply analyzes preference information, using natural language processing (NLP) techniques to identify and extract user behavior patterns, preference settings, and sentiment tendencies to generate preference features. This technical step ensures that the model can accurately capture users' personalized needs and interests, laying the foundation for subsequent matching analysis. Next, the model performs a detailed analysis of the information for each object to be recommended, again employing NLP techniques to extract object features from multiple dimensions such as item descriptions, attribute tags, and user reviews. The purpose of this step is to build a comprehensive and detailed item profile, facilitating subsequent in-depth comparison and matching with preference features.

[0089] The model then compares and analyzes preference features with object features, assessing their compatibility and similarity to generate a matching score. This matching score calculation is performed through reinforcement learning training of the IRM for recommendation scenarios, enabling it to directly ensure a high degree of match between recommended objects and user preferences based on text. The recommendation score prediction task is based on the matching score; the model performs a prediction task to generate a recommendation score, which intuitively reflects the degree of match between the recommended object and the user. Recommendation score prediction builds upon the previous inference steps; by analyzing the matching score, the model can accurately predict the user's potential preference level for each candidate object, thus making accurate recommendation decisions.

[0090] Through the execution of the aforementioned technical actions, this embodiment of the application achieves a deep understanding and precise matching of user personalized needs and the characteristics of the objects to be recommended. The system generates a series of reasoning trajectories through detailed analysis of preference and object information. These trajectories contain rich insights and understandings, directly related to the degree of matching between preference features and object features. The final generated recommendation score serves as the basis for recommendation decisions, ensuring that recommended content is not only based on user historical behavior but also delves into a deeper understanding of user interests and a refined mining of object attributes, thereby improving the personalization level and predictive accuracy of recommendations. This process fully utilizes the reasoning capabilities of the object recommendation model, enhancing the intelligence and efficiency of the recommendation system, providing users with more relevant recommendations, and improving user satisfaction and the overall performance of the recommendation system.

[0091] This application employs an IRM (Independent Reasoning Recognition) self-learning approach. Through interactive sampling and rule-based rewards, the model can self-improve its reasoning trajectory, jointly enhancing the accuracy of user analysis, item analysis, matching degree evaluation, and recommendation score prediction. Compared to distillation methods that rely on teacher models, this design allows the model to gain a deeper understanding of the recommendation scenario, internalizing reasoning skills and generating high-quality reasoning trajectories and predicted scores even without external supervision.

[0092] This application's embodiments, through the aforementioned steps, achieve accurate capture and matching degree evaluation of user preferences and item features, thereby generating recommendation scores. This process not only improves the personalization and accuracy of recommendations but also promotes the self-evolution of the model's reasoning ability, reducing reliance on expensive labeled data. Especially in rating prediction scenarios, the model can make more reasonable recommendations that are closer to real preferences based on the inherent relationship between user interests and item features. This innovative design effectively solves the problems of expensive and static supervision signals and superficial reasoning transfer in existing methods, greatly enhancing the accuracy of rating prediction and the efficiency of the recommendation system.

[0093] In the above embodiments of this application, determining a target object from multiple objects to be recommended based on their respective inference trajectories includes: determining the recommendation score corresponding to each object to be recommended from its respective inference trajectories; sorting the multiple objects to be recommended based on their respective recommendation scores to obtain a sorted object sequence; and obtaining the top-ranked object to be recommended from the sorted object sequence to obtain the target object.

[0094] The aforementioned "objects to be recommended" refer to multiple candidate items or content that may be recommended to the user in a recommendation system. These objects may include products, music, articles, etc., depending on the application area of ​​the recommendation system.

[0095] The aforementioned inference trajectory is a logical sequence generated by the object recommendation model in this embodiment when performing inference tasks for each object to be recommended. It encompasses the entire process from preference information extraction to recommendation score prediction. As a record of the model's inference process, the inference trajectory includes not only preference features and object features, but also the results of matching degree evaluation and recommendation score prediction.

[0096] The recommendation score mentioned above is a quantitative representation of the degree of matching between each recommended object and the user, as determined by the object recommendation model. This score is derived from the model's understanding of user preferences and analysis of item attributes, and is used for subsequent ranking and selection of recommended objects.

[0097] In one optional embodiment, a "recommendation score" is determined for each "object to be recommended" within its corresponding "inference trajectory." This step is achieved by parsing the recommendation score prediction task execution results in the inference trajectory, i.e., a quantitative evaluation of the user's matching degree for each object during the inference process. Based on the "recommendation scores" corresponding to multiple "objects to be recommended," these objects are sorted to obtain a "sorted object sequence." This sorting is typically done in descending order, meaning objects with higher recommendation scores are ranked higher in the sequence, providing a basis for subsequent target object selection. The "top-ranked" "objects to be recommended" are obtained from the "sorted object sequence," and these objects are identified as "target objects." This process may select a certain number or proportion of objects based on the specific application scenario and recommendation strategy to ensure that the objects recommended to the user match their interests and preferences.

[0098] This process in the embodiments of this application plays a central role in the rating prediction task. Through the joint execution of multiple inference tasks, the model generates an inference trajectory, which not only includes in-depth analysis of user and item features but also calculates the ratings that users might give to items. These ratings, as "recommendation scores," play a decisive role in subsequent ranking and target object determination. Compared with traditional recommendation methods that rely solely on historical interaction data, this strategy has the advantage of combining the inference capabilities of a large model with the improvement mechanism of reinforcement learning, thereby enhancing the personalization and accuracy of recommendations, especially in scenarios such as rating prediction that require a deep understanding of the relationship between users and items.

[0099] The steps in this embodiment of the application, by converting the reasoning results into recommendation scores, ranking candidate objects according to the scores, and finally selecting the object with the higher score as the recommendation, effectively realize personalized recommendation decision-making based on reasoning. This process not only improves the accuracy and relevance of recommendations, but also continuously improves the model's performance in rating prediction tasks through the self-improvement mechanism of reinforcement learning, enabling the recommendation system to better adapt to changes in user preferences and provide higher-quality recommendations.

[0100] For example, in a music recommendation scenario, the system first extracts preferences from users' historical ratings, then analyzes song metadata to generate inference trajectories. Each trajectory contains the user's potential evaluation of the song. Subsequently, the system ranks candidate songs based on the recommendation scores in the inference trajectories. For instance, user A's preferences are summarized as a high tendency to rate "soft melodies" and "classical music," and song B's metadata indicates that it possesses these characteristics. When song B's recommendation score is calculated and compared with the recommendation scores of other candidate songs, if its score is higher, song B will be listed as the "target" and ultimately pushed to user A in order to obtain positive feedback. This process ensures that recommendation decisions are based on deep inference and quantitative evaluation, improving the matching degree between recommended content and user interests.

[0101] In the above embodiments of this application, the method further includes: outputting an inference trajectory to the user; responding to receiving a target recommendation score from the user; evaluating the inference trajectory and the target recommendation score based on a preset reward rule to obtain a total reward value; and updating the model parameters of the object recommendation model based on the total reward value.

[0102] The target recommendation score mentioned above is a rating given by the user directly after receiving the inference trajectory. This feedback serves as a key signal in the reinforcement learning mechanism, guiding the adjustment of model parameters and the improvement of the strategy.

[0103] The aforementioned preset reward rules, defined in this embodiment, are a rule system used to quantify the quality of inference trajectories and the accuracy of target recommendation scores. The preset reward rules combine user feedback with desired goals, determining the reward value for each step of inference through a series of calculations, thereby affecting the model's overall reward evaluation.

[0104] The Total Reward Value (TRV) mentioned above is calculated according to a preset reward rule and is used to measure the overall performance of the model when handling a specific task (such as user feedback processing). TRV comprehensively considers the quality of the inference trajectory and the consistency between the target recommendation score and the user feedback score, and is a key indicator to guide the updating of model parameters.

[0105] In this embodiment, once the object recommendation model generates an inference trajectory, the system first displays the trajectory to the user. The user provides direct feedback on the recommended objects based on the analysis and interpretation of the objects within the inference trajectory. Next, the system evaluates the user's feedback recommendation score against the inference trajectory according to preset reward rules, aiming to quantify the quality of the inference trajectory and the accuracy of the recommendation score. This evaluation process may involve various calculations, such as comparing the deviation between user feedback and model predictions, evaluating the correctness of the inference trajectory's format and the rationality of its content, and comprehensively considering the consistency between the inference results and the user's actual behavior. Based on the above evaluation results, the system calculates the total reward value. The TRV directly reflects the degree to which the understanding of the inference process and recommendation decision satisfies the user's preferences and serves as the basis for subsequent model parameter adjustments. Finally, the system updates the model parameters of the object recommendation model based on the total reward value. This update is achieved by adjusting the weight matrix and neural network structure within the model. The parameter update follows the principles of reinforcement learning, aiming to improve the model's performance in subsequent inference tasks, more accurately predict the user's feedback recommendation score, thereby enhancing the personalization level of recommendations and user satisfaction.

[0106] The "preset reward rule" in this embodiment designs a rule-based reward mechanism that ensures that even in the absence of explicitly labeled data, the model can autonomously learn and improve its reasoning ability through user feedback. This mechanism not only solves the problem of expensive and static supervision signals, but also promotes the internalization of the model's reasoning skills and enhances the dynamic adaptability of the recommendation system.

[0107] The steps in this embodiment of the application are designed to form a closed-loop reinforcement learning mechanism by displaying the reasoning trajectory to the user, collecting user feedback, evaluating the model performance based on preset reward rules, and finally updating the model parameters. This mechanism not only enhances the model's self-learning and improvement capabilities but also ensures that recommendation decisions closely align with the user's true preferences, thereby improving the personalization level of the recommendation system and user satisfaction. Particularly in rating prediction scenarios, this mechanism guides the model to continuously adjust its reasoning strategy to more accurately predict the ratings that users might give, thus significantly improving recommendation accuracy and the positivity of user feedback.

[0108] For example, in a music recommendation scenario, the system displays a reasoning trajectory for a classical jazz piece to the user, analyzing in detail the user's preference for light melodies and classical music, as well as the stylistic characteristics of the piece. Based on this trajectory, the user directly provides their rating for the recommended song (i.e., the target recommendation score provided by the user). The system then calculates the total reward value based on preset reward rules, considering the correctness of the reasoning trajectory's format, the reasonableness of its content, and the consistency between the recommendation score and the user's feedback rating. Based on this reward evaluation, the system updates the internal parameters of the object recommendation model, thereby improving the model's performance in predicting user music ratings and its reasoning ability. This process ensures that the recommendation system not only generates recommendations that match user interests but also can self-improve based on specific user feedback, improving the accuracy of recommendations and user acceptance.

[0109] In the above embodiments of this application, the preset reward rules include: format reward rules and scoring reward rules; based on the preset reward rules, reward evaluation is performed on the inference trajectory and the target recommendation score to obtain a total reward value, including: based on the format reward rules, reward evaluation is performed on the output format of the inference trajectory to obtain a format reward value, wherein the format reward value is used to characterize whether the output format of the inference trajectory is correct; based on the scoring reward rules and the target recommendation score, reward evaluation is performed on the recommendation score in the inference trajectory to obtain a scoring reward value, wherein the scoring reward value is used to characterize the deviation between the recommendation score and the target recommendation score; the format reward value and the scoring reward value are summarized to obtain a total reward value.

[0110] The aforementioned format reward rules are a set of rules used in this application embodiment to judge whether the output format of the inference trajectory is correct. The format reward rules ensure that the inference trajectory follows a predefined structure and syntax, such as requiring "user interest extraction", "item key point summary", "matching degree evaluation" and "final score" to be placed between specific labels.

[0111] The aforementioned rating and reward rules are used to quantify the deviation between the recommended score and the target recommended score (i.e., the actual rating given by the user). The design of the rating and reward rules aims to guide the model to accurately predict the recommended score given by the user, reduce prediction errors, and improve the overall performance of the recommendation system.

[0112] In this embodiment, the system first evaluates the output format of the inference trajectory according to the format reward rule. If the inference trajectory follows a predefined structure, i.e., the output of each inference task is located within the corresponding label, and the final predicted score format is correct, the system will provide a Format Reward Value (FRV). The size of the FRV reflects the format standardization of the inference trajectory, ensuring the readability and compliance of the output. Subsequently, the system applies the rating reward rule to evaluate the recommended score in the inference trajectory. This evaluation is based on the difference between the predicted recommended score and the target recommended score reported by the user, calculating the Rating Reward Value (RRV). The calculation of RRV aims to quantify the accuracy of the model's prediction, encouraging the model to reduce prediction errors and improve the accuracy of rating predictions. Finally, the system summarizes the FRV and RRV to generate the Total Reward Value (TRV). The TRV comprehensively considers the format correctness of the inference trajectory and the accuracy of the recommended score, providing clear guidance for updating model parameters, ensuring that the model can continuously improve its rating prediction capabilities and enhance user satisfaction while maintaining the standardization of the output format.

[0113] The combined use of format-based reward rules and scoring reward rules constitutes the core of the "rule-based reward modeling" method in this application's embodiments. This method not only ensures the correctness of the format and the clarity of the structure of the model's output inference trajectory, but also guides the model to improve itself with the goal of reducing recommendation score errors. Compared to relying solely on supervised fine-tuning or general reward mechanisms, the rule-based reward modeling mechanism in this application's embodiments is more refined and can directly address the specific needs of the recommendation system, improving the model's performance in scoring prediction scenarios.

[0114] This application's embodiments introduce format reward rules and rating reward rules to evaluate the format of the inference trajectory and the accuracy of the recommendation score, achieving fine-grained control over the model's inference process and a significant improvement in rating prediction capabilities. This mechanism ensures that the model output is not only formatted correctly, easy to understand and verify, but also that its rating predictions more closely match users' real feedback. By guiding the update of model parameters through the total reward value, the recommendation system's performance on the rating prediction task is continuously improved, enhancing the user experience and the personalization level of recommendations.

[0115] For example, in a music recommendation scenario, suppose a user prefers light melodies and classical music, and the song to be recommended possesses these characteristics. The object recommendation model generates an inference trajectory, which includes analysis of user interests, summarization of song features, evaluation of matching degree, and prediction of the final recommendation score. The system first checks whether the inference trajectory follows a predefined structure, such as whether the user's preferences are located in...<analyze_user> "and"< / analyze_user> Between " and ", are the characteristics of the song in "<analyze_item> "and"< / analyze_item> "If the format is correct, a higher FRV is assigned. Then, the system compares the recommended score with the target recommended score reported by the user, and calculates the RRV based on the score difference. If the model's predicted score is close to the user's actual score, the RRV will also be high. Finally, the system summarizes the FRV and RRV to obtain the TRV. This TRV will serve as the basis for updating the model parameters, guiding the model to improve its score prediction ability while maintaining the correct format, thereby more accurately matching user preferences when recommending songs and improving recommendation satisfaction."

[0116] In the above embodiments of this application, the recommended score in the inference trajectory is evaluated for reward based on the scoring reward rules and the target recommended score to obtain a scoring reward value, including: obtaining the absolute value of the difference between the target recommended score and the recommended score to obtain the target deviation; and determining the scoring reward value based on the target deviation and the preset error.

[0117] The aforementioned target bias refers to the absolute value of the difference between the target recommendation score and the recommended score in this embodiment. This bias measures the gap between the recommendation score predicted by the model and the target recommendation score actually provided by the user, and is an important indicator for evaluating the accuracy of model predictions.

[0118] In this embodiment of the application, the aforementioned preset error specifically refers to the allowable error. An upper limit can be set for the acceptable scoring prediction deviation of the system to evaluate whether the model's prediction performance is within an acceptable range.

[0119] In this embodiment, the system first calculates the absolute value of the difference between the target recommendation score and the recommended score, obtaining the target deviation (TD). This calculation quantifies the actual deviation between the predicted recommendation score and user feedback, providing a basis for subsequent Rating Reward Value (RRV) calculation. Subsequently, the system determines the Rating Reward Value (RRV) based on TD and a preset allowable error (MAE). This determination process typically involves calculating the relative relationship between TD and MAE to determine whether the model's predicted rating deviation is within a reasonable range. Specifically, the RRV calculation employs a linear or nonlinear mapping to transform TD into a reward or penalty signal, aiming to incentivize the model to reduce prediction errors and improve the accuracy of rating predictions.

[0120] This application refines the calculation method for the score reward value. By quantifying the deviation between the prediction and the actual feedback and comparing it with the allowable error, the system can provide more accurate feedback on the model's performance in the score prediction task. This mechanism ensures that the update of model parameters not only focuses on the absolute accuracy of the score prediction but also considers the relative importance of the deviation, thereby enabling fair and effective comparison and improvement among multiple models or strategies.

[0121] By implementing the above steps, the strategy in this embodiment can provide a refined evaluation of the model's performance on the rating prediction task. The system directly quantifies the error in rating prediction by calculating the target deviation, and the generated rating reward value, based on the comparison between the target deviation and the allowable error, guides the update of the model parameters. This process not only improves the accuracy of rating prediction but also ensures that the model can continuously improve itself, getting closer to the user's true rating feedback with smaller deviations, thereby improving the overall performance of the recommendation system and user satisfaction.

[0122] For example, in a practical application, suppose a user rates a song 5 out of 5, while the object recommendation model predicts a recommendation score of 4.5. First, the system calculates the target deviation, i.e., the absolute difference between 5 and 4.5, resulting in a target deviation (TD) of 0.5. Then, the system determines the rating reward value by referring to a preset allowable error—for example, MAE is set to 1.0. If TD is less than MAE, it means the model's prediction error is within an acceptable range, and the system will award a higher rating reward value; if TD exceeds MAE, the reward value will be reduced accordingly, thus encouraging the model to reduce prediction deviation. This mechanism ensures that even if there is a slight deviation between the model's prediction and the user's feedback, as long as it is within the allowable error range, the model can still receive positive incentives, thereby continuously adjusting its strategy in subsequent predictions to more accurately reflect user preferences and improve recommendation quality.

[0123] In the above embodiments of this application, updating the model parameters of the object recommendation model based on the total reward value includes: generating the objective function value corresponding to the object recommendation model based on the total reward value and the inference trajectory; and updating the model parameters of the object recommendation model based on the objective function value.

[0124] The aforementioned objective function value refers to a numerical value calculated based on the total reward value and inference trajectory, used to guide the learning and parameter update process of the object recommendation model. It comprehensively considers both the reward feedback of the model's current behavior and the policy value of that behavior, serving as the core basis for improving model parameters.

[0125] In this embodiment, the system generates the objective function value corresponding to the object recommendation model based on the total reward value and the inference trajectory. This transformation process integrates the total reward value and the inference trajectory into the objective function, converting them into measurable variables to evaluate the behavioral quality of the object recommendation model and its contribution to the recommendation task. The total reward value directly reflects the model's performance in the current inference task, including an evaluation of the inference trajectory format and quantitative feedback on the accuracy of score prediction; while the inference trajectory carries the logical flow and contextual information of the inference, helping the system understand how the model handles specific recommendation scenarios. By combining these two to generate the objective function value, the system provides direction for updating the object recommendation model parameters, ensuring that the update is based on a comprehensive evaluation of the entire inference process, not just a single prediction result. Subsequently, the system updates the model parameters of the object recommendation model based on the objective function value.

[0126] Based on the calculated objective function value, the system updates the model parameters of the object recommendation model. This process follows the basic principles of reinforcement learning, which is to improve future performance by adjusting the model's internal parameters. The system aims to enable the object recommendation model to generate higher total reward values ​​and better inference trajectories when encountering similar inference tasks in the future. This means the model will more accurately predict users' ratings of recommended objects, and the inference process will be more consistent with recommendation logic, thereby effectively improving the personalized recommendation effect and overall performance of the recommendation system.

[0127] The objective function value generation and model parameter update mechanism in this application embodiment significantly enhances the strategic guidance for model learning. Unlike simply relying on the reward value of a single inference trajectory for parameter updates, the inter-group relative strategy improvement adopts a more comprehensive improvement strategy, adjusting parameters by evaluating the relative performance of the object recommendation model when processing samples within the same group. This mechanism not only focuses on the quality of individual inference trajectories but also considers the overall performance of the object recommendation model when processing a group of samples, thereby ensuring the robustness and efficiency of the parameter update direction, preventing the object recommendation model from falling into over-improvement of a single inference scenario, and avoiding the local good solution trap that the model may encounter during the learning process.

[0128] Through the aforementioned mechanism, the object recommendation model in this embodiment can continuously improve its reasoning ability through multiple iterations, more accurately predict user ratings, and generate more reasonable reasoning trajectories, thereby significantly improving the performance of the rating prediction task. This effect stems from the system's comprehensive evaluation of model behavior and fine-tuning of parameters, ensuring that the model not only handles single recommendation cases well but also performs excellently when processing the entire recommendation set, enhancing the personalization level and user satisfaction of the recommendation system.

[0129] Currently, improving recommender systems by leveraging the world knowledge and reasoning capabilities of models has led to a new paradigm of "generative reasoning + distillation fine-tuning." However, this paradigm suffers from limitations such as limited teacher capabilities, expensive and static supervision signals, and superficial reasoning transfer. To address these issues, this application proposes a Pure Reinforcement Learning (PRL) framework. This framework uses interactive sampling and rule-based rewards to learn reasoning trajectories, jointly improving user analysis, item analysis, matching, and rating. Based on this, a supervised fine-tuning (SFT) cold start is first performed using high-quality trajectories, followed by reinforcement learning (RL) training to accelerate convergence and improve robustness.

[0130] To address the main challenges of existing reasoning-enhanced rating prediction methods, this application proposes a reasoning enhancement module. This approach employs pure reinforcement learning, enabling the model to jointly improve four key processes—user analysis, item analysis, matching analysis, and rating—under a unified recommendation objective. To further enhance recommendation task performance, a hybrid enhancement module is proposed based on the reasoning enhancement module. This hybrid module acquires high-quality inference trajectories through a specialized sampling strategy for supervised fine-tuning (SFT) to cold-start the initial model, followed by task-specific RL training, resulting in faster model convergence and more robust recommendations. Figure 3 This is a schematic diagram of a recommended process according to an embodiment of this application, including an inference enhancement module and an enhancement hybrid module.

[0131] like Figure 3 As shown, in the inference enhancement module, the model receives input information. The User Interest Extraction module analyzes the user's tendencies and preferences for various attributes based on historical user data and target item information through an internal inference mechanism. The Summarize Item Key Aspects module focuses on the characteristics of the target item, extracting and summarizing keywords and features relevant to its rating prediction. The Evaluate Compatibility module, after understanding user interests and item characteristics, performs in-depth analysis to determine the degree of matching between the two—a crucial step before rating prediction. The Rating Prediction module provides a rating prediction for the user and a specific item, based on the preceding comprehensive analysis. Rule-based reward modeling calculates rewards according to a series of pre-defined rules after rating prediction. These rules aim to ensure the correct format of the inference trajectory and encourage the model to predict ratings close to reality. Inter-group relative policy improvement allows the model to learn based on the relative performance of its predictive behavior by adjusting and improving the policy, thereby improving the accuracy and efficiency of rating prediction.

[0132] like Figure 3 As shown, in the enhanced hybrid module, the Teacher Model is typically a model with strong reasoning ability, used to generate high-quality reasoning trajectories and initial rating predictions. The Student Model performs supervised learning (SFT) based on the data generated by the Teacher Model, initially establishing the foundation for recommendation reasoning. During the cold start phase, the model undergoes supervised fine-tuning using sample data with detailed reasoning processes, quickly acquiring basic recommendation and reasoning abilities. The Reinforcement Learning Refinement (RLRefinement) module further improves its recommendation reasoning skills through interactive sampling and rule-based rewards, achieving efficient learning and higher prediction accuracy.

[0133] The output of the enhanced hybrid module can be seen as a refined version trained by reinforcement learning, capable of independently completing the entire process from user analysis to rating prediction. The Hybrid Paradigm combines the fast start-up advantage of cold-start supervised fine-tuning with the deep improvement characteristics of RL training, aiming to accelerate model convergence and improve its robust performance in recommendation scenarios.

[0134] Through the above process and module combination, the inference enhancement module and the enhancement hybrid module work together to overcome the limitations of the traditional "teacher-student distillation" method, effectively improving the model's inference and rating prediction capabilities. The enhancement hybrid module and the inference enhancement module can be trained according to the enhancement hybrid module pipeline. By combining two stages—cold-start supervised fine-tuning and reinforcement learning refinement—the enhancement hybrid module pipeline aims to integrate the inference knowledge of the teacher model with the adaptive learning ability of the student model, thereby constructing a recommendation system model that is more accurate in rating prediction and has stronger generalization ability. This method not only accelerates the convergence during model training but also improves the model's adaptability to changing user preferences and the robustness of recommendation performance.

[0135] In reasoning-enhanced recommendation scenarios, requiring the model to explicitly extract user preferences and item features before inferring the matching degree between users and items can significantly improve the prediction accuracy of true ratings. High-quality rating predictions rely on a clear and accurate extraction process, but user preferences and item features are often difficult to quantify. To address this, this application forces the model to output user preferences and item features according to predefined rules and adopts a rule-based reward mechanism to guide the model to improve itself without the need for labeled data, thereby replacing the previous "teacher-student distillation" approach.

[0136] We designed global prompts to guide the model in generating inference trajectories beneficial for rating prediction. The inference process is divided into three steps: user interest extraction, item key point summarization, and matching degree evaluation. Given user history and item metadata, the model's different steps must output within specified token boundaries. In addition to the regular input (user history + target item), a structured inference template is introduced, decomposing the task with specific labels:<analyze user>< / analyze user> Extracting user interests;<analyze item>< / analyze item> Summarize the characteristics of the target item; <match>< / match> "Assess the match rate;" <rate>< / rate> Give the final score.

[0137] The difference between this application and ordinary reinforcement learning is that it does not require an external reward model to provide scores, and it improves both inference and score prediction based on rules. Unlike reinforcement learning based on thought processes, such as conventional GRPO, it introduces structured reasoning for recommendation scenarios. This prior effectively improves the interpretability of recommendations and prediction accuracy, preventing the reasoning process from becoming meaningless as reinforcement learning progresses. The reward rules are improved for recommendation scenarios, allowing reinforcement learning to directly improve the target based on the MAE metric, significantly enhancing learning performance. Ultimately, multi-step structured reasoning to score prediction is improved together under this rule. This application introduces a new sampling paradigm to obtain high-quality inference trajectories for cold start, followed by reinforcement learning with an inference enhancement module to further improve the model's upper limit of capability.

[0138] Rule-based methods can provide stable training rewards for RL. To avoid performance deficiencies caused by inconsistent goals among sub-modules in the past, "user preference summarization, item feature extraction, and matching inference" can be integrated into the same framework and improved using a unified reward rule.

[0139] First, define the criteria for correct formatting; the thought process must be encapsulated separately in...<analyze user> …< / analyze user> ,<analyze item> …< / analyze item> as well as <match> …< / match> Within the tags, the final output is located in <rate> …< / rate> The formatting reward is defined by the following formula: (The text within the `` tag must not contain unreadable content.)

[0140]

[0141] in, This is the format reward value, used to evaluate whether the inference trajectory generated by the model follows a predefined structured template. The purpose of the format reward is to ensure that the model outputs information in a specified format during inference, such as within a specific label.<analyze user> "、"<analyze item> "、" <match>"and" <rate>The system performs user preference analysis, summarizes item characteristics, assesses matching accuracy, and predicts final scores.

[0142] The 0.5 mentioned above is a positive reward value if the inference trajectory generated by the model is in the correct format (i.e., it follows the predefined label structure). This positive reward encourages the model to continue maintaining the correct format during subsequent training.

[0143] The -0.5 mentioned above is a negative reward value set to -0.5 if the inference trajectory is incorrectly formatted, i.e., it does not follow the predefined label structure. This punitive reward incentivizes the model to avoid similar formatting errors in subsequent attempts.

[0144] During training, following the continuous rating paradigm proposed in previous work, rule-based rewards can be used to encourage the model to approximate the user's true rating as closely as possible, rather than simply pursuing a "perfectly correct" reward. This objective is formalized by the following formula:

[0145]

[0146] in, As a rating reward, This represents the user's true rating of the item. The score predicted by the model. This represents the allowable error in score prediction.

[0147] The final total reward can be expressed by the following formula:

[0148] ;

[0149] in, For the total reward, For format rewards, Rewards are given for ratings.

[0150] This application employs Group Relative Policy Optimization (GRPO) for policy updates. For input x, G copies of the output are sampled from the old policy, and the objective function is expressed by the following formula:

[0151]

[0152] in, It is the update target of the model during the policy update process, aiming to improve the model's performance on G sampled outputs with different inputs x. This represents each pair of inputs x and a set of predictions in dataset D. Expected value This represents the old policy, that is, the policy before the update, i.e., the behavioral guidelines of the reinforcement learning model before the parameter update. Indicating a new strategy, This can be used as a reference strategy. It can be used for reinforcement learning. The parameter value is clip(·), which is the pruning function. This represents the output value for input data i at the t-th time step or iteration, where β represents the parameter.

[0153] While pure RL methods for inference enhancement modules have shown promise, there is a domain gap between the corpus of pre-trained models and recommendation data. Therefore, this application proposes a hybrid paradigm enhancement hybrid module, which first utilizes high-quality inference trajectories for cold-start SFT, and then uses RL-based fine-tuning inference within the inference enhancement module framework to further improve the upper limit of capabilities.

[0154] The initial stage of enhancing the hybrid module focuses on laying the foundation for the basic reasoning capabilities required for effective recommendations. We first perform a cold start on recommendation data containing long inference trajectories: we sample M data points from the complete dataset D to form a subset (which will also be used in subsequent inference enhancement modules). For each sample, we call the teacher model with stronger reasoning capabilities to generate an inference trajectory. If the predicted score ŷ matches the true y, it is considered high quality; if not, (x, y) are input into the teacher model, requiring it to generate a rationalized trajectory consistent with the correct score. Ultimately, two complementary datasets are obtained, represented by the following formula:

[0155]

[0156] in, This represents the total recommendation dataset. The sample includes examples where teachers can arrive at correct scores through natural reasoning. Then, the samples that require correction of reasoning to arrive at the correct conclusion are recorded. This indicates the corrected trajectory. This indicates the corrected output value.

[0157] The training objective of this application is to improve the autoregressive model, specifically expressed by the following formula:

[0158]

[0159] in, This represents a set of parameters that need to be found. It includes inference trajectory examples that need to be considered during model training. for One example, in which, For the output of the model, The inference trajectory generated for the teacher model is the target that the model attempts to learn and predict. This indicates summing over each time point or step in the reasoning trajectory.

[0160] In this application, the reasoning enhancement module abandons the "teacher-student" multi-model, multi-stage distillation process (i.e., the teacher model is responsible for information extraction and reasoning generation, while the student model undergoes supervised fine-tuning). Instead, it directly trains a single model through pure reinforcement learning, enabling it to autonomously acquire reasoning abilities for rating prediction. Unlike distillation methods, which are prone to overfitting to surface-level reasoning trajectories, the reasoning enhancement module allows the model to truly learn reasoning abilities for recommendation scenarios through interaction and improvement.

[0161] This application draws inspiration from the "cold-start RL" concept and proposes a hybrid paradigm to enhance the hybrid module. The enhanced hybrid module first constructs a small number of cold-start inference samples for the recommendation task, then performs supervised fine-tuning of the model to obtain initial capabilities; subsequently, RL is used to further improve the inference capabilities. By combining SFT cold-start based on a specific sampling strategy with task-specific RL, the enhanced hybrid module aims to achieve faster convergence and stronger recommendation inference performance.

[0162] This application significantly improves performance on the rating prediction task compared to previous methods. Taking a dataset as an example, this application adapts reinforcement learning fine-tuning to the large-scale recommendation domain, a method rarely explored previously. Compared to previously created supervised fine-tuning methods, this application eliminates the dependence on the teacher model, resulting in a significant performance improvement. The inference enhancement module abandons the "teacher-student" multi-model, multi-stage distillation process (i.e., the teacher model is responsible for information extraction and inference generation, while the student model undergoes supervised fine-tuning), instead directly training a single model through pure reinforcement learning, enabling it to autonomously acquire inference capabilities for rating prediction. Furthermore, drawing on the idea of ​​"cold-start RL," a hybrid paradigm enhancement module is proposed. The enhancement module first constructs a small number of cold-start inference samples for the recommendation task, performs supervised fine-tuning on the model to obtain initial capabilities, and then further improves the inference capabilities using RL. By combining SFT cold-start based on a specific sampling strategy with task-specific RL, the enhancement module aims to achieve faster convergence and stronger recommendation inference performance.

[0163] For the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions. This is because, according to this application, certain steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0164] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms, or by hardware. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods of the various embodiments of this application.

[0165] Figure 4 This is a flowchart of a training method for an object recommendation model according to an embodiment of this application, as shown below. Figure 4 As shown, the method may include the following steps:

[0166] Step S402: Obtain training data.

[0167] The training data includes: training preference information, training object information, and preset recommendation scores.

[0168] The training data mentioned above is a dataset used to train and improve the model. It contains training preference information, training object information, and preset recommendation scores. It is an important source of information for model learning during reinforcement learning and is used to guide the adjustment of model strategies.

[0169] The aforementioned training preference information refers to information about users' historical preferences, behavioral patterns, and personal characteristics found in the training data. This information is used by the model to understand and learn users' behavioral patterns and preferences.

[0170] The training object information mentioned above includes a detailed description of the recommended object, its attributes, and relevant data on the user's historical interactions. This helps the model understand the characteristics of the recommended object so as to more accurately predict the user's possible ratings for these objects.

[0171] The aforementioned preset recommendation scores are set based on historical data or expert knowledge and serve as a reference standard for model prediction scoring. They are used to evaluate the accuracy of model prediction results and guide the reward mechanism in the reinforcement learning process.

[0172] In this embodiment of the application, training preference information and training object information, including user historical behavior, preference information and detailed attributes of recommended objects, can be collected. At the same time, a series of preset recommendation scores are set as guidance and evaluation criteria for model learning.

[0173] Step S404: Use the initial recommendation model to perform multiple inference tasks on the training preference information and training object information to generate training inference trajectories.

[0174] The training inference trajectory includes the training execution results corresponding to multiple inference tasks, which include: preference information extraction task, object information extraction task, matching degree evaluation task, and recommendation score prediction task.

[0175] The initial recommendation model can be used to perform multiple inference tasks on the aforementioned training preference information and training object information to generate training inference trajectories. This process involves the model performing tasks such as preference information extraction, object information extraction, matching degree evaluation, and recommendation score prediction. Through this series of inference tasks, the model can generate training inference trajectories containing multiple inference results, providing rich behavioral samples for subsequent training.

[0176] Step S406: Based on the training inference trajectory and the preset recommendation score, perform reinforcement learning on the initial recommendation model to obtain the object recommendation model.

[0177] The object recommendation model is used to execute any of the methods in the above embodiments.

[0178] Based on the training inference trajectory and preset recommendation scores, reinforcement learning is performed on the initial recommendation model to obtain an object recommendation model. This process is the core of the embodiments of this application. By analyzing the model's performance in the training inference trajectory and combining it with the preset recommendation scores, the system can calculate the reward value for each step of inference, guide the adjustment of model parameters, and thus improve the model's recommendation ability.

[0179] The training of the model in this embodiment is not only based on data-driven supervised learning, but also incorporates deep training with reinforcement learning mechanisms. By comparing the recommended scores predicted by the model with the preset recommended scores, and by evaluating the structure and quality of the inference trajectory, the model in this embodiment can autonomously learn how to improve its inference strategy to more accurately predict users' ratings of recommended objects, thereby improving the personalized recommendation effect and overall performance of the recommendation system.

[0180] Through the above steps, the embodiments of this application enable the model to self-improve in the rating prediction task, that is, the model continuously adjusts its reasoning strategy during the learning process to improve the accuracy of the predicted score. This mechanism not only improves the personalized recommendation level of the recommendation system, but also enhances the model's reasoning ability and generalization ability, thereby significantly improving the model's performance in rating prediction scenarios.

[0181] For example, in a music recommendation scenario, the system collects a large amount of user listening records, preference information, and song metadata as training data, and sets a preset recommendation score based on historical rating data. During training, the system first uses the initial recommendation model to process training preference information and training object information, performing inference tasks such as user preference extraction, song feature analysis, user-song matching evaluation, and recommendation score prediction, generating an inference trajectory for each user and song. Subsequently, based on these inference trajectories and the preset recommendation score, the system adjusts the parameters of the initial recommendation model through a reinforcement learning mechanism to obtain the final object recommendation model. This model performs better in performing the aforementioned inference tasks, more accurately predicting users' ratings of recommended songs, thereby improving the personalization level of music recommendations, providing users with recommendation results that better match their musical tastes, and improving user satisfaction and the overall performance of the recommendation system.

[0182] Through the above steps, training data is obtained, including training preference information, training object information, and preset recommendation scores. The initial recommendation model is then used to perform multiple inference tasks on the training preference information and training object information, generating training inference trajectories. These training inference trajectories include the training execution results for each of the multiple inference tasks, which include preference information extraction, object information extraction, matching degree evaluation, and recommendation score prediction. Based on the training inference trajectories and preset recommendation scores, the initial recommendation model is subjected to reinforcement learning to obtain an object recommendation model. By comparing the inference trajectories of each object to be recommended, especially the execution results of its recommendation score prediction task, the model can identify objects with higher recommendation scores, which are then considered the target objects. This decision-making process is based on the matching degree between user preferences and object characteristics reflected in the inference trajectories, as well as the deep understanding of the recommendation scenario and improvement strategies gained by the model through reinforcement learning training. By pushing target objects to users, this application improves the accuracy of personalized recommendations. This is because the recommendation score prediction task directly reflects the user's potential rating tendency for each object to be recommended, while the matching degree evaluation task ensures that the recommended content is highly consistent with the user's preferences, thereby solving the technical problem of poor model recommendation performance in related technologies.

[0183] In the above embodiments of this application, an initial recommendation model is used to perform multiple inference tasks on training preference information and training object information to generate a training inference trajectory. This includes: adding training preference information and training object information to a prompt word template to obtain training prompt words, wherein the prompt word template contains multiple inference tasks and output labels corresponding to the multiple inference tasks; inputting the training prompt words into the initial recommendation model, and using an object recommendation model to perform multiple inference tasks on the training preference information and training object information to generate an inference trajectory, wherein the execution result of any inference task is located between the output labels corresponding to the inference task.

[0184] The aforementioned cue word template is a predefined text structure containing guidance statements for multiple inference tasks, as well as specific output labels used to distinguish and label the output results of each inference task. The purpose of the cue word template is to guide and constrain the model to generate inference trajectories in a specific order and format, ensuring the parsability and structure of the output.

[0185] In this embodiment, the system integrates training preference information and training object information into a preset prompt word template to generate training prompt words. This integration process ensures clear guidance for each inference task and clarifies the end boundary of each task through output labels. Next, the system inputs the training prompt words into the initial recommendation model, triggering the initial recommendation model to analyze and infer the training preference information and training object information, generating an inference trajectory. In this process, the initial recommendation model needs to complete user preference analysis, item feature summarization, matching degree evaluation, and recommendation score prediction according to multiple inference tasks. The result of each inference task is placed between the corresponding output labels, forming structured and semantically clear training prompt words.

[0186] The training prompts and structured inference trajectories in this embodiment provide a clear guiding framework for model learning. Unlike traditional free-format inference, this method requires the model to strictly follow the prompt template, ensuring that the results of each inference step are clearly identifiable, facilitating subsequent reward modeling and parameter tuning. Furthermore, placing the inference results in their corresponding positions helps the model understand and adhere to the structured output specifications of the recommendation domain, thereby improving the readability and usability of the inference results.

[0187] The steps described above ensure that the inference trajectory generated by the model contains detailed reasoning information and follows a clear organizational structure. This provides a solid foundation for subsequent rule-based reward modeling, facilitates effective updates to model parameters, and ultimately improves the accuracy of recommendation score predictions. Furthermore, the structured reasoning process helps the model learn key reasoning techniques in the recommendation domain, enhancing its generalization ability in similar tasks.

[0188] In the music recommendation scenario, the system prepared training data including user listening history, preference descriptions, and details of songs to be recommended. The system first designed prompt word templates, which included four inference tasks:<analyze_user> Analyze users' listening history and preferences< / analyze_user> "、"<analyze_item> Analyze song attributes< / analyze_item> "、" <match> Assess the match between users and songs.< / match> "as well as" <rate> Predicting user recommendation scores for songs< / rate> Each task is followed by a specific output label. Next, the system embeds each set of training preference information and training object information into the prompt word template, generating training prompt words. For example, for a user who prefers light melodies and classical music, and a new song with similar attributes, the training prompt words might include a description of the user's preferences and a description of the song's features, guiding the model to complete the four tasks mentioned above. Once the training prompt words are input into the initial recommendation model, the model begins to reason item by item. Each reasoning result is placed between the corresponding output labels, forming a structured reasoning trajectory. This process not only ensures the structured nature of the reasoning process but also promotes the model's deep learning in the music recommendation domain, ultimately improving the accuracy of predicting user song recommendation scores and providing users with more personalized music recommendation services.

[0189] In the above embodiments of this application, reinforcement learning is performed on the initial recommendation model based on the training inference trajectory and the preset recommendation score to obtain the object recommendation model, including: evaluating the training inference trajectory and the preset recommendation score based on the preset reward rule to obtain the total training reward value; and updating the model parameters of the initial recommendation model based on the total training reward value to obtain the object recommendation model.

[0190] The aforementioned preset reward rules in this embodiment define how to transform the structural quality of the training inference trajectory and the accuracy of the prediction score into quantifiable reward signals to guide the updating of model parameters. The preset reward rules not only include an evaluation of format correctness but also encompass reward values ​​calculated from the score prediction error, ensuring comprehensiveness and depth in the model learning process.

[0191] The total training reward value mentioned above is generated after evaluating the training inference trajectory and the preset recommendation score based on the preset reward rules. It is a combination of the sum or weighted average of all individual reward values, reflecting the overall performance of the model on one or a series of training samples. The TRV directly affects the direction and extent of model parameter updates and is a key basis for adjusting reinforcement learning parameters.

[0192] In this embodiment, the system first evaluates the training inference trajectories and preset recommendation scores based on preset reward rules. This process involves checking the structural correctness of each inference trajectory and calculating the reward value based on the deviation of the recommendation score prediction. The system combines the structural reward and the score reward to generate a total training reward value. This step ensures that the model receives effective feedback in both the dimensions of inference trajectory structure and recommendation score prediction accuracy.

[0193] Subsequently, the system updates the model parameters of the initial recommendation model based on the total training reward value. This update follows the reinforcement learning strategy, which adjusts the model parameters through positive and negative reward values ​​to improve the correctness of the structure and the prediction accuracy of the recommendation score in future inferences, ultimately resulting in an improved object recommendation model. Through this mechanism, the model can learn and improve itself, effectively enhancing its recommendation capabilities even in the absence of explicit teacher guidance and expensive labeled data.

[0194] The preset reward rules and total training reward calculation mechanism in this application provide specific guidance for model learning. Unlike previous supervised learning methods that rely on large amounts of labeled data, this application, through its finely designed reward rules, can guide the model to generate high-quality inference trajectories and make accurate recommendation score predictions even with limited training data and no manual annotation. Particularly in the "inter-group relative policy improvement" stage, the system compares the behavior of different groups of models and adjusts parameters using relative performance rather than absolute performance. This effectively avoids the model getting stuck in locally good solutions during the learning process, improving the robustness and generalization ability of the recommendation model.

[0195] In this embodiment, by generating a total training reward value based on a preset reward rule and updating the parameters of the initial recommendation model accordingly, the final object recommendation model achieves a significant improvement in the quality of inference trajectories and the accuracy of recommendation score prediction. This effect stems from the system's ability to guide the model to learn and adjust itself without relying on large amounts of labeled data or complex multi-stage distillation processes, greatly reducing the cost and time of training the recommendation system while improving the personalization level of recommendations and user satisfaction.

[0196] Taking music recommendation as an example, suppose the system collects a set of training preference information and training object information, including user music preferences and attributes of songs to be recommended. The system presets a recommendation score of 5 as the scoring standard. Next, the system inputs this information into the initial recommendation model, which generates inference trajectories, including analysis of user interests, summarization of song features, matching degree evaluation, and recommendation score prediction. For these inference trajectories, the system performs reward evaluation based on preset reward rules: structural rewards ensure the completeness of the inference trajectory, and scoring rewards are calculated based on the deviation between the predicted score and the preset recommendation score. The system combines structural rewards and scoring rewards to generate a total training reward value for each inference trajectory. Finally, based on these total training reward values, the system updates the parameters of the initial recommendation model, achieving an improvement from the initial model to an object recommendation model. This process not only enhances the model's inference ability but also improves its accuracy in predicting user preference scores in music recommendation scenarios, bringing users a more personalized music recommendation experience and improving the overall performance of the recommendation system.

[0197] In the above embodiments of this application, the preset reward rules include: format reward rules and scoring reward rules; based on the preset reward rules, reward evaluation is performed on the training inference trajectory and the preset recommendation score to obtain the total training reward value, including: based on the format reward rules, reward evaluation is performed on the output format of the training inference trajectory to obtain a training format reward value, wherein the training format reward value is used to characterize whether the output format of the training inference trajectory is correct; based on the scoring reward rules and the preset recommendation score, reward evaluation is performed on the training recommendation score in the inference trajectory to obtain a training scoring reward value, wherein the training scoring reward value is used to characterize the deviation between the training recommendation score and the preset recommendation score; the training format reward value and the training scoring reward value are summarized to obtain the total training reward value.

[0198] The aforementioned format reward rule specifies how to evaluate the correctness of the output format of the training inference trajectory. FRR aims to ensure that the inference trajectory generated by the model strictly follows the specified structure and format. This includes the extraction of user interests, the summarization of item features, the evaluation of matching degree, and the output of the final score prediction, all of which must be placed within a specific "OutputTag" (OT) to ensure the clarity of the structure and the integrity of the inference process.

[0199] The scoring reward rule described above defines a method for calculating rewards based on the deviation between the predicted recommended score in the inference trajectory and the preset recommended score. The goal of the scoring reward rule is to quantify the accuracy of the model's predictions, encourage the model to get as close as possible to the preset recommended score, and thus improve the performance of the scoring prediction task.

[0200] In this embodiment, the system evaluates the output format of the training inference trajectory and the accuracy of the predicted recommendation score based on preset reward rules. First, the system applies a format reward rule to check the format. If the structure and format of the inference trajectory are correct, a positive training format reward value is obtained; conversely, if the format is incorrect, the training format reward value is negative, thus guiding the model to improve the output structure of its inference trajectory. Then, the system uses a scoring reward rule to generate a training scoring reward value by comparing the deviation between the trained recommendation score and the preset recommendation score in the inference trajectory. The design of the training format reward value considers the deviation measure between the predicted score and the preset score, ensuring that the model not only focuses on the structural integrity of the inference trajectory but also emphasizes the accuracy of the predicted recommendation score. Finally, the system summarizes the format reward value and the scoring reward value to calculate the total training reward value.

[0201] The above summation or weighted averaging process combines reward signals from both structural normalization and the accuracy of recommendation score prediction, providing comprehensive guidance for model parameter updates and ensuring that the model can be effectively improved in both structure and function.

[0202] The format reward rules and scoring reward rules in this embodiment constitute a system of preset reward rules, providing comprehensive guidance for model training during reinforcement learning. Unlike previous supervised learning methods, this embodiment, through the combination of different reward values, ensures that the model follows the norms of structured reasoning and makes accurate recommendation score predictions without relying on manually labeled inference trajectories. This improvement not only reduces data preparation costs but also promotes the model's autonomous learning ability in the recommendation domain, enhancing the overall performance of the recommendation system.

[0203] By implementing the aforementioned preset reward rules, this embodiment of the application can guide the model to simultaneously learn the output format of high-quality inference trajectories and accurate recommendation score predictions, achieving a dual improvement in both structure and functionality. This improvement stems from the system's ability to quantitatively evaluate the model's structural regularity and the accuracy of recommendation score predictions during the inference process, thereby providing more comprehensive guidance during model parameter adjustment and ensuring the model's robustness and accuracy in recommendation tasks.

[0204] For example, in a music recommendation scenario, suppose the system's preset rating (recommendation score) is 5 points, indicating that the user really likes the song. During training, the model generates an inference trajectory for a song, including a summary of user preferences, song features, matching degree, and the final predicted recommendation score. The system first applies a format reward rule to check the output format of the inference trajectory. If the inference trajectory is output according to the specified structure (e.g., "..."), then the system will consider the inference trajectory as a valid inference trajectory.<analyze_user> "to"< / analyze_user> "Extract user interests from between,"<analyze_item> "to"< / analyze_item> Summarize the characteristics of the items, and... <rate> "to"< / rate> The model receives a positive training format reward value when predicting a score close to 5. Next, the system uses a rating reward rule to evaluate the deviation between the predicted recommendation score and the preset 5-point rating, generating a training rating reward value. If the model's predicted recommendation score is close to 5, the training rating reward value will be higher; otherwise, it will be lower. Finally, the system sums the training format reward value and the training rating reward value to obtain the total training reward value, which is used to guide the update of model parameters. This mechanism promotes the generation of inference trajectories that are both structurally sound and accurately rated, significantly improving the personalization level of music recommendations and user satisfaction, providing users with a recommendation experience that better matches their musical tastes.

[0205] In the above embodiments of this application, updating the model parameters of the initial recommendation model based on the total training reward value to obtain the object recommendation model includes: generating the training objective function value corresponding to the initial recommendation model based on the total training reward value and the training inference trajectory; and updating the model parameters of the initial recommendation model based on the training objective function value.

[0206] The aforementioned training objective function value refers to a numerical indicator that combines the total training reward value and the training inference trajectory to quantify the overall performance of the initial recommendation model when performing recommendation tasks, and is used to guide the direction of model parameter updates.

[0207] In this embodiment, the system calculates the correlation between the total training reward value and the training inference trajectory, converting it into a training objective function value that can guide parameter adjustment. In this conversion, the total training reward value reflects the gap between the model's current behavior and the expected recommendation behavior, while the training objective function value is a quantitative expression of this gap, used to guide the model parameters to adjust towards better recommendation performance. With the training objective function value, the system adjusts the parameters of the initial recommendation model through iterative training, enabling the model to generate inference trajectories that better conform to the formatted reward rules and are closer to the preset recommendation score when performing the next inference task. This parameter update process follows the reinforcement learning strategy, gradually improving the model's recommendation ability by refining the training objective function value.

[0208] The embodiments of this application, through rule-based reward evaluation and objective function value generation, enable the model to not only self-evaluate the quality of its inference trajectory, but also compare its relative performance on multiple sets of training data. This allows the model to consider a wider range of scenarios when adjusting parameters, thereby enhancing the model's learning stability and recommendation performance.

[0209] This application's embodiments guide model parameter updates by training the objective function value, effectively promoting model self-improvement in two key areas: inference path generation and recommendation score prediction. This technical effect is reflected in the model's ability to produce clearly structured and logically coherent inference trajectories, while significantly improving the accuracy of recommendation score prediction, directly enhancing the personalized recommendation level and user satisfaction of the recommendation system. Because this process avoids reliance on large amounts of labeled data, it also significantly reduces the development and maintenance costs of the recommendation system.

[0210] For example, in a video game recommendation scenario, assume the system has collected a series of user play history and game metadata as part of its training dataset. The system pre-sets some recommendation scores as target references for model learning. First, the system uses the initial recommendation model to infer from this training data, generating an inference trajectory that includes user preference analysis, game feature summary, matching degree evaluation, and recommendation score prediction. Then, the system evaluates the generated inference trajectory and recommendation scores based on pre-set reward rules, calculating the total training reward value. The system combines the total training reward value with the inference trajectory to generate a training objective function value. This function value quantifies the performance of the initial recommendation model in the recommendation process, including the structure of the inference trajectory and the accuracy of the predicted recommendation score.

[0211] Next, the system enters the parameter update phase. Based on the training objective function value, the system adjusts the model parameters to ensure that in subsequent recommendation tasks, the model can generate inference trajectories that better conform to the format reward rules, and that the predicted recommendation scores are more accurate and closer to the preset recommendation scores. After this series of training and parameter adjustments, the resulting object recommendation model shows a significant improvement in performance on video game recommendation tasks. It can not only predict more accurate game ratings based on users' play history and preferences, but also the output of inference trajectories is more structured and easier to understand, providing users with a more personalized and high-quality game recommendation experience. This also reduces the need for manual intervention and enhances the automatic learning and adaptability of the recommendation system.

[0212] In the above embodiments of this application, the method further includes: sampling the training data to obtain sub-training data; using the inference trajectory generation model to perform multiple inference tasks on the sub-preference information and sub-object information in the sub-training data to generate a target inference trajectory, wherein the target inference trajectory includes: the target execution results corresponding to the multiple inference tasks respectively; using the sub-training data and the target inference trajectory to perform supervised training on the model to be trained to obtain an initial recommendation model.

[0213] The aforementioned sub-training data is a subset of data selected from the complete training dataset using a specific sampling strategy. It is used to generate the target inference trajectory and for the supervised training phase of the initial cold-start recommendation model. This data contains sub-preference information and sub-object information, forming the foundation for model learning and training.

[0214] The aforementioned inference trajectory generation model is a component used to generate target inference trajectories in the embodiments of this application. During supervised training, this model processes and generates execution results related to specific inference tasks to enrich the model's inference capabilities, which is particularly important during the cold start phase.

[0215] In this embodiment, the operations of acquiring sub-training data and generating the target inference trajectory lay the foundation for subsequent supervised training and obtaining the initial recommendation model. First, a subset of data is selected from the complete training dataset as sub-training data using a sampling strategy. This process aims to reduce the amount of training data, improve training efficiency, and ensure the representativeness and diversity of the data. Subsequently, the system uses the inference trajectory generation model to perform multiple inference tasks on sub-preference information and sub-object information to generate the target inference trajectory. The target inference trajectory includes the execution results of multiple inference tasks, such as user interest analysis, item feature summarization, matching degree evaluation, and rating prediction. These results are strictly organized according to a preset output label format.

[0216] Finally, using the sub-training data and the target inference trajectory, the system performs supervised training on a model to be trained, obtaining an initial recommendation model. This process guides the learning of the execution rules of the inference task and the ability to predict scores by using the target inference trajectory, sub-preference information, and sub-object information as input, thereby establishing a preliminary inference ability and recommendation score prediction mechanism.

[0217] The above operations are part of a hybrid paradigm for reinforcement modules, designed to lay a solid foundation for subsequent reinforcement learning training phases through cold-start supervised fine-tuning of high-quality inference trajectories. Unlike the pure RL fine-tuning framework of the inference reinforcement module, the reinforcement hybrid module incorporates a supervised cold-start (SFT-CS) training phase before reinforcement learning. By utilizing a teacher model with strong inference capabilities to generate inference trajectories, it performs preliminary training on the IRM, overcoming the domain difference problem of pre-trained large models in the recommendation domain, accelerating the learning process, and improving the model's robustness and recommendation capabilities.

[0218] Through the above steps, the embodiments of this application can achieve rapid initialization of the model's reasoning ability and preliminary calibration of recommendation score prediction. This technical effect is reflected in the model's ability to master basic reasoning skills in a short time, creating favorable conditions for in-depth learning in subsequent stages. It also significantly improves the model's performance in recommendation scenarios, especially in the rating prediction task, achieving a reduction in mean absolute error, directly improving the accuracy of the recommendation system and the user experience.

[0219] For example, taking book recommendation as an example, the system first has a massive database of user reading records and book descriptions as a complete training dataset. To cold-start a recommendation system model to be trained, the system samples data from this database of specific user and book combinations as sub-training data. Next, the system uses an inference trajectory generation model to perform inference tasks on the sampled sub-preference and sub-object information, including analyzing user reading preferences, summarizing book themes and styles, evaluating the matching degree between users and books, and predicting the book ratings that users might give. This process generates target inference trajectories, each containing the execution results of the above inference tasks, and the results are separated by preset output labels (e.g., "...").<analyze_user> "to"< / analyze_user> ","<analyze_item> "to"< / analyze_item> "," <rate> "to"< / rate> To differentiate and organize them.

[0220] Subsequently, the system uses sub-training data and the target inference trajectory to conduct supervised training on a recommendation model to be trained. This training process adjusts the model parameters by comparing the inference trajectory generated by the model with the target inference trajectory, and by comparing the model's predicted recommendation score with the actual score, until the model can roughly master the inference skills and recommendation score prediction. This process yields an initial recommendation model. Although the model has not yet achieved optimal performance, it already possesses a certain level of inference ability and a foundation for recommendation score prediction, providing a strong starting point for the reinforcement learning training phase under the subsequent inference enhancement module framework. Through this series of operations, the initial recommendation model demonstrates a significant improvement in recommendation capabilities in book recommendation scenarios. It can generate more reasonable rating predictions based on user reading preferences and book attributes, reducing recommendation errors and increasing user acceptance of the recommendation results.

[0221] In the above embodiments of this application, the inference trajectory generation model is used to perform multiple inference tasks on the sub-preference information and sub-object information in the sub-training data to generate a target inference trajectory. This includes: using the inference trajectory generation model to perform multiple inference tasks on the sub-preference information and sub-object information in the sub-training data to generate an initial inference trajectory; matching the initial recommendation score in the initial inference trajectory with the sub-recommendation score in the sub-training data to obtain a matching result, wherein the matching result is used to characterize whether the initial recommendation score and the sub-recommendation score are consistent; if the matching result indicates that the initial recommendation score and the sub-recommendation score are consistent, adding the initial inference trajectory to the target inference trajectory; if the matching result indicates that the initial recommendation score and the sub-recommendation score are inconsistent, processing the sub-training data and the target inference trajectory using the inference trajectory generation model to generate a new inference trajectory, and adding the new inference trajectory to the target inference trajectory, wherein the new recommendation score in the new inference trajectory is consistent with the sub-recommendation score.

[0222] According to an embodiment of this application, an object recommendation apparatus for implementing the above-described object recommendation method is also provided. Figure 5 This is a schematic diagram of an object recommendation device according to an embodiment of this application, such as... Figure 5 As shown, the device 500 includes: an acquisition module 502, a generation module 504, a determination module 506, and a push module 508.

[0223] The system comprises the following modules: an acquisition module for acquiring user preference information and object information corresponding to multiple objects to be recommended, wherein the preference information represents the user's preference for multiple objects to be recommended; a generation module for using an object recommendation model to perform multiple inference tasks on the preference information and the object information corresponding to the objects to be recommended, generating an inference trajectory corresponding to the objects to be recommended, wherein the inference trajectory includes the execution results of multiple inference tasks, including preference information extraction, object information extraction, matching degree evaluation, and recommendation score prediction; the object recommendation model is trained through reinforcement learning; a determination module for determining the target object from multiple objects to be recommended based on the inference trajectories corresponding to the multiple objects to be recommended; and a push module for pushing the target object to the user.

[0224] The acquisition module 502, generation module 504, determination module 506, and push module 508 described above correspond to steps S202 to S208 in the above embodiments. The four modules and their corresponding steps implement the same instances and application scenarios, but are not limited to the content disclosed in the above embodiments. The modules or units described above may be hardware or software components stored in memory and processed by one or more processors. These modules may also be part of a device and may run in the server 10 provided in the above embodiments.

[0225] In the above embodiments of this application, the generation module is used to add preference information and object information to the prompt word template to obtain the target prompt word. The prompt word template contains multiple inference tasks and output labels corresponding to the multiple inference tasks. The target prompt word is input into the object recommendation model, and the object recommendation model is used to perform multiple inference tasks on the preference information and object information to generate an inference trajectory. The execution results of the inference tasks are located between the output labels corresponding to the inference tasks.

[0226] In the above embodiments of this application, the generation module is used to perform a preference information extraction task on the preference information using an object recommendation model to obtain the preference features corresponding to the preference information extraction task; to perform an object information extraction task on the object information using an object recommendation model to obtain the object features corresponding to the object information extraction task; to perform a matching degree estimation task on the preference features and object features using an object recommendation model to obtain the matching degree corresponding to the matching degree estimation task; and to perform a recommendation score prediction task on the matching degree using an object recommendation model to obtain the recommendation score corresponding to the recommendation score prediction task, wherein the recommendation score is used to characterize the degree of matching between the object to be recommended and the user.

[0227] In the above embodiments of this application, the determining module is used to determine the recommendation score corresponding to the object to be recommended from the inference trajectory corresponding to each object to be recommended; based on the recommendation scores corresponding to each of the multiple objects to be recommended, the multiple objects to be recommended are sorted to obtain a sorted object sequence; and the object to be recommended at the top of the sorted object sequence is obtained to obtain the target object.

[0228] In the above embodiments of this application, the device is further configured to output an inference trajectory to the user; respond to receiving a target recommendation score from the user; evaluate the inference trajectory and the target recommendation score based on a preset reward rule to obtain a total reward value; and update the model parameters of the object recommendation model based on the total reward value.

[0229] In the above embodiments of this application, the preset reward rules include: format reward rules and scoring reward rules; based on the preset reward rules, the device is further configured to evaluate the output format of the inference trajectory based on the format reward rules to obtain a format reward value, wherein the format reward value is used to characterize whether the output format of the inference trajectory is correct; based on the scoring reward rules and the target recommendation score, evaluate the recommendation score in the inference trajectory to obtain a scoring reward value, wherein the scoring reward value is used to characterize the deviation between the recommendation score and the target recommendation score; and summarize the format reward value and the scoring reward value to obtain a total reward value.

[0230] In the above embodiments of this application, the device is further configured to obtain the absolute value of the difference between the target recommended score and the recommended score to obtain the target deviation; and to determine the score reward value based on the target deviation and the preset error.

[0231] In the above embodiments of this application, the device is further configured to generate an objective function value corresponding to the object recommendation model based on the total reward value and the inference trajectory; and update the model parameters of the object recommendation model based on the objective function value.

[0232] According to an embodiment of this application, a training apparatus for an object recommendation model used to implement the above-described training method for the object recommendation model is also provided. Figure 6 This is a schematic diagram of a training device for an object recommendation model according to an embodiment of this application, as shown below. Figure 6 As shown, the device 600 includes: an acquisition module 602, a generation module 604, and an enhancement module 606.

[0233] The acquisition module is used to acquire training data, which includes training preference information, training object information, and preset recommendation scores. The generation module is used to perform multiple inference tasks on the training preference information and training object information using the initial recommendation model to generate training inference trajectories. The training inference trajectories include the training execution results corresponding to the multiple inference tasks, which include preference information extraction, object information extraction, matching degree evaluation, and recommendation score prediction. The reinforcement module is used to perform reinforcement learning on the initial recommendation model based on the training inference trajectories and preset recommendation scores to obtain an object recommendation model. The object recommendation model is used to execute the method of any one of claims 1 to 8.

[0234] The acquisition module 602, generation module 604, and enhancement module 606 described above correspond to steps S402 to S406 in the above embodiments. The three modules and their corresponding steps implement the same instances and application scenarios, but are not limited to the content disclosed in the above embodiments. The modules or units described above may be hardware or software components stored in memory and processed by one or more processors. These modules may also be part of a device and may run in the server 10 provided in the above embodiments.

[0235] In the above embodiments of this application, the generation module is used to add training preference information and training object information to the prompt word template to obtain training prompt words. The prompt word template contains multiple inference tasks and output labels corresponding to the multiple inference tasks. The training prompt words are input into the initial recommendation model, and the object recommendation model is used to perform multiple inference tasks on the training preference information and training object information to generate inference trajectories. The execution result of any inference task is located between the output labels corresponding to the inference task.

[0236] In the above embodiments of this application, the reinforcement module is used to evaluate the training inference trajectory and the preset recommendation score based on the preset reward rules to obtain the total training reward value; and to update the model parameters of the initial recommendation model based on the total training reward value to obtain the object recommendation model.

[0237] In the above embodiments of this application, the preset reward rules include: format reward rules and scoring reward rules; based on the preset reward rules, the reinforcement module is further used to evaluate the output format of the training inference trajectory based on the format reward rules to obtain a training format reward value, wherein the training format reward value is used to characterize whether the output format of the training inference trajectory is correct; based on the scoring reward rules and the preset recommendation score, the module evaluates the training recommendation score in the inference trajectory to obtain a training score reward value, wherein the training score reward value is used to characterize the deviation between the training recommendation score and the preset recommendation score; the training format reward value and the training score reward value are summarized to obtain a total training reward value.

[0238] In the above embodiments of this application, the reinforcement module is used to generate the training objective function value corresponding to the initial recommendation model based on the total training reward value and the training inference trajectory; and to update the model parameters of the initial recommendation model based on the training objective function value.

[0239] In the above embodiments of this application, the device is further configured to sample training data to obtain sub-training data; use the inference trajectory generation model to perform multiple inference tasks on the sub-preference information and sub-object information in the sub-training data to generate a target inference trajectory, wherein the target inference trajectory includes: the target execution results corresponding to the multiple inference tasks respectively; and use the sub-training data and the target inference trajectory to perform supervised training on the model to be trained to obtain an initial recommendation model.

[0240] In the above embodiments of this application, the generation module is used to perform multiple inference tasks on the sub-preference information and sub-object information in the sub-training data using an inference trajectory generation model to generate an initial inference trajectory; the initial recommendation score in the initial inference trajectory is matched with the sub-recommendation score in the sub-training data to obtain a matching result, wherein the matching result is used to characterize whether the initial recommendation score and the sub-recommendation score are consistent; if the matching result indicates that the initial recommendation score and the sub-recommendation score are consistent, the initial inference trajectory is added to the target inference trajectory; if the matching result indicates that the initial recommendation score and the sub-recommendation score are inconsistent, the inference trajectory generation model is used to process the sub-training data and the target inference trajectory to generate a new inference trajectory, and the new inference trajectory is added to the target inference trajectory, wherein the new recommendation score in the new inference trajectory is consistent with the sub-recommendation score.

[0241] The preferred embodiments involved in the above embodiments of this application are the same as the solutions, application scenarios and implementation processes provided in the above embodiments, and will not be repeated here.

[0242] Embodiments of this application may provide a computing device. Figure 7 This is a structural block diagram of a computing device according to an embodiment of this application. Figure 7 As shown, the computing device 100 may include one or more (one shown in the figure) processors 102, memory 104, memory controller, and peripheral interfaces.

[0243] The aforementioned computing device can be understood as an integrated intelligent terminal, including but not limited to servers, desktop computers, PCs (Personal Computers), and all-in-one model machines. Furthermore, the computing device may have the model described in the above embodiments of this application pre-installed.

[0244] Specifically, this computing device can pre-install various types of models, including but not limited to models in fields such as natural language processing, visual processing, speech processing, code processing, and multimodal task processing, thus providing diverse model choices. In different product forms, this computing device can support one or more model usage methods, including but not limited to model training, model invocation, model fine-tuning, model deployment, model inference, and application. In some product forms, this computing device also supports model management, including but not limited to multi-type model management (supporting the management of discriminative, generative, and other model types), model version control (supporting the control of different model versions), and model evaluation (evaluating model performance and effectiveness based on model evaluation tools). In other product forms, this computing device can also create applications based on models, providing API calling capabilities. Models can be called into created applications through API interfaces, and application management tools are provided to control and manage applications.

[0245] Furthermore, this computing device can also include data management (supporting the creation and management of model tuning datasets), a training center (providing abundant training resources to help users learn and master AI technologies), and basic control capabilities (providing enterprise-level basic control capabilities to ensure system security and efficient operation). Through these functions, it provides a comprehensive, integrated device for AI development, training, deployment, and application.

[0246] The memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the methods and apparatus in the embodiments of this application. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, thereby implementing the methods in the above embodiments. The memory may include high-speed random access memory (RAM) and non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memories. In some instances, the memory may further include memory remotely located relative to the processor, and these remote memories can be connected to terminal A via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks (LANs), mobile communication networks, and combinations thereof.

[0247] The processor can invoke an executable program stored in memory via a transmission device to execute the method described in any of the above embodiments.

[0248] Embodiments of this application may provide an electronic device. Figure 8 This is a structural block diagram of an electronic device according to an embodiment of this application. Figure 8 As shown, the electronic device may include: an input / output device 112; a memory 114; and a processor 116, wherein the processor 116 is connected to the input / output device 112 and the memory 114 via a bus 118.

[0249] The memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the methods and apparatus in the embodiments of this application. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, thereby implementing the methods in the above embodiments. The memory may include high-speed random access memory (RAM) and non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memories. In some instances, the memory may further include memory remotely located relative to the processor, and these remote memories can be connected to terminal A via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks (LANs), mobile communication networks, and combinations thereof.

[0250] The processor can invoke an executable program stored in memory via a transmission device to execute the method described in any of the above embodiments.

[0251] Those skilled in the art will understand that, Figure 8 The structure shown is schematic. Electronic devices can also be smartphones (such as Android phones, iOS phones, etc.), tablets, PDAs, mobile internet devices (MIDs), PADs, and other terminal devices. This diagram does not limit the structure of the aforementioned electronic devices. For example, electronic devices may include more or fewer components (such as network interfaces, display devices, etc.) than shown in the diagram, or have different configurations than those shown.

[0252] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing the hardware related to the terminal device. The program can be stored in a computer-readable storage medium, which may include: a flash drive, read-only memory (ROM), random access memory (RAM), a magnetic disk, or an optical disk, etc.

[0253] Embodiments of this application also provide a computer-readable storage medium. Optionally, in this embodiment, the aforementioned computer-readable storage medium can be used to store program code executed by the method provided in the above embodiments.

[0254] Optionally, in this embodiment, the storage medium may be located in a computing device or an electronic device.

[0255] Optionally, in this embodiment, the computer-readable storage medium is configured to store an executable program. When the executable program runs, it controls the device where the computer-readable storage medium is located to perform the method described in any of the above embodiments.

[0256] Embodiments of this application also provide a computer program product. Optionally, in this embodiment, the computer program product may include a computer program. When executed by a processor, the computer program implements the methods provided in the above embodiments.

[0257] Embodiments of this application also provide a computer program product. Optionally, the computer program product may include a non-volatile computer-readable storage medium. The non-volatile computer-readable storage medium can be used to store a computer program. When the computer program is executed by a processor, it implements the method provided in the above embodiments.

[0258] Embodiments of this application also provide a computer program. Optionally, in this embodiment, when the computer program is executed by a processor, it implements the method provided in the above embodiments.

[0259] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0260] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are illustrative; for example, the division of units is a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined, integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling, direct coupling, or communication connection shown or discussed may be through some interfaces, indirect coupling of units or modules, or communication connection, and may be electrical or other forms.

[0261] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of this embodiment.

[0262] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0263] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0264] The above description represents the preferred embodiments of this application. For those skilled in the art, various improvements and modifications can be made without departing from the principles of this application, and these improvements and modifications should also be considered within the scope of protection of this application.< / rate> < / match> < / rate>

Claims

1. An object recommendation method, characterized in that, include: Obtain user preference information and object information corresponding to multiple objects to be recommended; The object recommendation model is used to perform multiple inference tasks on the preference information and the object information corresponding to the object to be recommended, thereby generating an inference trajectory corresponding to the object to be recommended. The inference trajectory includes the execution results of the multiple inference tasks, which include: preference information extraction task, object information extraction task, matching degree evaluation task, and recommendation score prediction task. The object recommendation model is obtained through reinforcement learning training. Based on the reasoning trajectories corresponding to the multiple objects to be recommended, a target object is determined from the multiple objects to be recommended; The target object is pushed to the user.

2. The method according to claim 1, characterized in that, The method of using an object recommendation model to perform multiple inference tasks on the preference information and the object information corresponding to the object to be recommended, and generating an inference trajectory corresponding to the object to be recommended, includes: The preference information and the object information are added to the prompt word template to obtain the target prompt word, wherein the prompt word template contains the multiple inference tasks and the output labels corresponding to the multiple inference tasks respectively; The target prompt is input into the object recommendation model, and the object recommendation model is used to perform the multiple inference tasks on the preference information and the object information to generate the inference trajectory, wherein the execution result of the inference task is located between the output labels of the inference task.

3. The method according to claim 2, characterized in that, The step of using the object recommendation model to perform the multiple inference tasks on the preference information and the object information to generate the inference trajectory includes: The object recommendation model is used to perform the preference information extraction task on the preference information to obtain the preference features corresponding to the preference information extraction task; The object recommendation model is used to perform an object information extraction task on the object information to obtain the object features corresponding to the object information extraction task; The object recommendation model is used to perform a matching degree estimation task on the preference features and the object features to obtain the matching degree corresponding to the matching degree estimation task. The object recommendation model is used to perform the recommendation score prediction task on the matching degree to obtain the recommendation score corresponding to the recommendation score prediction task, wherein the recommendation score is used to characterize the matching degree between the object to be recommended and the user.

4. The method according to claim 1, characterized in that, The step of determining the target object from the plurality of objects to be recommended based on the inference trajectories corresponding to the plurality of objects to be recommended includes: The recommendation score corresponding to the recommended object is determined from the reasoning trajectory corresponding to each of the objects to be recommended. Based on the recommendation scores corresponding to the multiple objects to be recommended, the multiple objects to be recommended are sorted to obtain a sorted object sequence; The top-ranked object to be recommended is obtained from the sorted object sequence to obtain the target object.

5. The method according to any one of claims 1 to 4, characterized in that, The method further includes: The inference trajectory is output to the user; In response to receiving the target recommendation score from the user feedback; Based on preset reward rules, the inference trajectory and the target recommendation score are evaluated for rewards to obtain the total reward value; The model parameters of the object recommendation model are updated based on the total reward value.

6. The method according to claim 5, characterized in that, The preset reward rules include: format reward rules and scoring reward rules; the step of evaluating the inference trajectory and the target recommendation score based on the preset reward rules to obtain a total reward value includes: Based on the format reward rule, the output format of the inference trajectory is evaluated to obtain a format reward value, wherein the format reward value is used to characterize whether the output format of the inference trajectory is correct; Based on the scoring reward rules and the target recommendation score, the recommendation score in the inference trajectory is evaluated to obtain a scoring reward value, wherein the scoring reward value is used to characterize the deviation between the recommendation score and the target recommendation score; The total reward value is obtained by summing the format reward value and the score reward value.

7. The method according to claim 6, characterized in that, The step of evaluating the recommendation score in the inference trajectory based on the scoring reward rule and the target recommendation score to obtain a scoring reward value includes: The target deviation is obtained by acquiring the absolute value of the difference between the target recommendation score and the recommendation score. The scoring reward value is determined based on the target deviation and the preset error.

8. The method according to claim 5, characterized in that, The step of updating the model parameters of the object recommendation model based on the total reward value includes: Based on the total reward value and the inference trajectory, the objective function value corresponding to the object recommendation model is generated; The model parameters of the object recommendation model are updated based on the objective function value.

9. A training method for an object recommendation model, characterized in that, include: Acquire training data, wherein the training data includes: training preference information, training object information, and preset recommendation scores; The initial recommendation model is used to perform multiple inference tasks on the training preference information and the training object information to generate a training inference trajectory. The training inference trajectory includes the training execution results corresponding to the multiple inference tasks. The multiple inference tasks include: preference information extraction task, object information extraction task, matching degree evaluation task, and recommendation score prediction task. Based on the training inference trajectory and the preset recommendation score, reinforcement learning is performed on the initial recommendation model to obtain an object recommendation model, wherein the object recommendation model is used to perform the method described in any one of claims 1 to 8.

10. The method according to claim 9, characterized in that, The step of using the initial recommendation model to perform multiple inference tasks on the training preference information and the training object information to generate a training inference trajectory includes: The training preference information and the training object information are added to the prompt word template to obtain training prompt words, wherein the prompt word template contains the multiple inference tasks and the output labels corresponding to the multiple inference tasks respectively; The training prompts are input into the initial recommendation model, and the object recommendation model is used to perform the multiple inference tasks on the training preference information and the training object information to generate the inference trajectory. The execution result of any inference task is located between the output labels corresponding to the inference task.

11. The method according to claim 9, characterized in that, The step of performing reinforcement learning on the initial recommendation model based on the training inference trajectory and the preset recommendation score to obtain an object recommendation model includes: Based on the preset reward rules, the training inference trajectory and the preset recommendation score are evaluated to obtain the total training reward value; The model parameters of the initial recommendation model are updated based on the total training reward value to obtain the object recommendation model.

12. The method according to claim 11, characterized in that, The preset reward rules include: format reward rules and scoring reward rules; the step of evaluating the training inference trajectory and the preset recommendation score based on the preset reward rules to obtain the total training reward value includes: Based on the format reward rule, the output format of the training inference trajectory is evaluated to obtain a training format reward value, wherein the training format reward value is used to characterize whether the output format of the training inference trajectory is correct; Based on the scoring reward rules and the preset recommendation score, the training recommendation score in the inference trajectory is evaluated to obtain a training score reward value, wherein the training score reward value is used to characterize the deviation between the training recommendation score and the preset recommendation score; The training format reward value and the training score reward value are summed to obtain the total training reward value.

13. The method according to claim 11, characterized in that, The step of updating the model parameters of the initial recommendation model based on the total training reward value to obtain the object recommendation model includes: Based on the total training reward value and the training inference trajectory, the training objective function value corresponding to the initial recommendation model is generated; The model parameters of the initial recommendation model are updated based on the training objective function value.

14. The method according to claim 9, characterized in that, The method further includes: The training data is sampled to obtain sub-training data; The inference trajectory generation model is used to perform the multiple inference tasks on the sub-preference information and sub-object information in the sub-training data to generate a target inference trajectory, wherein the target inference trajectory includes: the target execution results corresponding to the multiple inference tasks respectively; The initial recommendation model is obtained by using the sub-training data and the target inference trajectory to perform supervised training on the model to be trained.

15. The method according to claim 14, characterized in that, The step of using the inference trajectory generation model to perform the multiple inference tasks on the sub-preference information and sub-object information in the sub-training data to generate the target inference trajectory includes: The inference trajectory generation model is used to perform the multiple inference tasks on the sub-preference information and sub-object information in the sub-training data to generate an initial inference trajectory. The initial recommendation score in the initial inference trajectory is matched with the sub-recommendation score in the sub-training data to obtain a matching result, wherein the matching result is used to characterize whether the initial recommendation score and the sub-recommendation score are consistent; If the matching result indicates that the initial recommendation score is consistent with the sub-recommendation score, the initial inference trajectory is added to the target inference trajectory; If the matching result indicates that the initial recommendation score and the sub-recommendation score are inconsistent, the inference trajectory generation model is used to process the sub-training data and the target inference trajectory to generate a new inference trajectory, and the new inference trajectory is added to the target inference trajectory, wherein the new recommendation score in the new inference trajectory is consistent with the sub-recommendation score.

16. An electronic device, characterized in that, include: Memory, which stores executable programs; A processor, connected to a memory via a bus, is used to run the program, wherein the program, when running, performs the method according to any one of claims 1 to 15.

17. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored executable program, wherein, when the executable program is executed, it controls the device on which the computer-readable storage medium is located to perform the method according to any one of claims 1 to 15.

18. A computer program product, characterized in that, It includes a computer program that, when executed by a processor, implements the method described in any one of claims 1 to 15.

Images