Preference inference model training method and apparatus

Abstract

The application relates to the technical field of artificial intelligence, and provides a preference inference model training method and device. The method comprises the following steps: generating a plurality of training sample sets, wherein the plurality of training sample sets comprise a first training sample set, a second training sample set and a third training sample set; the first training sample set comprises training samples for distinguishing preference differences under different conversation contexts and identifying outdated preference interference; the second training sample set comprises training samples for identifying user negative expressions; the third training sample set comprises training samples for a long conversation history scenario; based on the plurality of training sample sets, a basic large language model is supervised and fine-tuned to obtain a preference inference model; and the preference inference model is used for inferring user preferences related to a current request. The preference inference model training method provided by the application can improve the user preference inference capability of the model in the scenarios of context preference conflict, preference evolution, negative expression and long conversation history.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a method and apparatus for training a preference inference model. Background Technology

[0002] Large Language Model (LLM) personalization aims to enable the model to provide not only accurate, relevant, and useful content in response to a user's current request, but also content that aligns with the user's individual preferences. This capability enhances the efficiency of communication between the user and the LLM and provides a better user experience. Therefore, accurately inferring user preferences is crucial. In practice, the interaction history between users and the LLM naturally contains rich preference signals. However, extracting preferences from these historical records remains insufficiently researched and challenging. Therefore, how to leverage appropriate model training strategies to mine user preferences is an urgent problem to be solved. Summary of the Invention

[0003] This application provides a method and apparatus for training a preference inference model, which addresses the shortcomings of existing technologies in extracting preferences from historical records and can accurately infer user preferences from interaction history.

[0004] This application provides a method for training a preference inference model, including the following steps: Multiple training sample sets are generated, including a first training sample set, a second training sample set, and a third training sample set. The first training sample set includes training samples for distinguishing preference differences in different conversational contexts and identifying outdated preference interference. The second training sample set includes training samples for identifying user negation expressions. The third training sample set includes training samples for long dialogue history scenarios. Based on these multiple training sample sets, a supervised fine-tuning of the basic large language model is performed to obtain a preference inference model. The preference inference model is used to infer user preferences related to the current request based on the user's interaction history with the large language model and the user's current request.

[0005] According to the preference inference model training method provided in this application, the first training sample set includes multiple training samples, and each training sample includes a core session, a distracting session, a target request, and a target preference; the core session is the session content in the training sample that reflects the target preference that matches the target request; the distracting session includes a strong distracting session and / or a weak distracting session; the weak distracting session refers to session content that is not related to the target request.

[0006] According to the preference inference model training method provided in this application, the conversation topic corresponding to the strong interference session is the same as the conversation topic corresponding to the core session, and the strong interference session satisfies at least one of the following: the conversation context corresponding to the strong interference session is different from the conversation context corresponding to the core session; or, the conversation context corresponding to the strong interference session is the same as the conversation context corresponding to the core session, and the user preference reflected by the strong interference session is an outdated preference that is earlier than the target preference.

[0007] According to a preference inference model training method provided in this application, when the interfering session includes the strongly interfering session, generating the first training sample set includes: Multiple session scenarios are generated; the multiple session scenarios include a core scenario and at least one interference scenario; the multiple session scenarios correspond to the same session topic; a first candidate preference and a second candidate preference are generated for each session scenario; a target preference allocation rule is determined from a set of preset preference allocation rules; based on the target preference allocation rule and the first candidate preference and the second candidate preference corresponding to each session scenario, a target candidate preference is determined for each session scenario; based on the core scenario and the target candidate preference corresponding to the core scenario, the core session, the target request, and the target preference are generated; based on each interference scenario and the target candidate preference corresponding to each interference scenario, the strongly interference session is generated.

[0008] According to the preference inference model training method provided in this application, the generation of multiple conversational scenarios includes: Determine the conversation topic; obtain the target context dimension from multiple preset context dimensions; generate multiple conversation contexts based on the conversation topic and the target context dimension.

[0009] According to the preference inference model training method provided in this application, the first candidate preference is used to construct the preference conflict between the core context and each of the interference contexts, wherein the first candidate preference corresponding to the core context is opposite to the first candidate preference corresponding to each of the interference contexts; and the second candidate preference corresponding to each of the conversation contexts is different from the first candidate preference corresponding to the conversation context.

[0010] According to the preference inference model training method provided in this application, the plurality of preference allocation rules include the following: The first preference allocation rule is used to indicate that the second candidate preference corresponding to the core scenario and each of the interference scenarios is used as the corresponding target candidate preference. The second preference allocation rule is used to indicate that the first candidate preference corresponding to the core situation is used as the target candidate preference corresponding to the core situation, the first candidate preference corresponding to the first interference situation in each interference situation is used as the target candidate preference corresponding to the first interference situation, and the first candidate preference or the second candidate preference corresponding to the second interference situation other than the first interference situation in each interference situation is used as the target candidate preference corresponding to the second interference situation. The third preference allocation rule is used to indicate that the core context is divided into a pre-evolution stage and a post-evolution stage. In the pre-evolution stage, the user preference that is opposite to the first candidate preference corresponding to the core context is taken as the target candidate preference corresponding to the core context. In the post-evolution stage, the first candidate preference corresponding to the core context is taken as the target candidate preference corresponding to the core context, and the second candidate preference corresponding to each interference context is taken as the target candidate preference corresponding to each interference context.

[0011] This application also provides a preference inference model training device, including the following modules: A generation module is used to generate multiple training sample sets, including a first training sample set, a second training sample set, and a third training sample set; wherein, the first training sample set includes training samples for distinguishing preference differences in different conversational contexts and identifying outdated preference interference, the second training sample set includes training samples for identifying user negative expressions, and the third training sample set includes training samples for long dialogue history scenarios. The fine-tuning module is used to perform supervised fine-tuning of the basic large language model based on the multiple training sample sets to obtain a preference inference model; the preference inference model is used to infer user preferences related to the current request based on the user's interaction history with the large language model and the user's current request.

[0012] According to the preference inference model training device provided in this application, the first training sample set includes multiple training samples, each training sample including a core session, a distracting session, a target request, and a target preference; the core session is the session content in the training sample that reflects the target preference that matches the target request; the distracting session includes a strong distracting session and / or a weak distracting session; the weak distracting session refers to session content that is not related to the target request.

[0013] According to the preference inference model training device provided in this application, the conversation topic corresponding to the strong interference session is the same as the conversation topic corresponding to the core session, and the strong interference session satisfies at least one of the following: the conversation context corresponding to the strong interference session is different from the conversation context corresponding to the core session; or, the conversation context corresponding to the strong interference session is the same as the conversation context corresponding to the core session, and the user preference reflected by the strong interference session is an outdated preference that is earlier than the target preference.

[0014] This application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the preference inference model training method as described above.

[0015] This application also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the preference inference model training method as described above.

[0016] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the preference inference model training method as described above.

[0017] Through the above technical solutions, this approach constructs multiple training sample sets covering different training objectives, achieving systematic training of user preference inference capabilities: The first training sample set includes training samples used to distinguish preference differences in different conversational contexts and identify outdated preference interference, providing the model with basic contextualized preference learning materials. This enables the model to stably capture preference signals related to the current request from the user's interaction history, effectively addressing the problem of preference differences in different contexts under the same topic. The second training sample set introduces training samples related to user negation expressions, enabling the model to understand the user's reverse preference signals and improving robustness to noisy interaction information. The third training sample set is trained for long context scenarios, enabling the model to filter noise and locate key preference information from ultra-long interaction histories, solving the problem of difficulty in preference extraction under long historical interactions. Based on this, the preference inference model obtained through supervised fine-tuning can simultaneously address multiple challenges such as historical interaction noise, dynamic evolution of preferences, scenario preference conflicts, and long context constraints during the interaction between the user and the large language model. It accurately infers user preferences from the interaction history, enabling the large language model's response to meet the user's personalized needs, improving communication efficiency and user experience. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a flowchart illustrating a preference inference model training method provided in this application.

[0020] Figure 2 This is a schematic diagram of a method for responding to users based on inferred user preferences, as provided in this application.

[0021] Figure 3 This is a schematic diagram of a preference inference model training method provided in this application.

[0022] Figure 4 This is a schematic diagram of an experimental result provided in this application.

[0023] Figure 5 This is a schematic diagram of another experimental result provided in this application.

[0024] Figure 6 This is a schematic diagram of the structure of a preference inference model training device provided in this application.

[0025] Figure 7 This is a schematic diagram of the structure of the electronic device provided in this application. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0027] LLM personalization aims to enable models to provide accurate, relevant, and useful content that aligns with a user's individual preferences when responding to their current request. This capability enhances the efficiency of communication between the user and the LLM and provides a better user experience. Therefore, accurately inferring user preferences is crucial. In practice, the user's interaction history with the LLM naturally contains rich preference signals. However, extracting preferences from these historical records remains under-researched and challenging: historical records can be long and noisy, preferences can evolve over time, and even within the same topic, preferences can vary across different contexts.

[0028] Existing methods for personalization using user interaction history mainly fall into the following three categories.

[0029] The first category is user-specific parameter adaptation methods. These methods target a specific user, using that user's historical dialogue data to perform additional training on the model or efficiently fine-tune the parameters, enabling the model to learn that user's personalized behavioral patterns. The drawback of this type of method is that it requires additional training or parameter maintenance for each user, resulting in high computational and deployment costs, making it difficult to scale to large-scale user scenarios.

[0030] The second category is retrieval-based methods. These methods segment the user's interaction history into several historical segments. When a user makes a current request, the most relevant historical segment is retrieved, and the retrieval results are used as context input to the downstream LLM (Local Language Management) to assist in generating personalized answers. The drawback of this type of method is that it relies heavily on the semantic similarity between the current request and historical segments. In long interaction histories, truly relevant preference segments may be interfered with by a large amount of irrelevant context, leading to unstable retrieval results.

[0031] The third category is memory-based or user profiling-based methods. These methods compress or extract user interaction history into user profiles, preference memories, or long-term memory banks. Relevant memories are retrieved in subsequent requests to guide the model in generating answers that match user preferences. The drawback of this type of method is that it requires pre-determining which historical information to retain before future requests occur, and it may lose the fine-grained preferences needed for subsequent requests when compressing the interaction history.

[0032] In summary, how to overcome the above problems and, based on a reasonable model training strategy, uncover user preferences is an urgent issue to be addressed.

[0033] Based on this, this application provides a method for training a preference inference model. Specifically, multiple training sample sets can be generated, including a first training sample set, a second training sample set, and a third training sample set. The first training sample set includes training samples used to distinguish preference differences in different conversational contexts and to identify outdated preference interference. The second training sample set includes training samples used to identify user negation expressions. The third training sample set includes training samples used for long dialogue history scenarios. Subsequently, a supervised fine-tuning of a basic large language model can be performed based on the multiple training sample sets to obtain a preference inference model. The preference inference model is used to infer user preferences related to the current request based on the user's interaction history with the large language model and the user's current request.

[0034] Compared with existing solutions, this solution does not require retraining for each user, nor does it rely on retrieved fragments or pre-compressed memory representations. This solution constructs multiple training sample sets covering different training objectives, achieving systematic training of user preference inference capabilities: The first training sample set includes training samples used to distinguish preference differences in different conversational contexts and identify outdated preference interference, providing the model with basic contextualized preference learning materials. This enables the model to stably capture preference signals related to the current request from the user's interaction history, effectively addressing the problem of preference differences in different contexts under the same topic; the second training sample set introduces training samples related to user negation expressions, enabling the model to understand the user's reverse preference signals and improving robustness to noisy interaction information; the third training sample set is trained for long-context scenarios, enabling the model to filter noise and locate key preference information from extremely long interaction histories, solving the problem of difficulty in preference extraction under long-history interactions. Based on this, the preference inference model obtained through supervised fine-tuning can simultaneously address multiple challenges such as historical interaction noise, dynamic evolution of preferences, scenario preference conflicts, and long context constraints during the interaction between users and the large language model. It can accurately infer user preferences from the interaction history, enabling the large language model's response to meet the user's personalized needs and improve communication efficiency and user experience.

[0035] The following is combined Figures 1 to 7 This application describes the preference inference model training method and apparatus.

[0036] Figure 1 This is a flowchart illustrating a preference inference model training method provided in this application, as shown below. Figure 1 As shown, the method includes the following steps S101-S102.

[0037] S101 generates multiple training sample sets.

[0038] Multiple training sample sets include a first training sample set (also known as the core training subset), a second training sample set (also known as the negation enhancement training subset), and a third training sample set (also known as the long context training subset).

[0039] The core training subset includes training samples for distinguishing preference differences across different conversational contexts and identifying outdated preference interference. This allows the model to be trained to infer user preferences related to user requests from interaction history. The negation enhancement training subset includes training samples for recognizing user negation expressions, training the model to recognize such expressions. The long context training subset includes training samples for long dialogue history scenarios, training the model to infer preferences in long context scenarios.

[0040] The core training subset, the negation-enhanced training subset, and the long-context training subset will be explained in detail below.

[0041] (a) Core training subset In one example, the core training subset may include multiple training samples, each of which may include a core session, a distractor session, a target request, and a target preference. The core session and the distractor session can be collectively referred to as the interaction history.

[0042] The core conversation consists of conversation content from the training samples that reflects the target preferences that match the target request.

[0043] Interference sessions may include strong interference sessions and / or weak interference sessions.

[0044] Among them, the topic of the strong interference session is the same as that of the core session. And the strong interference session can satisfy at least one of the following: the context of the strong interference session is different from that of the core session; or, the context of the strong interference session is the same as that of the core session, and the user preference reflected by the strong interference session is an outdated preference that is earlier than the target preference.

[0045] A weak interference session refers to session content that is unrelated to the target request.

[0046] This application does not impose a specific limit on the number of training samples in the core training subset. In one example, the number of training samples in the core training subset can be 10,000.

[0047] This application does not specify the number of interaction histories included in each training sample. In one instance, each training sample may include 8 interaction histories, meaning the total number of core sessions and interfering sessions is 8.

[0048] In one example, weakly interfering sessions can be sampled from the wildchat-4.8M dataset to simulate long, noisy, irrelevant content in a real interaction history.

[0049] In one example, when the training samples include both strongly interfering sessions and core sessions, generating a core training subset may include the following steps A-F.

[0050] Step A: Generate multiple conversational scenarios.

[0051] Multiple conversation scenarios correspond to the same conversation topic.

[0052] Multiple conversational scenarios include a core scenario and at least one distracting scenario.

[0053] In one example, the conversation topic can be determined, and the target context dimension can be obtained from multiple preset context dimensions. Based on the conversation topic and the target context dimension, multiple conversation contexts can be generated.

[0054] This application does not specifically limit the topic of the conversation. For example, the topic of the conversation may include, but is not limited to, email writing, food, travel, accommodation, etc.

[0055] The embodiments of this application do not specifically limit the multiple preset context dimensions. For example, context dimensions may include, but are not limited to, time, location, group, and activity.

[0056] The embodiments of this application do not specifically limit the number of generated session scenarios. For example, the number of generated session scenarios can be 3, 5, more or fewer.

[0057] This application does not specifically limit the method for determining the conversation topic in its embodiments. In one example, it can be determined from a plurality of preset conversation topics. In another example, it can be determined based on a preset dataset. Exemplarily, the preset dataset can be a generation (Gen) subset from the Infinity-Instruct dataset.

[0058] Specifically, taking the determination of conversation topics based on a preset dataset (a subset of the Infinity-Instruct Gen dataset) and multiple preset context dimensions including time, location, group, and activity as an example, and generating 3-5 conversation scenarios, the process begins by obtaining multiple initial user messages from the Infinity-Instruct Gen subset, treating each initial message as an initial user request. These initial user requests can then be filtered for security reasons, such as removing those containing harmful or unsafe content, and each filtered initial user request can be mapped to a conversation topic. For each conversation topic, a context dimension can be randomly selected from the four context dimensions (time, location, group, and activity) as the target context dimension. Finally, based on the conversation topic and the target context dimension, 3-5 conversation scenarios can be generated.

[0059] In one example, a security classification model can be used to perform security filtering on the initial user request; for example, the security classification model can be LLaMA-Guard-3-8B.

[0060] For example, taking the preset dataset as a subset of Gen in the Infinity-Instruct dataset, the conversation topic as email writing, and the context dimension as group, the electronic device can generate multiple conversation scenarios for the context dimension group, including but not limited to: writing emails to teachers, writing emails to friends, writing emails to clients, etc.

[0061] After generating multiple conversational scenarios in the above manner, one of the conversational scenarios can be identified as the core scenario, and the remaining conversational scenarios can be identified as interference scenarios.

[0062] Step B generates a first candidate preference (also known as candidate 1) and a second candidate preference (also known as candidate 2) for each conversation context.

[0063] The first candidate preference is used to construct the preference conflict between the core context and each interference context. The first candidate preference corresponding to the core context is opposite to the first candidate preference corresponding to each interference context. For example, the first candidate preference corresponding to the core context is A, and the first candidate preference corresponding to the interference context can be ¬A. Through this design, training samples with different user preferences in different conversation contexts under the same topic can be constructed.

[0064] The second candidate preference for each session context is different from the first candidate preference for that session context. In other words, the second candidate preference for each session context is an additional user preference that is different from the first candidate preference for that session context.

[0065] In this way, each session context has two candidate user preferences, thus allowing electronic devices to construct different types of sessions.

[0066] Step C: Determine the target preference allocation rule from among the preset multiple preference allocation rules.

[0067] Step D: Based on the target preference allocation rules and the first and second candidate preferences corresponding to each session context, determine the target candidate preference for each session context.

[0068] After generating first and second candidate preferences for each session scenario using the above method, in one example, a preference allocation rule can be selected from multiple preset preference allocation rules with equal selection probability as the target preference allocation rule. Alternatively, a preference allocation rule can be selected randomly from multiple preference allocation rules as the target preference allocation rule.

[0069] The target preference allocation rule is used to define how user preferences are assigned to the core context and each distracting context. The target preference allocation rule instructs the determination of the candidate preference (i.e., the target candidate preference) to be used based on the first and second candidate preferences corresponding to each session context.

[0070] In one example, multiple preference allocation rules may include several of the following: a first preference allocation rule, a second preference allocation rule, and a third preference allocation rule. The first preference allocation rule may also be called the ordinary type, the second preference allocation rule may also be called the preference conflict type, and the third preference allocation rule may also be called the preference evolution type.

[0071] The first preference assignment rule is used to indicate that the second candidate preferences corresponding to the core scenario and each interference scenario are used as their respective target candidate preferences. This first preference assignment rule is used to construct the interaction history of ordinary types.

[0072] The second preference assignment rule instructs that the first candidate preference corresponding to the core context be used as the target candidate preference for the core context, the first candidate preference corresponding to the first interference context in each interference context be used as the target candidate preference for the first interference context, and the first or second candidate preference corresponding to the second interference context (excluding the first interference context) in each interference context be used as the target candidate preference for the second interference context. This second preference assignment rule is used to construct the interaction history of preference conflict types to train the model to distinguish the preference differences of users in different conversation contexts on the same conversation topic.

[0073] The first interference scenario can be one or more interference scenarios randomly selected from multiple generated interference scenarios.

[0074] The third preference allocation rule is used to divide the core context into a pre-evolution stage and a post-evolution stage. In the pre-evolution stage, the user preference opposite to the first candidate preference corresponding to the core context (i.e., outdated preference) is used as the target candidate preference corresponding to the core context. In the post-evolution stage, the first candidate preference corresponding to the core context is used as the target candidate preference corresponding to the core context, and the second candidate preference corresponding to each interference context is used as the target candidate preference corresponding to each interference context. This third preference allocation rule is used to construct the interaction history of preference evolution types to train the model to identify changes in user preferences over time within the same conversation topic and the same conversation context.

[0075] Specifically, for multiple conversational scenarios under the selected target scenario dimension, one preference allocation rule can be selected from the first preference allocation rule, the second preference allocation rule, and the third preference allocation rule as the target preference allocation rule. Based on this target preference allocation rule, corresponding target candidate preferences are selected for the core scenario and each interference scenario.

[0076] Step E: Based on the core context and the corresponding target candidate preferences, generate the core session, target request, and target preference.

[0077] This application does not specify the number of core sessions to be generated. In one example, one or two core sessions may be generated.

[0078] The core conversations are multi-turn feedback dialogues, meaning each core conversation can include multiple rounds of dialogue. For example, each core conversation can include 2–8 rounds of dialogue. Understandably, during the dialogue, the user (not a real user) can continuously guide the Artificial Intelligence (AI)'s responses through feedback, gradually aligning the AI's answers with the preferences corresponding to the core context. Furthermore, the core conversations need to cover all aspects of the preferences corresponding to the core context and end after the user expresses satisfaction.

[0079] Specifically, after determining the target candidate preferences corresponding to the core context through the above method, one or two core sessions can be generated based on the core context and its corresponding target candidate preferences, as well as target requests that match the core context and the session topics corresponding to the core context, and the target candidate preferences corresponding to the core context can be used as target preferences.

[0080] It is understandable that, when the target preference allocation rule is the third preference allocation rule, the target preference is the target candidate preference corresponding to the core situation in the later stage of evolution.

[0081] It is understandable that a session generated based on the core context and the target candidate preference corresponding to the core context in the pre-evolution stage (i.e., user preference that is opposite to the first candidate preference corresponding to the core context) can be regarded as a strong interference session. At this time, the session context corresponding to the strong interference session is the same as the session context corresponding to the core session, and the user preference reflected by the strong interference session is an outdated preference that is earlier than the target preference.

[0082] Step F: Generate a strong interference session based on each interference scenario and the target candidate preference corresponding to each interference scenario.

[0083] This application does not impose a specific limit on the number of strongly interfering sessions generated. For example, four, five, fewer, or more strongly interfering sessions can be generated. In one example, the number of strongly interfering sessions can be sampled from a preset set of numbers. Exemplarily, the preset set of numbers may include 1 to 6.

[0084] A highly disruptive session is a multi-turn feedback dialogue, meaning each highly disruptive session can include multiple rounds of dialogue; for example, each highly disruptive session can include 2–8 rounds of dialogue. Understandably, during the dialogue, the user can continuously guide the AI's responses through feedback, gradually aligning the AI's answers with specified preferences. Furthermore, a highly disruptive session must cover all aspects of the user's assigned preferences and end after the user expresses satisfaction.

[0085] In one example, after determining the number of highly disruptive sessions, these sessions can be allocated to at least one of the various disruptive scenarios. For instance, a multivariate sampling method can be used to allocate the number of highly disruptive sessions to at least one of the various disruptive scenarios.

[0086] It is understandable that some interference scenarios may not require the generation of strongly interfering sessions.

[0087] In one example, after determining each interference scenario, the target candidate preference corresponding to each interference scenario, and the number of strong interference sessions allocated to each interference scenario in the above manner, a corresponding number of strong interference sessions can be generated based on each interference scenario and the target candidate preference corresponding to each interference scenario, according to the number of strong interference sessions allocated to each interference scenario.

[0088] In one example, in order to orderly group core sessions and strong interference sessions, the electronic device may include multiple session slots. After the core sessions and strong interference sessions are generated as described above, the core sessions and strong interference sessions can be stored in multiple session slots, with each session containing one session.

[0089] This application does not specify the number of session slots in the embodiments; for example, the number of session slots can be 8.

[0090] It is understandable that when the number of session slots is greater than the number of core sessions and strong interference sessions, after storing the core sessions and strong interference sessions in the session slots, the remaining session slots can be filled with weak interference sessions.

[0091] It is understood that, under the third preference allocation rule mentioned above, the outdated preference sessions generated in the pre-evolution stage are located before the core sessions generated in the post-evolution stage; the outdated preference sessions serve as strong interference sessions and are used to train the model to identify the latest applicable preferences.

[0092] In one example, in order to eliminate excessively long and redundant samples and standardize the length of training samples, after obtaining each training sample, the length of each training sample can be determined, and training samples with a length greater than a preset length threshold can be deleted.

[0093] This application does not specifically limit the preset length threshold. In one example, the preset length threshold can be 8192. Accordingly, after obtaining each training sample, training samples with a length exceeding 8192 tokens can be deleted to obtain the core training subset.

[0094] (b) Negation-enhanced training subset The negation-enhanced training subset is a set of training samples used to improve the model's sensitivity and robustness to user negation expressions. The purpose of the negation-enhanced training subset is to enable the final preference inference model to accurately identify the user's true preferences from the user's negative feedback and avoid generating answers that conflict with the user's explicitly rejected preferences.

[0095] This application does not impose a specific limit on the number of training samples in the negative reinforcement training subset. In one example, the number of training samples in the negative reinforcement training subset can be 4,000.

[0096] In one example, each training sample in the negative reinforcement training subset may include one core session and seven weak interference sessions. This application does not specifically limit the length of each training sample in the negative reinforcement training subset. In one example, the total length of each training sample in the negative reinforcement training subset does not exceed 8192 tokens.

[0097] This application does not specifically limit the construction type of training samples in the negation-enhanced training subset. In one example, the negation-enhanced training subset may include feedback-based negation-enhanced data and data that directly expresses preferences based on negation forms.

[0098] This application does not specifically limit the number of feedback-based negative enhancement data and the number of data that directly express preferences based on negative forms. In one example, the number of feedback-based negative enhancement data can be 3000, and the number of data that directly express negative preferences can be 1000.

[0099] The two construction types will be explained separately below.

[0100] Feedback-based negative reinforcement data refers to training data that indirectly expresses negative preferences through user (non-real user) feedback to AI responses. Each piece of feedback-based negative reinforcement data consists of a user request and two contrasting preference definitions. These two contrasting preference definitions include a primary preference and a secondary preference; the primary preference is the mainstream or more common preference, and the secondary preference is a less mainstream preference. The corresponding core conversation is a short, feedback-based dialogue where the user makes a request, the AI ​​provides an answer, and the user then provides feedback. This feedback can explicitly favor the secondary preference and reject the primary preference, or it can simply express dissatisfaction with the primary preference. The target preference in feedback-based negative reinforcement data is directly obtained from user feedback, rather than inferred by comparing candidate options.

[0101] For example, taking email writing as an example, a user requests help to write an email, with a primary preference for a formal business email style and a secondary preference for a casual email style. The AI's initial response adopts a formal business email style. The user then provides feedback, expressing dissatisfaction with the formal style or a preference for a casual style, such as, "This email is too formal; I don't really like this style," or "I actually prefer a more casual feel." Based on this feedback, the target preference can be directly determined as a casual email style.

[0102] Data based on direct expression of preference through negation refers to training data where users explicitly express their dislike by using negative expressions directly in their requests. This type of data conveys negative preference signals through direct negation.

[0103] For example, when sending a user request, a user could specify that they want the user to write an email, but avoid overly formal or official language, or recommend tourist destinations, avoiding overly crowded popular tourist spots. This content directly identifies the user's preferred content that they explicitly reject.

[0104] The training data for the negation-enhanced training subset can contain lexical cues of varying intensities. In one example, these lexical cues can include both strong and implicit lexical cues. For instance, strong lexical cues could include, but are not limited to, "prefer" and "avoid," while implicit lexical cues could include, but are not limited to, "find…a bit uninteresting" and "I'll go for…." Through this negation-enhanced training containing lexical cues of varying intensities, the preference inference model can better identify genuine preferences from negative user feedback, reducing the risk of misunderstanding negation expressions.

[0105] By setting a negative reinforcement training subset, the model can fully learn various negative expression features, accurately identify true preferences from negative user feedback, and reduce the probability of the model misunderstanding negative expressions.

[0106] (c) Long context training subset The long context training subset is a set of training samples used to improve the model's preference inference ability in long context scenarios. The purpose of the long context training subset is to enable the preference inference model to accurately locate and infer user preferences when dealing with very long dialogue history between users and LLM in real applications.

[0107] In one example, the long context training subset may include two subsets, such as a first long context subset and a second long context subset. This application does not specifically limit the number of training samples in each long context subset; in one example, the number of training samples in each long context subset may be 500.

[0108] The lengths of the various long context subsets can be the same or different. In one instance, the length of the first long context subset can be 65,536 tokens, and the length of the second long context subset can be 131,072 tokens.

[0109] This application does not specifically limit the construction method of the long context training subset. In one example, the long context training subset can be generated based on the construction process of the negative reinforcement training subset; the length of each basic training sample does not exceed 8192 tokens. Then, dialogue content is randomly sampled from the dialogue dataset between real users and the large language model, and the sampled dialogue content is appended to the conversation in the basic training samples until the basic training samples reach the preset long context length range, thereby obtaining the long context training samples. By adding weakly relevant or irrelevant content in the interaction history, the context length and noise level of the training data are made closer to the historical length accumulated by real users after long-term interaction, thereby enhancing the robustness of the model in extracting and inferring user preferences in long contexts.

[0110] By setting a long-context training subset, the model's ability to capture user preferences in massive amounts of conversational information can be improved, ensuring the accuracy of preference inference in long-context scenarios.

[0111] In one example, the above S101 can be executed through a preset model, such as generating a conversation topic, a conversation context, generating a first candidate preference and a second candidate preference for each conversation context, generating a core conversation and a highly disruptive conversation, etc.

[0112] In one example, the preset model can be Generative Pre-trained Transformer (GPT) - Open Source Software (OSS) - 120B.

[0113] S102, based on multiple training sample sets, performs supervised fine-tuning of the basic large language model to obtain the preference inference model.

[0114] Preference inference models are used to infer user preferences related to the current request based on the user's interaction history with the large language model and the user's current request.

[0115] After obtaining the core training subset, negation-enhanced training subset, and long-context training subset through the above methods, these subsets can be merged to obtain the final training sample set. Then, using this final training sample set, a pre-defined fine-tuning technique can be applied to supervised fine-tuning of the basic large language model to obtain the preference inference model, PrefInfer.

[0116] In one example, the underlying large language model could be the Qwen3-4B-Instruct-2507 model.

[0117] In one example, the preset fine-tuning technique can be Low-Rank Adaptation (LoRA).

[0118] In one example, during fine-tuning, training samples of similar length can be grouped into the same batch to reduce the number of padding tokens and improve training efficiency.

[0119] In one example, each training sample in this scheme is represented in English.

[0120] This application does not impose specific limitations on the hardware configuration used for model training. In one example, the global batch size for supervised fine-tuning of the basic large language model can be set to 8, and training can be performed based on 4 NVIDIA A800 graphics processing units (GPUs).

[0121] After training the preference inference model using the above technical solution, this method can be applied to real-world human-computer interaction scenarios. In practical applications, when a user request is received at the current moment, the system can obtain the user's interaction history with the large language model and accurately infer the user preferences that match the current user request from this history. Then, the electronic device can generate a response that meets the user's personalized needs based on the inferred user preferences, thereby completing the current round of human-computer interaction.

[0122] Suppose the user request input at the current moment is q, and the user's interaction history with the Large Language Model (LLM) is H={s1,...,sn}. Each session si contains one or more rounds of dialogue between the user and the LLM. The electronic device can infer the user preference p that matches the user request q from the interaction history H={s1,...,sn}. The user preference p can be a concise natural language description; for example, the user preference p can be a preference for traditional methods while avoiding external dependencies. Then, based on the user preference p, the electronic device can generate a response that meets the user's personalized needs, thus completing the current round of human-computer interaction.

[0123] For example, refer to Figure 2 As shown, it is assumed that the user's interaction history with the large language model includes Figure 2 The interaction history shown allows the electronic device to infer user preferences such as "polite, formal, and concise" that match the user's current request, "Write an email to my tutor to schedule a meeting." The electronic device can then generate... Figure 2 The response shown matches the user's preference for "polite, formal, and concise": "Okay. I will maintain a polite, formal, and concise style..." Through the above technical solution, during model training, core conversations and highly disruptive conversations are generated based on conversation topics, conversation contexts, and user preferences assigned to each conversation context. This ensures that the generated core and highly disruptive conversations can reliably reflect the corresponding user preferences without requiring manual annotation of each real-world interaction. This effectively improves training efficiency and saves labor costs. Furthermore, this solution does not require retraining for each user, nor does it rely on retrieved fragments or pre-compressed memory representations. It can infer user preferences from the user's interaction history with the large language model and the user's current request, thereby enabling more flexible preference inference and personalized responses.

[0124] The preference inference model training method provided in this application will be described in detail below with reference to the above embodiments. Figure 3 A schematic diagram of another preference inference model training method provided in this application embodiment is shown below. Figure 3 As shown, the method includes: Initial user requests are obtained from the Infinity-Instruct Gen subset and mapped to session topics. A context dimension is randomly selected from four dimensions: time, location, group, and activity. Under the selected context dimension, 3 to 5 session scenarios are generated based on the session topic and context dimension. These scenarios include one core scenario and at least one interfering scenario. A first candidate preference and a second candidate preference are generated for each session scenario. A target preference allocation rule is determined with equal selection probability from the first, second, and third preference allocation rules. Based on the target preference allocation rule, the target candidate preference for each session scenario is determined. Based on the core scenario and its corresponding target candidate preference, 1 to 2 core sessions, target requests, and target preferences are generated. 1 to 6 strong interfering sessions are sampled and assigned to one or more interfering scenarios. Strong interfering sessions are generated based on the interfering scenarios assigned to the strong interfering sessions and their corresponding target candidate preferences. This yields the core training subset.

[0125] A negative reinforcement training subset and a long context training subset are generated. These three subsets are then merged to obtain the final training sample set. This final training sample set is input into the basic large language model and fine-tuned under supervision using LoRA to obtain the preference inference model, PrefInfer.

[0126] To comprehensively verify the effectiveness of the preference inference model training method provided in this application, this experiment evaluates it from two core dimensions: first, whether the trained preference inference model can accurately infer user preferences related to the current request from the interaction history; and second, whether the inferred user preferences can effectively guide the large language model to generate responses that better match user preferences. Based on the above objectives, three types of evaluation tasks are set up below, and several representative methods are selected as comparison baselines.

[0127] The three types of evaluation tasks are as follows: Direct Inference Task: Given the interaction history H and the current request q, the goal is to infer the user preference p associated with the current request q. This task directly evaluates the inference accuracy of the preference inference model itself.

[0128] Category selection task: Given the interaction history H, the current request q, and four candidate responses, the objective is to select the response that best matches the user's preferences from the candidates. This task evaluates the auxiliary effect of preference information on response selection.

[0129] Response Generation Task: Given the interaction history H and the current request q, the goal is to generate a response consistent with the user's preferences. This task evaluates the effect of preference information on improving the quality of response generation.

[0130] The following comparison of the three types of evaluation tasks is based on four methods: fully interactive historical input method, retrieval method, long-term personalized multimodal LLMs (PersonaVLM), and this solution: Full interactive history input method: Three basic large language models with different parameter sizes are selected as baselines: Model A (approximately 4B parameters), Model B (approximately 35B parameters, using a hybrid expert architecture), and Model C (large-scale general model). In all settings, the model receives the complete interaction history H and the current request q as input, without performing additional preference inference or information filtering.

[0131] Retrieval Method: For each current request q, retrieve the top 10 dialogues with the highest semantic similarity to q from the interaction history H. Concatenate the retrieval results with the current request q in chronological order and input them into the model B described above. This method is used to compare the impact of information compression and filtering on preference inference.

[0132] PersonaVLM: As a representative memory-based approach, PersonaVLM maintains a user-centric memory database and supports categorization selection and response generation tasks by retrieving relevant memories. This method is used to compare the effectiveness of different memory mechanisms in supporting preference adherence.

[0133] This approach: In direct inference tasks, the preference inference model PrefInfer, trained using this approach, is directly used to infer user preference p based on the interaction history H and the current request q. In classification selection and response generation tasks, p is first inferred by PrefInfer, and then p and q are input together into the corresponding downstream large language model (i.e., model A, model B, or model C mentioned above). This setting is used to compare whether providing inferred preferences is more effective than directly providing the complete interaction history.

[0134] This experiment uses two publicly available datasets for evaluation: the Contextual User Preference Inference Dataset (CUPID) and the Preference Evaluation (PrefEval) dataset.

[0135] The CUPID dataset supports two task settings: direct inference and response generation, used to evaluate whether a model can identify preferences relevant to the current request. The dataset contains three subsets: a consistent subset to evaluate cases of consistent preferences, a contrastive subset to evaluate cases of conflicting preferences in different contexts, and a changing subset to evaluate cases where preferences change within the same context. Through this subset design, CUPID can comprehensively cover different preference expression patterns.

[0136] The PrefEval dataset is used to evaluate preference adherence under long dialogue histories, including two tasks: classification selection and response generation. It is constructed by inserting different numbers of unrelated dialogue contexts between the dialogue containing preferences and the current request, thus simulating the continuously growing interaction history in real-world scenarios. This experiment uses three settings: 25 rounds, 150 rounds, and 300 rounds, corresponding to interaction history lengths of approximately 9k, 54k, and 103k tokens, respectively. Regarding preference types, PrefEval includes a subset of explicit preferences and two subsets of implicit preferences: implicit choice-based preferences and implicit persona-driven preferences. It should be noted that the training data in this scheme does not directly construct these two types of implicit preference patterns; therefore, the PrefEval dataset can be used to test the model's generalization ability in implicit preference adherence.

[0137] Experimental results refer to Figure 4 and Figure 5 As shown. Figure 4 and Figure 5 A higher value indicates better performance. Figure 5 The overall results are averaged across seven settings (CUPID for generating responses and PrefEval for categorization selection and six settings for generating responses across three interaction history lengths), with the CUPID generated response results being averaged after being scaled by 10. PersonaVLM lags behind other methods in 9k interaction history lengths; to save testing time, its performance in 54k and 103k interaction history lengths was not tested.

[0138] Based on a comprehensive analysis of the experimental results, this solution has at least the following technical advantages: First, improve the accuracy of preference inference. For example... Figure 4As shown, the PrefInfer method of this application achieved a precision of 68.12%, a recall of 67.52%, and an F1 score of 67.82% in the direct inference task, significantly outperforming the full interaction history input method and the retrieval method. This indicates that the proposed method can effectively identify user preferences related to the current request from the interaction history and has high inference accuracy.

[0139] Secondly, improve the preference alignment capability of downstream models. For example... Figure 5 As shown, after inputting the preferences inferred by PrefInfer as conditional information into the downstream large language model, the performance of each baseline model in classification selection and response generation tasks was significantly improved. Taking model A (approximately 4B parameters) as an example, after adding PrefInfer, its overall average score increased from 54.86 to 89.12. This indicates that the preferences inferred by this method can serve as compact and explicit conditional information, effectively guiding the downstream model to generate responses that better match user preferences.

[0140] Third, it adapts to scenarios with long interactive histories. For example... Figure 5 As shown, when the interaction history length of PrefEval increases from 25 rounds (approximately 9k tokens) to 300 rounds (approximately 103k tokens), the performance of this solution combined with various downstream models, although slowly decreasing, still remains at a high level. For example, Model C (large-scale general model) + PrefInfer achieves a classification selection score of 95.17% in the 25-round setting and still maintains 90.63% in the 300-round setting. This indicates that this solution can effectively mitigate the interference of irrelevant information in long contexts on the model and has good stability in long interaction history scenarios.

[0141] Fourth, it possesses model-independent enhancement effects. For example... Figure 5 As shown, this solution can be combined with different downstream large language models, including Model A (approximately 4B parameters), Model C (a large-scale general-purpose model), and Model B (approximately 35B parameters, a hybrid expert architecture), and produces more significant gains for models with weaker capabilities. For example, after adding PrefInfer, Model A's overall average score improved by 34.26 percentage points, while the inherently stronger Model C only improved by 17.35 percentage points. This demonstrates that this solution has good versatility and deployment flexibility, and can be integrated with different downstream models as a model-independent enhancement module.

[0142] Furthermore, this approach also demonstrates good performance on the implicit preference subset of PrefEval, indicating that although the training data does not directly construct implicit preference patterns, the model still has a certain generalization ability and can capture preference information that users do not explicitly express but can infer from their behavior from the interaction history.

[0143] The preference inference model training device provided in this application is described below. The preference inference model training device described below can be referred to in correspondence with the preference inference model training method described above.

[0144] Figure 6 This is a schematic diagram of a preference inference model training device provided in an embodiment of this application, with reference to... Figure 6 As shown, the device includes: The generation module 601 is used to generate multiple training sample sets, including a first training sample set, a second training sample set, and a third training sample set; wherein, the first training sample set includes training samples for distinguishing preference differences in different conversational contexts and identifying outdated preference interference, the second training sample set includes training samples for identifying user negative expressions, and the third training sample set includes training samples for long dialogue history scenarios. The fine-tuning module 602 is used to perform supervised fine-tuning of the basic large language model based on the multiple training sample sets to obtain a preference inference model; the preference inference model is used to infer user preferences related to the current request based on the user's interaction history with the large language model and the user's current request.

[0145] Figure 7 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 7 As shown, the electronic device may include a processor 710, a communications interface 720, a memory 730, and a communication bus 740. The processor 710, communications interface 720, and memory 730 communicate with each other via the communication bus 740. The processor 710 can call logical instructions in the memory 730 to execute a preference inference model training method. This method includes generating multiple training sample sets, including a first training sample set, a second training sample set, and a third training sample set. The first training sample set includes training samples used to distinguish preference differences in different conversational contexts and identify outdated preference interference; the second training sample set includes training samples used to identify user negation expressions; and the third training sample set includes training samples used for long dialogue history scenarios. Subsequently, based on the multiple training sample sets, a supervised fine-tuning of the basic large language model can be performed to obtain a preference inference model. The preference inference model is used to infer user preferences related to the current request based on the user's interaction history with the large language model and the user's current request.

[0146] Furthermore, the logical instructions in the aforementioned memory 730 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0147] On the other hand, this application also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the preference inference model training method provided by the above methods. This method includes: generating multiple training sample sets, including a first training sample set, a second training sample set, and a third training sample set; wherein, the first training sample set includes training samples for distinguishing preference differences in different conversational contexts and identifying outdated preference interference, the second training sample set includes training samples for identifying user negation expressions, and the third training sample set includes training samples for long dialogue history scenarios. Subsequently, based on the multiple training sample sets, a supervised fine-tuning of the basic large language model can be performed to obtain a preference inference model; the preference inference model is used to infer user preferences related to the current request based on the user's interaction history with the large language model and the user's current request.

[0148] Furthermore, this application also provides a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, this computer program is implemented to perform the preference inference model training method provided by the aforementioned methods. This method includes: generating multiple training sample sets, including a first training sample set, a second training sample set, and a third training sample set; wherein the first training sample set includes training samples for distinguishing preference differences in different conversational contexts and identifying outdated preference interference; the second training sample set includes training samples for identifying user negation expressions; and the third training sample set includes training samples for long dialogue history scenarios. Subsequently, a supervised fine-tuning of a basic large language model can be performed based on the multiple training sample sets to obtain a preference inference model; the preference inference model is used to infer user preferences related to the current request based on the user's interaction history with the large language model and the user's current request.

[0149] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0150] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0151] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A method for training a preference inference model, characterized in that, include: Multiple training sample sets are generated, including a first training sample set, a second training sample set, and a third training sample set; wherein, the first training sample set includes training samples for distinguishing preference differences in different conversational contexts and identifying outdated preference interference, the second training sample set includes training samples for identifying user negative expressions, and the third training sample set includes training samples for long dialogue history scenarios. Based on the multiple training sample sets, the basic large language model is subjected to supervised fine-tuning to obtain a preference inference model; the preference inference model is used to infer user preferences related to the current request based on the user's interaction history with the large language model and the user's current request.

2. The preference inference model training method according to claim 1, characterized in that, The first training sample set includes multiple training samples, and each training sample includes a core session, a distraction session, a target request, and a target preference; The core session is the session content in the training samples that reflects the target preference that matches the target request; the interference sessions include strong interference sessions and / or weak interference sessions; the weak interference sessions refer to session content that is not related to the target request.

3. The preference inference model training method according to claim 2, characterized in that, The strong interference session corresponds to the same session topic as the core session, and the strong interference session satisfies at least one of the following: the session context corresponding to the strong interference session is different from the session context corresponding to the core session; or, the session context corresponding to the strong interference session is the same as the session context corresponding to the core session, and the user preference reflected by the strong interference session is an outdated preference that is earlier than the target preference.

4. The preference inference model training method according to claim 2 or 3, characterized in that, In the case that the interference session includes the strong interference session, the first training sample set is generated, including: Multiple conversational scenarios are generated; each conversational scenario includes a core scenario and at least one interfering scenario; the conversational topics corresponding to the multiple conversational scenarios are the same. Generate a first candidate preference and a second candidate preference for each of the aforementioned conversation scenarios; Determine the target preference allocation rule from a set of preset preference allocation rules; Based on the target preference allocation rule, and the first candidate preference and the second candidate preference corresponding to each of the conversation scenarios, a target candidate preference is determined for each of the conversation scenarios; Based on the core context and the target candidate preferences corresponding to the core context, the core session, the target request, and the target preference are generated. The strong interference session is generated based on each interference scenario and the target candidate preference corresponding to each interference scenario.

5. The preference inference model training method according to claim 4, characterized in that, The generation of multiple session scenarios includes: Determine the topic of the conversation; Obtain the target context dimension from multiple preset context dimensions; Based on the conversation topic and the target context dimension, multiple conversation contexts are generated.

6. The preference inference model training method according to claim 4, characterized in that, The first candidate preference is used to construct a preference conflict between the core context and each of the interference contexts, wherein the first candidate preference corresponding to the core context is opposite to the first candidate preference corresponding to each of the interference contexts. The second candidate preference corresponding to each of the conversation scenarios is different from the first candidate preference corresponding to the conversation scenario.

7. The preference inference model training method according to claim 4, characterized in that, The multiple preference allocation rules include the following: The first preference allocation rule is used to indicate that the second candidate preference corresponding to the core scenario and each of the interference scenarios is used as the corresponding target candidate preference. Second preference allocation rule; The second preference allocation rule is used to indicate that the first candidate preference corresponding to the core situation is used as the target candidate preference corresponding to the core situation, the first candidate preference corresponding to the first interference situation in each interference situation is used as the target candidate preference corresponding to the first interference situation, and the first candidate preference or the second candidate preference corresponding to the second interference situation other than the first interference situation in each interference situation is used as the target candidate preference corresponding to the second interference situation. The third preference allocation rule is used to indicate that the core context is divided into a pre-evolution stage and a post-evolution stage. In the pre-evolution stage, the user preference that is opposite to the first candidate preference corresponding to the core context is taken as the target candidate preference corresponding to the core context. In the post-evolution stage, the first candidate preference corresponding to the core context is taken as the target candidate preference corresponding to the core context, and the second candidate preference corresponding to each interference context is taken as the target candidate preference corresponding to each interference context.

8. A preference inference model training device, characterized in that, include: A generation module is used to generate multiple training sample sets, including a first training sample set, a second training sample set, and a third training sample set; wherein, the first training sample set includes training samples for distinguishing preference differences in different conversational contexts and identifying outdated preference interference, the second training sample set includes training samples for identifying user negative expressions, and the third training sample set includes training samples for long dialogue history scenarios. The fine-tuning module is used to perform supervised fine-tuning of the basic large language model based on the multiple training sample sets to obtain a preference inference model; the preference inference model is used to infer user preferences related to the current request based on the user's interaction history with the large language model and the user's current request.

9. The preference inference model training device according to claim 8, characterized in that, The first training sample set includes multiple training samples, and each training sample includes a core session, a distraction session, a target request, and a target preference; The core session is the session content in the training samples that reflects the target preference that matches the target request; the interference sessions include strong interference sessions and / or weak interference sessions; the weak interference sessions refer to session content that is not related to the target request.

10. The preference inference model training device according to claim 9, characterized in that, The strong interference session corresponds to the same session topic as the core session, and the strong interference session satisfies at least one of the following: the session context corresponding to the strong interference session is different from the session context corresponding to the core session; or, the session context corresponding to the strong interference session is the same as the session context corresponding to the core session, and the user preference reflected by the strong interference session is an outdated preference that is earlier than the target preference.

Images