Double-source experience-based agent policy self-evolution method, system and electronic device

Abstract

Embodiments of the present disclosure provide an agent policy self-evolution method and system based on double-source experience and an electronic device. The agent autonomously triggers services based on current state data, without relying on explicit instructions from target users, thereby achieving active user care and improving interaction efficiency. On this basis, a first candidate experience set is obtained by searching from an intent closed loop source database according to the current state data. A target action instruction is generated by the agent based on the first candidate experience set. A result feedback is obtained after the target action instruction is executed. The utility value of a target experience item is updated according to the result feedback. The target experience item is updated in the intent closed loop source database according to the updated utility value. The agent can learn from historical active care experience, improve interaction efficiency, and simultaneously manage experience from different sources differently, thereby improving the interaction accuracy of the agent.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, and in particular to a method, system, and electronic device for the self-evolution of intelligent agent policies based on dual-source experience. Background Technology

[0002] With the rapid development of large-scale language model technology, LLM-based agents have been widely applied in various human-computer interaction scenarios such as life assistants, emotional companionship, health management, task execution, and robotics. These agents typically possess natural language understanding and generation capabilities, and can provide services to users through dialogue or command execution.

[0003] In most current commercialized intelligent agents, the user issues commands and the agent responds. However, in real-life scenarios, users often don't actively seek help when they are fatigued, anxious, or busy, but their physiological signals, behavioral patterns, and environmental characteristics already indicate a need for intervention. In related technologies, because intelligent agents can generally only passively wait for user commands to interact, the interaction efficiency is reduced. Summary of the Invention

[0004] The main objective of this disclosure is to propose a method, system, and electronic device for agent policy self-evolution based on dual-source experience, in order to solve the above-mentioned problems.

[0005] To achieve the above objectives, a first aspect of this disclosure proposes an agent policy self-evolution method based on dual-source experience, comprising: Acquire trigger data, which includes at least one of the target user's real-time perception data, environmental data, and rule-triggered conditions; The intelligent agent is invoked to generate current state data based on the trigger data, wherein the intelligent agent autonomously triggers services based on the current state data without relying on explicit instructions from the target user; The first candidate experience set is obtained by retrieving from the intent closed-loop source database based on the current state data. The intent closed-loop source database is used to store experience entries. The experience entries are identified by source type. The source type is used to indicate whether the experience entry is generated by the agent in response to the explicit instructions of the target user's history or by the agent autonomously triggering a service. The agent is invoked to generate a target action instruction based on the first candidate experience set, and the result feedback after executing the target action instruction is obtained. The utility value of the target experience entry is updated based on the feedback of the results, and the target experience entry is updated in the intent closed-loop source database based on the updated utility value.

[0006] In some embodiments, invoking the agent to generate a target action instruction based on the first candidate experience set includes: The intelligent agent is invoked to extract memory information, and an event object is output based on the first candidate experience set, the current state data, and the memory information. When the event object indicates that an intent needs to be created, the current user intent corresponding to the real-time sensing data is created; Based on the current state data and the current user intent, a second candidate experience set is obtained by retrieving from the intent closed-loop source database; The preferred experience item is determined from the second candidate experience set; The agent is invoked to generate target action instructions based on the preferred experience entries.

[0007] In some embodiments, determining the preferred experience entry from the second candidate experience set includes: Obtain a preset set of avoidable actions, and remove experience entries related to the avoidable action set from the second candidate experience set; Action clustering is performed on the second candidate experience set after elimination to obtain multiple cluster subsets; For each of the cluster subsets, an average utility value is calculated, and the subsets are sorted according to the average utility value. The preferred experience item is then determined based on the sorting result.

[0008] In some embodiments, updating the utility value of the target experience item based on the result feedback includes: For each step in the process of completing the current user intent, the current reward is determined based on the result feedback, and the utility value of the target experience item is updated in real time based on the current reward to obtain the updated utility value; After the current user intent is completed, a cumulative reward is determined based on the current reward for each step in the process of completing the current user intent, and the updated utility value for each step in the process of completing the current user intent is corrected based on the cumulative reward.

[0009] In some embodiments, the agent policy self-evolution method further includes: For each step in the process of completing the current user intent, the utility value of the original default policy corresponding to the current state data is obtained. When the corrected updated utility value exceeds the sum of the utility value of the original default policy and the first preset threshold, and the number of accesses to the policy corresponding to the target action instruction reaches the minimum sample size, the policy corresponding to the target action instruction is configured as the new default policy corresponding to the current state data. When the current reward corresponding to the new default strategy is less than the second preset threshold N times consecutively, the default strategy corresponding to the current state data is restored to the original default strategy.

[0010] In some embodiments, the outcome feedback includes at least two of the following: explicit evaluation, emotional shift, changes in vital signs, conversation duration, and follow-up interval. Determining the current reward based on the outcome feedback includes: Determine the weights of at least two of the following dimensions: explicit evaluation, emotional shift, changes in vital signs, conversation duration, and follow-up interval. Normalize at least two of the explicit ratings, the emotional shift, the changes in vital signs, the duration of the conversation, and the follow-up interval to obtain normalized scores for each of the explicit ratings, the emotional shift, the changes in vital signs, the duration of the conversation, and the follow-up interval. The current reward is determined based on the normalized score and the corresponding dimensional weight.

[0011] In some embodiments, the agent policy self-evolution method further includes: When the result feedback includes an explicit instruction from the target user, an explicit instruction experience entry is created in the intent closed-loop source database.

[0012] In some embodiments, the explicit indication experience entry is configured with a retrieval bias that gives the explicit indication experience entry a higher retrieval ranking priority than the remaining experience entries in the intent closed-loop source database, given the same utility value. The explicit indication indicates that the initial utility value of the experience entry is greater than the initial utility value of the remaining experience entries in the intentional closed-loop source database; The learning rate of the explicitly indicated experience entry is lower than the learning rate of the remaining experience entries in the intended closed-loop source database.

[0013] In some embodiments, the intent closed-loop source database includes a user intent closed-loop source database and an agent intent closed-loop source database. The user intent closed-loop source database and the agent intent closed-loop source database are not physically isolated and are distinguished by the source type of the same database to achieve dual source.

[0014] In some embodiments, the agent policy self-evolution method further includes: When the first candidate experience set is obtained through third-party experience accumulation, the utility value of the target experience item is not updated, and the result feedback is stored as an associated memory in the target experience item.

[0015] To achieve the above objectives, a second aspect of this disclosure proposes an agent policy self-evolution system based on dual-source experience, wherein the system executes the agent policy self-evolution method described in the first aspect embodiment.

[0016] To achieve the above objectives, a third aspect of this disclosure provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the agent policy self-evolution method described in the first aspect embodiment.

[0017] To achieve the above objectives, a fourth aspect of the present disclosure provides a storage medium, which is a computer-readable storage medium storing a computer program that, when executed by a processor, implements the agent policy self-evolution method described in the first aspect embodiment.

[0018] The beneficial effects of the embodiments disclosed herein include: By acquiring trigger data, the agent generates current state data based on the trigger data. The agent autonomously triggers services based on this current state data, without relying on explicit commands from the target user, thus achieving proactive user care and improving interaction efficiency. Based on this, the agent retrieves a first candidate experience set from the intent closed-loop source database according to the current state data. The agent then generates a target action command based on this first candidate experience set, obtains feedback after executing the target action command, updates the utility value of the target experience item based on the feedback, and updates the target experience item in the intent closed-loop source database based on the updated utility value. Since the intent closed-loop source database stores experience items, these items are identified by their source type. The source type indicates whether the experience item was generated by the agent in response to a historical explicit command from the target user or by the agent autonomously triggering a service. This allows the agent to learn from historical proactive care experiences, determining which proactive strategies are more likely to be accepted by the user and produce positive effects in different states and at different times, improving interaction efficiency. Simultaneously, it enables differentiated management of experiences from different sources, thereby improving the agent's interaction accuracy. Attached Figure Description

[0019] Figure 1 A schematic diagram of the overall process of the agent strategy self-evolution method provided in the embodiments of this disclosure; Figure 2 The overall system architecture diagram provided for embodiments of this disclosure; Figure 3 An intent lifecycle state transition diagram provided for embodiments of this disclosure; Figure 4 A flowchart of a dual-source empirical RL engine retrieval application provided in this embodiment of the disclosure; Figure 5 A timing diagram of the evolution of a dual-source experiential RL engine-general process learning provided in the embodiments of this disclosure; Figure 6 A timing diagram for a dual-source experience RL engine with user explicit instruction for special subclass processing provided in this embodiment of the disclosure; Figure 7 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this disclosure. Detailed Implementation

[0020] The accompanying drawings in the embodiments clearly and completely describe the technical solutions in the embodiments of this disclosure. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. Based on the embodiments in this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0021] It is understood that when the above embodiments of this disclosure are applied to specific products or technologies, permission or consent from the subject is required, and the collection, use and processing of related data must comply with relevant laws, regulations and standards.

[0022] Furthermore, when this embodiment of the disclosure needs to retrieve relevant data, it will obtain separate permission or separate consent for the relevant data through pop-up windows or redirection to a confirmation page. After clearly obtaining separate permission or separate consent for the relevant data, it will then obtain the necessary relevant data for enabling this embodiment of the disclosure to operate normally.

[0023] In this disclosure, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0024] Reference Figure 1 , Figure 1 This is a schematic diagram of the overall process of the agent policy self-evolution method provided in this embodiment of the disclosure. The above method can be executed by a terminal or by a server, and includes the following steps S101 to S105: Step S101: Obtain trigger data.

[0025] In this step, the triggering data includes at least one of the following: real-time perception data of the target user, environmental data, and rule-based triggering conditions. The system acquires multimodal real-time perception data of the target user through multiple sensors. Real-time perception data refers to multi-dimensional data reflecting the user's current state and environmental conditions, collected in real time through various sensing devices during user interaction. Specifically, real-time perception data may include one or more of the following: voice, vision, vital signs, spatiotemporal information, etc.

[0026] Voice data refers to the user's voice signal collected through a microphone, including the content of the user's speech, tone of voice, and emotional characteristics, such as sighs, laughter, and changes in speech rate. Voice data not only contains semantic information but also rich paralinguistic information, which can be used to infer the user's emotional state.

[0027] Visual data refers to image or video information captured by a camera, including facial expressions, body movements, and activity levels. Facial expressions can include frowning, smiling, and closing eyes; body movements can include gestures and postures; and activity levels can include working, resting, and exercising. Visual data can be used to extract features such as facial key points and posture key points from the user's image.

[0028] Vital signs data refers to physiological indicators collected from users through wearable devices or non-contact sensors, including heart rate, heart rate variability, respiratory rate, skin conductance, body temperature, and blood oxygen saturation. Vital signs data can objectively reflect a user's physiological arousal level and stress state.

[0029] Spatiotemporal data refers to the location information and current time information of a user obtained through a positioning system and a clock, including the type of space the user is in, specific location coordinates, current time, day of the week, etc. The type of space can be a bedroom, study, office, outdoors, etc.

[0030] Environmental data refers to the physical parameters of the user's environment collected by environmental sensors, including ambient temperature, humidity, light intensity, noise level, and air quality. Environmental data is crucial for understanding the environmental context in which a user is situated.

[0031] The triggering condition for the rule is a pre-set rule, such as a timer set on the terminal device. This timer can be configured by the parent using a mobile app, such as putting the child to bed at 9 pm.

[0032] The collection of the aforementioned trigger data should follow the principle of minimum necessity, collecting only the data necessary to realize the interactive function of the intelligent agent, and obtaining explicit authorization from the user before collection.

[0033] By collecting multimodal trigger data, a comprehensive information foundation is provided for subsequent state perception, enabling the intelligent agent to accurately understand the user's current state.

[0034] It should be noted that the embodiments disclosed herein do not require all types of trigger data to exist simultaneously. Even with limited data sources, such as only one microphone available, the user's state can still be inferred from the semantic and acoustic features in the speech recognition results. These acoustic features could include, for example, tone of voice, speech rate, and pause patterns. The dimensions and precision of the state description can be dynamically adjusted based on the available data sources. In actual deployment, if a certain type of data source is temporarily unavailable, for example, if the user is not wearing a wearable device, data for that dimension will be ignored in subsequent state generation steps without affecting the normal execution of the overall process.

[0035] Step S102: Invoke the agent to generate current state data based on the trigger data.

[0036] In this step, the agent autonomously triggers services based on current state data, without relying on explicit instructions from the target user. The agent performs fusion analysis and semantic processing on the trigger data collected in step S101. For example, the current state data can be state text, which can be generated as structured natural language state text using template filling. Specifically, the multimodal perception data is preprocessed, converting speech signals into text and extracting acoustic features. The user's emotional state and action features are extracted using a pre-trained facial expression recognition model and pose estimation model. Emotional states can be calm, anxious, tired, or happy, while action features can be rubbing eyes, yawning, pacing, etc. Heart rate variability and resting heart rate deviation are calculated from wearable device data. Sensor readings are converted into readable semantic descriptions.

[0037] Next, the results of the multi-source analysis are dynamically spliced ​​according to a preset template. Taking the sleep guidance scenario as an example, an example of the status text corresponding to the current status data can be: "User A, 30 years old, resting heart rate 72. Current time: Tuesday afternoon 3:00 PM, location: study. The user is working, and the voice recognition detects a sigh and rapid keyboard typing. The visual captures a frown and rubbing of the temples. Vital signs: current heart rate 92 beats / min (high, with an upward trend in the last 5 minutes). Environment: quiet, room temperature 26℃, humidity 45%."

[0038] It should be noted that the extraction of high-order semantic features from the aforementioned multimodal real-time perception data can be achieved through sentiment computing models, facial expression recognition models, or dedicated classifiers. Those skilled in the art can select appropriate models according to the actual deployment scenario and map the confidence index output by the model to degree adverbs in natural language descriptions. For example, if the sentiment confidence is >0.7, it is described as "obvious...", otherwise it is described as "slight...".

[0039] Step S103: Retrieve from the intent closed-loop source database based on the current state data to obtain the first candidate experience set.

[0040] The intent closed-loop source database is used to store experience entries. Experience entries are identified by source type, which indicates whether the experience entry is generated by the agent in response to explicit instructions from the target user's history or by the agent autonomously triggering a service.

[0041] In this step, the state text corresponding to the current state data can be converted into an embedding vector through the Embedding model. Using the embedding vector as the query condition, semantic similarity retrieval is performed in the two experience databases mentioned above. Specifically, the retrieval method can be to calculate the cosine similarity between the query vector and the state text embedding vector of each experience entry in the database.

[0042] In some embodiments, the intent closed-loop source database may include a user intent closed-loop source database and an agent intent closed-loop source database. The user intent closed-loop source database stores experience entries generated by the agent in response to intents triggered by explicit commands in past interactions. For example, if a user said to the agent three days ago, "Play some music for me," and the agent executed the command and received feedback, the resulting experience entry would be stored in this database. The agent intent closed-loop source database stores experience entries generated by the agent autonomously triggering services in past interactions. For example, if the agent detected signs of decreased attention in a user on a weekday afternoon a week ago, proactively created a "attention maintenance" intent, and executed the corresponding action, the resulting experience entry would be stored in this database.

[0043] In one embodiment, the two databases can physically share the same storage system, but each experience entry is identified by a source type field, which takes the value "user_intent" or "agent_intent". That is, the user intent closed-loop source database and the agent intent closed-loop source database are not physically isolated; they are distinguished by the source type within the same database to achieve dual-source functionality, thereby improving retrieval efficiency. During retrieval, the system performs retrieval operations on both databases separately and merges the returned retrieval results into a first candidate experience set. During the merging process, each candidate experience entry retains its source type identifier for source-related differentiation processing in subsequent steps.

[0044] Experience entries can contain the following fields: state_text, action, reward, next_state, intent_description, source_type (values ​​are user_intent / agent_intent / user_teaching), q_value, visit_count, associated_memory (List[string]), avoid_actions (List[string]), and expired_at.

[0045] It is understandable that the user intent closed-loop source database and the agent intent closed-loop source database can also be physically isolated.

[0046] Step S104: Invoke the agent to generate the target action instruction based on the first candidate experience set, and obtain the result feedback after executing the target action instruction.

[0047] After the target action instruction is executed, the result feedback collection phase begins. The method of collecting result feedback is related to the type of action. For example, if the action is a command-based physical operation, such as playing audio or adjusting lights, the feedback is mainly reflected in the user's reaction to the operation result; if the action is a conversational output, such as asking a question or making a reminder, the feedback is reflected in the user's verbal response and behavioral changes.

[0048] Specifically, feedback can include one or more dimensions such as explicit evaluation, emotional shift, changes in vital signs, dialogue duration, and follow-up interval. Explicit evaluation refers to whether the user made a clear verbal evaluation, such as "pretty good" or "not quite right"; emotional shift refers to whether the user's emotional state has shifted; changes in vital signs refer to whether the user's physiological indicators have changed in the expected direction; dialogue duration can indicate whether the user should remain in the interaction scenario; and the follow-up interval can indicate how long afterward the user will proactively initiate a similar interaction.

[0049] In some embodiments, the action selection formula is as follows:

[0050] in: Let Q be the current Q value of action a in state s.

[0051] The coefficient for exploration is calculated using the following formula: c decreases linearly from an initial value of 0.5 to a minimum value of 0.1, ensuring sufficient exploration in the early stages and focused utilization in the later stages.

[0052] This represents the total number of interactions for this intent cluster, i.e., the total number of times this type of intent has been executed.

[0053] This represents the number of times action a has been accessed in state s.

[0054] Let N be the natural logarithm of N.

[0055] As a measure of uncertainty, The smaller the value, the larger the value, encouraging the exploration of actions that have not been fully tried.

[0056] When n(s,a)=0, the default policy is used. The default policy can be generated based on any of the following: rule templates from a general knowledge base, preliminary suggestions generated by the LLM based on user profiles and zero samples of the current state, or baseline actions output by a pre-trained general policy network. The initial Q-value of the default policy is set to 0.5, and the exploration coefficient c is set to 0.8 during cold start, decreasing linearly to 0.1 with the total number of interactions N.

[0057] Related technologies typically employ random exploration with fixed probabilities, neglecting to consider which state-action pairs have been fully explored, leading to wasted exploration opportunities on fully explored actions. In this embodiment, however, uncertainty measurement automatically focuses on actions that have not been fully explored, resulting in higher exploration efficiency, especially in scenarios with large state spaces.

[0058] Step S105: Update the utility value of the target experience item based on the result feedback, and update the target experience item in the intent closed-loop source database based on the updated utility value.

[0059] In this step, the target experience entry is the experience entry corresponding to the target action instruction, that is, the target action instruction is generated by the policy in the target experience entry. The utility value of the target experience entry can be the action reward value of reinforcement learning, i.e., the Q value. In other embodiments, it can also be other parameters that have similar effects and can measure the model training effect. This disclosure uses the action reward value as an example for explanation. The immediate reward can be calculated first based on the result feedback, and then the action reward value of the relevant state-action pair can be updated using the immediate reward. When the intent closed-loop source database includes a user intent closed-loop source database and an agent intent closed-loop source database, if the first candidate experience set belongs to the agent intent closed-loop source, the update operation is performed on the agent intent closed-loop source database; if the first candidate experience set belongs to the user intent closed-loop source, the update operation is performed on the user intent closed-loop source database.

[0060] In each interaction, a complete loop of state perception, decision-making, execution, and learning is completed, the Q-value gradually converges, and the policy is continuously optimized. As the number of interactions increases, the agent's estimation of the optimal policy for a specific user in a specific state becomes more and more accurate, thus realizing the self-evolution of personalized policies.

[0061] In some embodiments, when the first candidate experience set is obtained through third-party experience accumulation, the utility value of the target experience item is not updated. Instead, the result feedback is stored as an associated memory in the target experience item. This distinguishes between experience sources from user-initiated execution and third-party execution, preventing the policy network from being contaminated by non-user-initiated execution trajectories. The result feedback can be stored as an associated memory in the associated memory list of the target experience item.

[0062] By acquiring trigger data, the agent generates current state data based on the trigger data. The agent autonomously triggers services based on this current state data, without relying on explicit commands from the target user, thus achieving proactive user care and improving interaction efficiency. Based on this, the agent retrieves a first candidate experience set from the intent closed-loop source database according to the current state data. The agent then generates a target action command based on this first candidate experience set, obtains feedback after executing the target action command, updates the utility value of the target experience item based on the feedback, and updates the target experience item in the intent closed-loop source database based on the updated utility value. Since the intent closed-loop source database stores experience items, these items are identified by their source type. The source type indicates whether the experience item was generated by the agent in response to a historical explicit command from the target user or by the agent autonomously triggering a service. This allows the agent to learn from historical proactive care experiences, determining which proactive strategies are more likely to be accepted by the user and produce positive effects in different states and at different times, improving interaction efficiency. Simultaneously, it enables differentiated management of experiences from different sources, thereby improving the agent's interaction accuracy.

[0063] In some embodiments, an agent is invoked to generate a target action instruction based on a candidate experience set. Specifically, the agent may extract memory information, output a first event object based on a first candidate experience set, current state data, and memory information; respond to the first event object to create the current user intent corresponding to the real-time perception data; retrieve from the intent closed-loop source database based on the current state data and the current user intent to obtain a second candidate experience set; determine the preferred experience item from the second candidate experience set; and invoke the agent to generate a target action instruction based on the preferred experience item.

[0064] The role of the first candidate experience set is to provide the agent with several reference cases that are similar in state to the current state, enabling the agent to understand how it was handled in the past under similar state conditions. For example, these reference cases can be organized into cue words in the form of Few-Shot examples for the agent's large language model to reference.

[0065] The agent performs reasoning based on the above input and outputs a first event object. The first event object is a structured decision result, containing the event type, state summary, confidence score, and an optional preliminary intent description. The event type can be intent creation, recording a phase log and updating the state, or calculating final satisfaction and triggering feedback. The state summary provides a concise overview of the current state data, and the confidence score represents the agent's degree of certainty regarding its judgment.

[0066] When the first event object indicates that an intent needs to be created, a current user intent is created for this interaction, which can be specifically an intent object (IntentObject). The intent object includes intent ID (intent_id), source (source, value is user / agent), status (status), current stage (current_stage), total number of stages (total_stages), state snapshot (state_snapshot), stage log (stage_log), and final satisfaction (final_satisfaction), etc.

[0067] For example, when the system detects that a user's heart rate is high at night and is accompanied by sighing, the agent outputs an event object instructing the creation of an emotional relief intent. This intent object is then created, and its source type is marked as "agent" to distinguish it from intents actively triggered by the user.

[0068] Unlike the first search, this search includes not only the current state data but also the current user intent, making the search more accurate. Specifically, the current state data can be represented by state description text, and the current user intent can be represented by intent description text. The state text and intent description text can be concatenated to form a composite query text, which is then embedded into a vector and used to calculate similarity with experience entries in the database. In some embodiments, due to the stronger constraints of the query conditions, the similarity threshold used to retrieve the second candidate experience set can be higher than that used for the first candidate experience set, ensuring that the recalled experience is more relevant to the current task.

[0069] In some embodiments, preferred experience entries can be determined based on Q-score sorting. Once a preferred experience entry is determined, the action field stored in that entry provides a reference description of the action. However, because factors such as user state and environmental context may differ from historical experience, historical actions are not simply copied directly. Instead, the agent is invoked to combine the preferred experience entry with the current complete context to generate a specific target action instruction.

[0070] For example, suppose the action description in the preferred experience item is "play a soft piano piece for the user," but the user currently says "I'm tired of listening to piano music." The agent can adjust based on this difference and generate a new action instruction, "play natural white noise." This approach, which combines retrieval and generation, utilizes the Q-score ranking information accumulated from historical experience while preserving the generalization and adaptability of the language model.

[0071] In some embodiments, preferred experience items are determined from a second candidate experience set. Specifically, a preset set of avoidable actions can be obtained, and experience items related to the set of avoidable actions can be removed from the second candidate experience set. Action clustering is performed in the removed second candidate experience set to obtain multiple cluster subsets. For each cluster subset, an average utility value is calculated, and the subsets are sorted according to the average utility value. The preferred experience item is determined based on the sorting result.

[0072] The avoid_actions set is stored in each experience entry. When a user explicitly states in their feedback that they refuse a certain type of action, that action type is recorded in the avoid_actions list of that experience entry. For example, if a user says "It's too noisy, don't play this" after the agent plays rock music, then "play rock music" will be recorded in the avoid_actions list of that experience entry.

[0073] Before ranking the strategies, the union of the `avoid_actions` lists of all experience items in the second candidate experience set is first collected. Then, all experience items whose actions belong to this union are removed from the candidate set. This hard filtering method is more effective than suppressing an action by lowering the Q-value because hard filtering ensures that a rejected action can never be selected, while lowering the Q-value only reduces its probability of being selected. In scenarios involving explicit user rejection, hard filtering is more in line with user expectations and more efficient, as it does not require waiting for the accumulation of multiple negative feedbacks to reduce the Q-value to an effective suppression level.

[0074] After hard filtering, the candidate experience set may contain multiple experience entries that point to the same underlying action but have different expressions. For example, "play light music," "play some relaxing music," and "open a music app to play piano music" may all point to the actual operation of "playing light music." Therefore, further action clustering is performed in the second candidate experience set after elimination to avoid the same action occupying multiple positions in the ranking due to the diversity of expressions, which would affect the accuracy of the ranking results.

[0075] In one embodiment, clustering can be based on embedding vector similarity. First, the action descriptions in each experience item are encoded, and then they are aggregated using cosine similarity. Actions with similarity higher than a threshold are grouped into the same cluster.

[0076] For each cluster subset, the Q-values ​​of all experience entries within that cluster are weighted and averaged. In one embodiment, the weighting employs a time decay strategy, assigning higher weights to newer experiences and lower weights to older experiences.

[0077] After calculating the average Q-value of all clusters, sort them from highest to lowest Q-value. The empirical entry with the highest Q-value in the first-ranked cluster subset is determined as the preferred empirical entry, and the rest are output as candidate empirical entries in turn.

[0078] The above processing can effectively prevent actions that users explicitly refuse from being executed repeatedly, while eliminating the interference of diverse action descriptions on the ranking results. Combined with the time decay strategy, it can better match the recent changes in user preferences, making the selected preferred experience items more in line with the current user's actual needs, and further improving the accuracy of agent decision-making and user satisfaction.

[0079] In some embodiments, the utility value of the target experience item is updated based on the result feedback. Specifically, for each step in the process of completing the current user intent, the current reward is determined based on the result feedback, and the utility value of the target experience item is updated in real time based on the current reward to obtain the updated utility value. After the current user intent is completed, the cumulative reward is determined based on the current reward of each step in the process of completing the current user intent, and the updated utility value of each step in the process of completing the current user intent is corrected based on the cumulative reward.

[0080] The process of executing an intent can involve multiple steps. After each step of the intent execution is completed, the current reward is determined based on the feedback of the results.

[0081] The current reward is determined based on the feedback results. Specifically, the weights of at least two of the following dimensions are determined: explicit evaluation, emotional shift, changes in vital signs, dialogue duration, and follow-up interval. At least two of these dimensions are then normalized to obtain normalized scores. The current reward is determined based on the normalized scores and their corresponding weights.

[0082] Specifically, the current reward The calculation formula is as follows:

[0083] in: It is a set of actual observable dimensions, a non-empty subset of which takes the values ​​{explicit evaluation, emotional shift, changes in vital signs, conversation duration, and follow-up interval}.

[0084] The basic weights for the i-th dimension can be: explicit evaluation w1=0.35, emotional shift w2=0.25, vital sign changes w3=0.20, conversation duration w4=0.10, and follow-up interval w5=0.10.

[0085] The normalized score for the i-th dimension has a value range of [0, 1].

[0086] The normalization rules for each dimension can be: Explicit ratings can be user-generated ratings that are linearly mapped to [0,1], and the ratings can range from one star to five stars.

[0087] Emotional shift can be the change in emotional intensity based on the output of the emotion computing model. Positive shift is mapped to [0.5,1], and negative shift is mapped to [0,0.5). Positive shift can indicate emotional improvement, and negative shift can indicate emotional deterioration.

[0088] Changes in vital signs can be based on device data, with physiological indicators trending towards the normal range mapped to [0.5, 1], and deviations from the normal range mapped to [0, 0.5]. Conversation duration can be compared with historical average duration. Moderate duration is mapped to [0.5,1]. Moderate duration means that the user neither feels rushed nor feels that it is too long.

[0089] The revisit interval can be the time interval between when a user initiates a similar interaction. A shorter interval is mapped to [0.5, 1], indicating that the user is more willing to use the service again.

[0090] If there is no observable data in any dimension, then r=0.5, as a neutral value.

[0091] The action reward value for real-time updated reinforcement learning can be represented as:

[0092] in: The status text at step t. For the action selected at step t, The current reward obtained at step t. This is a discount factor, with a value range of [0,1]. The preferred value is... =0.9. The closer it is to 1, the more the agent values ​​long-term gains; The closer the value is to 0, the more short-sighted the agent. In scenarios requiring multi-stage guidance, =0.9 can achieve a good balance between short-term feedback and long-term benefits. The learning rate is updated in real time, with a value ranging from [0.05, 0.2], preferably... =0.1. A higher learning rate enables the agent to respond quickly to single-step feedback. For the next state The maximum Q value among all available actions is used to estimate the expected future return. To update the error in real time, it represents the gap between the actual return and the current estimate.

[0093] Real-time updates can adjust the Q-value estimate immediately after each interaction, thereby guiding the next policy choice. For example, in the first step of a sleep-inducing intention, the agent guides the playing of a piano piece, but the user's heart rate does not drop as expected, i.e., the reward value is low. The real-time update will immediately lower the Q-value of this state-action pair to favor trying other actions in the next policy choice.

[0094] When the intent is fully completed, such as when the user successfully falls asleep or the user explicitly indicates that they no longer need the service, the cumulative discounted reward for the entire trajectory from the start to the end of the intent can be calculated as follows: ;

[0095] in: The total number of steps of the intention, The cumulative discount reward from step t to the end of the intention. The instant reward for step k. This represents the final satisfaction score upon completion of the intended service, ranging from [0,1], and is derived from the user's overall evaluation of the entire service. This is a discount factor, with the same value as γ in the real-time update, preferably 0.9. The learning rate is used for feedback, with a value ranging from [0.01, 0.1], preferably... =0.05. A low learning rate ensures that the Q-value is not over-corrected in a single complete trajectory, avoiding drastic policy fluctuations.

[0096] By combining real-time updates with utility value correction, the Q-value can receive rapid responses at each step and be accurately corrected after the intention is completed. Using only utility value correction means the agent must wait until the intention is complete before updating the Q-value. In long-term intentions, such as travel planning and health management, the agent cannot adjust its strategy based on immediate feedback during intention execution, leading to a degraded user experience. Real-time updates ensure immediate policy adjustments after each feedback step, while utility value correction ensures that the long-term benefits of the entire trajectory are considered. The two methods complement each other.

[0097] It is important to emphasize that correcting the updated utility value and the real-time update are executed sequentially. Correcting the updated utility value occurs based on the real-time update and does not undo or roll back the results of the real-time update.

[0098] In some embodiments, for each step in the process of completing the current user intent, the utility value of the original default policy corresponding to the current state data is obtained. When the corrected updated utility value exceeds the sum of the utility value of the original default policy and a first preset threshold, and the number of accesses to the policy corresponding to the target action instruction reaches the minimum sample size, the policy corresponding to the target action instruction is configured as the new default policy corresponding to the current state data. When the current reward corresponding to the new default policy is less than the second preset threshold for N consecutive times, the default policy corresponding to the current state data is restored to the original default policy.

[0099] After each Q-value update, the default policy corresponding to the current state data is evaluated to determine whether a high-performing policy should be promoted to the new default policy, and whether a safety rollback should be triggered after the promotion. Policy promotion refers to setting the action policy that performs better in a certain state as the default preferred policy for that state, so that the agent can directly adopt this policy when encountering similar states in the future, without having to re-sort and select from the candidate set each time. Safety rollback refers to automatically reverting to the original default policy before the promotion when the newly promoted default policy subsequently performs poorly, preventing the system from making erroneous policy promotions due to accidental positive feedback, which could continuously affect the user experience.

[0100] The triggering conditions for policy improvement include two conditions, both of which must be met simultaneously. The first condition is that the corrected updated utility value exceeds the sum of the original default policy's utility value and a first preset threshold. This first preset threshold is used to prevent minor fluctuations from triggering policy improvement; only when the Q-value of the new policy is significantly better than the old policy is it considered worthwhile to improve. The second condition is that the policy corresponding to the target action instruction has been accessed at least a minimum number of samples, ensuring that the policy has undergone sufficient real-world interaction verification. When both conditions are met, the system configures this policy as the new default policy corresponding to the current state data, and this policy will be prioritized in subsequent situations with the same or similar states.

[0101] When the current reward obtained by the new default strategy is lower than the second preset threshold for N consecutive times in actual execution, it indicates that the actual effect of the strategy is far below expectations, and there may be problems with overfitting or evaluation bias. At this time, the default strategy is automatically restored to the original default strategy before the improvement, so as to avoid the continuous decline in user experience caused by the continuous use of the ineffective strategy and to ensure the robustness of strategy evolution.

[0102] The policy escalation condition can be expressed as: ( =0.15, =3) When the Q value of a certain strategy exceeds the Q value of the current default strategy plus a preset threshold δ, and the number of times the strategy is accessed reaches the minimum number of samples N_min, it will be automatically promoted to the new default strategy.

[0103] in, A threshold for policy enhancement is set, with a value ranging from [0.05, 0.3], and δ=0.15 being preferred. The larger δ is, the higher the threshold for policy enhancement, avoiding premature enhancement of policies that have not yet been fully validated.

[0104] For the minimum number of samples, exemplarily, =3. Ensure that the strategy has at least enough visits to support statistical significance before it is promoted.

[0105] If the current reward r received by the new default policy is less than 0.3 for two consecutive times, it will automatically revert to the original default policy.

[0106] For example, in a sleep guidance scenario, if the Q-value of the "play ocean wave sound" strategy reaches 0.72 after adjustment, while the Q-value of the current default strategy "play white noise" is 0.55, and the first preset threshold is 0.15, 0.72 is greater than the sum of 0.55 and 0.15, and the "play ocean wave sound" strategy has been accessed 4 times in this state, meeting the minimum sample size requirement of 3 times, "play ocean wave sound" is promoted to the new default strategy. However, in the next two interactions, this strategy only received immediate rewards of 0.2 and 0.25 respectively, falling below the second preset threshold of 0.3 for two consecutive times, triggering a safety fallback and restoring the default strategy to "play white noise".

[0107] In some embodiments, when the result feedback includes an explicit instruction from the target user, an explicit instruction experience entry is created in the intent closed-loop source database.

[0108] Explicit instructions refer to teaching commands given by the user through natural language, such as "Play some soft music for me next time you're in a bad mood" or "Don't bother me when I'm irritable." Explicit instructions are not part of ordinary interactive feedback; they are actively taught to the system by the user.

[0109] Specifically, the user's natural language expression can first be semantically parsed to extract structured information such as triggering conditions, expected actions, and prohibited actions. For example, for the explicit instruction "Play some light music for me next time you're in a bad mood, not rock music," the parsed triggering condition is "in a bad mood," the recommended action is "play light music," and the prohibited action is "play rock music." The parsed results are then encapsulated into explicit instruction experience entries, stored in the intent closed-loop source database, and the source type of each entry is marked as "user_teaching" to distinguish it from other ordinary experience entries during subsequent retrieval and processing.

[0110] It is understandable that when an agent receives an explicit instruction from the target user during the process of autonomously triggering a service, an explicit instruction experience entry is created in the agent's intent closed-loop source database; when an agent receives an explicit instruction from the target user during the process of providing a service based on user instructions, an explicit instruction experience entry is created in the user's intent closed-loop source database.

[0111] In some embodiments, the explicitly indicated experience entry is configured with a retrieval bias, which is used to give the explicitly indicated experience entry a higher retrieval ranking priority than the remaining experience entries in the intention closed-loop source database, given the same utility value; the initial utility value of the explicitly indicated experience entry is greater than the initial utility value of the remaining experience entries in the intention closed-loop source database; and the learning rate of the explicitly indicated experience entry is lower than the learning rate of the remaining experience entries in the intention closed-loop source database.

[0112] Specifically, a retrieval bias is configured for explicit instruction experience items, giving them higher priority during strategy retrieval ranking. The retrieval bias refers to adding a fixed bonus, for example, 0.3 points, to the calculated average utility value of experience items marked as "user_teaching" during the strategy retrieval ranking phase. This bias ensures that even with the same or similar Q-values, explicit instruction experience items are selected by the system before ordinary experience items. This prioritizes user teaching instructions during strategy selection, preventing them from being overwhelmed by a large number of ordinary interactive experiences.

[0113] In addition, newly created explicit instruction experience entries are assigned a higher initial Q value. The initial Q value for a regular experience entry can be set to 0.5, while the initial Q value for an explicit instruction experience entry can be set to 0.8. A higher initial Q value means that the user's explicit instructions have a higher level of trust from the moment they are created, eliminating the need for repeated trial and error (from low to high) to reach a higher policy priority, as is the case with regular experiences. This ensures that the user's explicit instructions take effect immediately.

[0114] Simultaneously, a lower learning rate is configured for explicit instruction experience items. In the dual-track Q-value update, the real-time learning rate for explicit instruction experience items can be set to 0.05, while that for ordinary experience items can be set to 0.1. During correction, the learning rate can be set to 0.02, while that for ordinary experience items can be set to 0.05. This lower learning rate results in smaller fluctuations and slower decay of the Q-value of explicit instruction experience items in subsequent updates, preventing it from being rapidly diluted by a large amount of subsequent ordinary interaction experience. Even when other strategies are repeatedly executed and various feedbacks are obtained in subsequent interactions, the user's explicit instructions can still maintain a high Q-value over a long period, continuously influencing the agent's behavioral choices.

[0115] For example, user B says, "Play some soothing music next time you're feeling down." An explicit instruction experience item is created with an initial Q-value of 0.8, a retrieval bias of 0.3, and a learning rate of 0.05. A week later, the user is feeling down and executes a strategy retrieval. The weighted Q-value of the candidate "play soothing music" is 0.75. After adding the retrieval bias of 0.3, the ranking score reaches 1.05, while the ranking scores of other candidate strategies are all below 0.8. The user ultimately chooses to play soothing music and is satisfied. In the following weeks, even when the system tries various other strategies in this state, the Q-value decay rate of the explicit instruction experience item is significantly slower because its learning rate is only half that of ordinary experience, and the instruction remains effective.

[0116] The following is a general description of the architecture of the agent policy self-evolution system based on dual-source experience provided in the embodiments of this disclosure. This system is used to execute the agent policy self-evolution method described above. (Refer to...) Figure 2 , Figure 2 The overall system architecture diagram provided for embodiments of this disclosure includes a perception input layer, a state context aggregator, an Agent Runtime, an intent lifecycle manager, a dual-source experience RL engine, a storage layer, and an Action output layer.

[0117] The perception input layer collects trigger data, including voice / text data, visual data, tactile data, environmental data, user vital signs data, and trigger data such as rules.

[0118] The state context aggregator generates natural language state text based on the trigger data provided by the perceptual input layer.

[0119] AgentRuntime extracts memories, including short-term working memory, user long-term memory, and current intent memory. First, it performs state experience retrieval to assist LLM in inferring intent. Specifically, it calls a dual-source experience RL engine to retrieve Top-K historical experiences based solely on the current state, returning a candidate experience set containing state, intent, policy, Q-value, and associated memories. Next, it performs a comprehensive LLM analysis including explicit command judgment. This is done by concatenating candidate experiences, the current state, memories, and user voice (if any), and the LLM outputs event objects (CreateIntent / SubmitStage / CompleteIntent, etc.). If the action is user-initiated, user input (voice / text / action, etc.) serves as a strong signal to assist LLM judgment. Finally, it generates an LLM policy, combining a policy template with the current context to generate specific actions, such as voice responses and device control commands.

[0120] The Intent Lifecycle Manager creates an IntentObject after receiving the CreateIntent event, records the stage log and updates the status after receiving the SubmitStage event, and calculates the final satisfaction level and triggers Q-value correction after receiving the CompleteIntent event.

[0121] The dual-source experience-based reinforcement learning (RL) engine performs two retrieval modes: Mode A, state retrieval, takes the current state text as input and outputs Top-K candidate experiences for the agent to proactively infer intent; Mode B, policy retrieval, takes the explicit intent and current state as input and outputs preferred / alternative / avoidance policies for subsequent action generation. It also performs reinforcement learning, updating in real-time in response to the SubmitStage event and updating the Q-value in response to the CompleteIntent event.

[0122] The storage layer is used to store the experience pool and the intent state library.

[0123] The Action output layer is used to implement the policy action dispatcher, environment / user state updates, and new sensory inputs.

[0124] Reference Figure 3 , Figure 3 The intent lifecycle state transition diagram provided in this embodiment can be divided into intent creation, timeout termination, user cancellation, execution failure, pause / resume intent, and normal completion.

[0125] Reference Figure 4 , Figure 4 The flowchart of the dual-source experience RL engine retrieval application provided in this embodiment of the disclosure is as follows: First, trigger data is acquired as input to generate state text. During the intent creation phase, mode A state retrieval is performed. This retrieval uses only the state text as the query condition, sets a similarity threshold, and recalls the Top-K similar experiences from the dual-source experience database (i.e., the user intent closed-loop source database and the agent intent closed-loop source database). The recall results are provided to the LLM as a Few-Shot context to assist the LLM in determining whether to create an intent and generate an intent description. After intent creation is complete, the intent execution phase begins. Mode B strategy retrieval is performed using the concatenated vector of the state text and intent description text as the query condition. A similarity threshold is set, and the Top-K candidate experiences are recalled and constrained filtering is performed to obtain a preset set of avoidable actions. Experience entries related to the avoidable action set are forcibly removed from the candidate experiences. Decisions are made based on the strategy experience, and finally, an action is executed or a response is generated. The dual-source experience RL engine learns online.

[0126] Mode A is state retrieval, with only state text as input, a similarity threshold of 0.7, and recall of the top-20 results. No constraint filtering is applied, and it provides a Few-Shot context for LLM during the intent creation phase. Mode B is policy retrieval, with state text and intent description as input, a query vector q=concat(s,i), a similarity threshold of 0.75, and recall of the top-10 results. After constraint filtering and utility reordering, the preferred / alternative / avoidance policies are output. Constraint filtering is implemented through the avoidance action set, and utility reordering is achieved through Q-values ​​and retrieval biases. Specifically, the retrieval process consists of two stages: the first stage is semantic recall, calculating cosine similarity and recalling experiences with similarity greater than the threshold. The second stage aggregates the union of avoid_actions from all recalled experiences and removes actions belonging to the avoidance policy set. The remaining experiences are clustered by action, and a weighted average Q-value (time decay weight) is calculated. If the action is explicitly indicated by the user and effective, an additional 0.3 points are added. The results are then sorted and output.

[0127] The principles of the embodiments of this disclosure are explained below using actual timing flow.

[0128] reference Figure 5 , Figure 5The dual-source experience RL engine provided in this embodiment provides a general process learning evolution sequence diagram. The Agent Runtime senses input and creates a CreateIntent event, such as "User A, 23:30, Bedroom, Heart Rate 105, Irritable, Can't Sleep," which is sent to the Intent Lifecycle Manager. The Intent Lifecycle Manager calls the dual-source experience RL engine to perform experience retrieval, semantically recalling from the dual-source experience database. If no match is found, a cold start is performed, returning an empty list. The dual-source experience RL engine sends a default policy as a policy template to the Intent Lifecycle Manager, such as "Piano music + warm yellow LED strip light + aromatherapy + air conditioning." The Agent... The runtime executes policy A based on the default policy sent by the dual-source experience RL engine. For example, this could be "play piano music, add warm yellow soft light from the LED strip, release lavender from the aromatherapy diffuser, and set the air conditioner to 24°C." It then obtains feedback on the execution result of policy A, creates a SubmitStage event (e.g., "Useless, mind too cluttered (no significant change in heart rate),"), and sends it to the intent lifecycle manager. The intent lifecycle manager calls the dual-source experience RL engine for real-time updates, creating experience entry 1 in the user intent closed-loop source database or the agent intent closed-loop source database. This entry could be "state = ..., policy = piano music ..., Q = 0.1." Simultaneously, the intent lifecycle manager calls the dual-source experience RL engine again to perform policy switching. The runtime executes strategy B based on the switched strategy, for example, "white noise + breathing guidance, light strip adjusted to dark blue, aromatherapy continued, air conditioning 22℃". It then obtains feedback on the execution result of strategy B, creates a SubmitStage event (e.g., "Breathing guidance is somewhat helpful, but white noise is annoying (heart rate drops to 95)"), and sends it to the intent lifecycle manager. The intent lifecycle manager calls the dual-source experience RL engine for real-time updates, creating experience entry 2 in the user intent closed-loop source database or the agent intent closed-loop source database (e.g., "state = ..., strategy = ..., Q = 0.3"). Simultaneously, the intent lifecycle manager calls the dual-source experience RL engine again to perform strategy switching. The runtime executes strategy C based on the switched strategy, such as "progressive muscle relaxation guidance, light strip off, aromatherapy machine off, air conditioner 22℃", then obtains the execution result feedback of strategy C, creates a CompleteIntent event, such as "user enters sleep state (heart rate drops to 85)", and sends it to the intent lifecycle manager. The intent lifecycle manager calls the dual-source experience RL engine for real-time updates, creating experience entry 3 in the user intent closed-loop source database or the agent intent closed-loop source database, such as "state=..., strategy=..., Q=0.6". At the same time, the dual-source experience RL engine corrects the Q value after the intent is completed, and further updates the Q values ​​of the aforementioned three experience entries.

[0129] Reference Figure 6 , Figure 6 The sequence diagram for the processing of a special subclass of explicit user instruction in the dual-source experience RL engine provided in this embodiment is as follows: The Agent Runtime senses the input, creates a CreateIntent event, such as "User A, 22:30, Bedroom, Feeling down", and sends it to the Intent Lifecycle Manager. The Intent Lifecycle Manager calls the dual-source experience RL engine to perform experience retrieval, performs semantic recall from the dual-source experience database, and returns a strategy suggestion after constraint filtering and utility rearrangement, such as "Rain sound". The Agent Runtime executes strategy A according to the strategy suggestion sent by the dual-source experience RL engine, such as "Play rain sound", and then obtains the execution result feedback of strategy A, creates a SubmitStage event, such as "Don't want to listen to rain sound, I want a tree hole to confide in", and sends it to the Intent Lifecycle Manager. The Intent Lifecycle Manager detects that it belongs to the explicit user instruction and creates an explicit instruction experience in the special subclass of the user intent closed-loop source database, such as "Initial Q=0.8, Learning rate=0.1, Retrieval bias=High". When the Agent Runtime subsequently detects that User A is feeling down again, the dual-source experience RL engine retrieves explicit instruction experiences and ordinary experiences. During utility reordering, due to the bias in explicit instruction experiences, they are ranked first. The dual-source experience RL engine returns a strategy suggestion, "Guide to Confess in the Tree Hole," to the Intent Lifecycle Manager. The Agent Runtime executes strategy B, "Guide to Confess in the Tree Hole," obtains feedback on the execution result of strategy B, and creates a CompleteIntent event, such as "Haha, I feel better." The Intent Lifecycle Manager calls the dual-source experience RL engine for real-time updates, updating the Q-value corresponding to the explicit instruction experience. Since the learning rate is set to 0.1, the Q-value corresponding to the explicit instruction experience slowly increases.

[0130] The following practical examples illustrate the application scenarios of the methods provided in the embodiments of this disclosure.

[0131] It should be noted that the current reward r_t, Q-value, and other values ​​in the following examples are illustrative and used to demonstrate the working principle and numerical change pattern of the dual-track Q-value update. In a real system, the reward value is dynamically calculated from multi-dimensional satisfaction, and the Q-value gradually converges with the number of interactions. Those skilled in the art can obtain parameter combinations suitable for specific scenarios through conventional methods such as grid search and Bayesian optimization.

[0132] In a sleep guidance scenario (user intent closed loop), user A tells the system "can't sleep" at 22:30 one day. The system collects the user's voice, heart rate of 92 beats / min, sighs, and a quiet environment. The system generates status text S0: "User A, 30 years old, resting heart rate 72. Current time: Tuesday 22:30, location: bedroom. User voice: 'can't sleep,' heart rate: 92 beats / min (high), sighs." Intent description I0 = "help the user fall asleep," source = user.

[0133] Select the execution strategy (Mode B retrieval) and execute the action. The following table records the complete data for each step within this intent: ; In step 1, the piano piece received no significant positive feedback from the user (r=0.2), and the Q value dropped from 0.50 to 0.48 due to real-time updates.

[0134] Step 2, with its white noise and breathing guidance, yielded a higher reward (r=0.7), and the Q value rose to 0.63.

[0135] After step 5 is completed, the Q-value is adjusted based on the true cumulative reward for the entire trajectory (G0 = 0.2 + 0.9 × 0.7 + 0.9). 2 ×0.65 + 0.9 3 ×0.8 + 0.9 4 ×0.85 ≈ 2.40) Correcting the Q value at each step, with the correction magnitude all less than 0.05, reflects the principle of no rollback and gradual correction.

[0136] Strategy Boosting Check: The action "white noise + breathing guidance" has an average Q value of 0.67 in step 2, which is greater than the default strategy Q value of 0.5 + δ(0.15) = 0.65. Furthermore, this action has been performed 3 times in similar situations (≥N_min=3), therefore it is promoted to the default strategy for this situation. The next time a user says "can't sleep" in a similar situation, white noise + breathing guidance will be the preferred strategy.

[0137] In a third-party service recommendation scenario (user intent loop + associative memory), user B requests a dentist appointment, and the agent selects dental clinic A. The user reports poor service. The agent records the associative memory ("The user is dissatisfied with the service at clinic A, complaining about long waiting times and a cold attitude from the doctor") but does not update the Q-value. Two weeks later, the user makes another appointment, and the agent retrieves the associative memory, proactively suggesting, "Your experience at clinic A wasn't good. Would you like to try clinic B? The user reviews are positive." The user accepts this suggestion and is satisfied. Afterward, when the user complains about clinic A, the agent can empathize by referencing the associative memory.

[0138] In this process, third-party execution only records associated memories and does not update the Q-value, thus preventing the policy network from being contaminated by execution trajectories other than its own. Associated memories are stored in experience entries as a List[string] and are returned along with the experience entries during subsequent retrievals. The LLM can view these historical records in the prompt.

[0139] In the proactive care scenario (Agent intent loop), the agent, through perception data, detects that user A's heart rate is high (92 beats / min, resting heart rate 72), sighing, and frowning on Tuesday afternoon. The agent proactively creates an "emotional relief" intent (intent description = "help the user relieve stress and fatigue"), dims the lights, plays soft music, and gently greets the user. The user accepts this, and their heart rate drops to 78 beats / min. On Thursday evening, the agent detects the user's fatigue and procrastination (still in the study at 10:30 PM, yawning frequently). The agent proactively reminds the user, "It's getting late, would you like to get ready to rest?" and adjusts the environmental transition (gradually dimming the lights and playing soothing sound effects), allowing the user to fall asleep smoothly.

[0140] The Agent intent closed-loop source allows the Agent to proactively initiate service processes based on state awareness without explicit user instructions. The intent creation phase uses Pattern A state retrieval to recall user acceptance and preference for proactive care under similar historical states, providing a Few-Shot context for LLM intent description generation. The Q-value update of proactive intents follows the same dual-track update mechanism as user-initiated intents, but the source type is marked as `agent_intent` for subsequent analysis of the effectiveness of the Agent's proactive services.

[0141] This disclosure also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the aforementioned intelligent agent strategy self-evolution method. This electronic device can be any intelligent terminal, including tablet computers, in-vehicle computers, etc.

[0142] Please see Figure 7 , Figure 7 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this disclosure. The electronic device includes: The processor 701 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this disclosure. The memory 702 can be implemented as a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 702 can store operating devices and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 702 and is called and executed by the processor 701 using the intelligent agent policy self-evolution method of the embodiments of this disclosure. The input / output interface 703 is used to implement information input and output; The communication interface 704 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.). Bus 705 transmits information between various components of the device (e.g., processor 701, memory 702, input / output interface 703, and communication interface 704); The processor 701, memory 702, input / output interface 703, and communication interface 704 are connected to each other within the device via bus 705.

[0143] This disclosure also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described agent policy self-evolution method.

[0144] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0145] The embodiments described in this disclosure are for the purpose of more clearly illustrating the technical solutions of this disclosure and do not constitute a limitation on the technical solutions provided by this disclosure. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by this disclosure are also applicable to similar technical problems.

[0146] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this disclosure, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0147] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0148] Those skilled in the art will understand that all or some of the steps, apparatuses, or functional modules / units in the methods disclosed above can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0149] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in this disclosure and the foregoing drawings are used to distinguish similar objects and are not necessarily used to describe a particular order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, apparatus, product, or device that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or devices.

[0150] It should be understood that in this disclosure, "at least one item" means one or more, and "more than one" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0151] In the several embodiments provided in this disclosure, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0152] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0153] Furthermore, the functional units in the various embodiments of this disclosure can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0154] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this disclosure, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this disclosure. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0155] The preferred embodiments of the present disclosure have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present disclosure. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and spirit of the present disclosure shall be within the scope of the claims of the present disclosure.

Claims

1. A method for agent policy self-evolution based on dual-source experience, characterized in that, include: Acquire trigger data, which includes at least one of the target user's real-time perception data, environmental data, and rule-triggered conditions; The intelligent agent is invoked to generate current state data based on the trigger data, wherein the intelligent agent autonomously triggers services based on the current state data without relying on explicit instructions from the target user; The first candidate experience set is obtained by retrieving from the intent closed-loop source database based on the current state data. The intent closed-loop source database is used to store experience entries. The experience entries are identified by source type. The source type is used to indicate whether the experience entry is generated by the agent in response to the explicit instructions of the target user's history or by the agent autonomously triggering a service. The agent is invoked to generate a target action instruction based on the first candidate experience set, and the result feedback after executing the target action instruction is obtained. The utility value of the target experience entry is updated based on the feedback of the results, and the target experience entry is updated in the intent closed-loop source database based on the updated utility value.

2. The agent policy self-evolution method according to claim 1, characterized in that, The step of invoking the agent to generate target action instructions based on the first candidate experience set includes: The agent is invoked to extract memory information, and an event object is output based on the first candidate experience set, the current state data, and the memory information. When the event object indicates that an intent needs to be created, the current user intent corresponding to the real-time sensing data is created; Based on the current state data and the current user intent, a second candidate experience set is obtained by retrieving from the intent closed-loop source database; The preferred experience item is determined from the second candidate experience set; The agent is invoked to generate target action instructions based on the preferred experience entries.

3. The agent policy self-evolution method according to claim 2, characterized in that, The step of determining the preferred experience item from the second candidate experience set includes: Obtain a preset set of avoidable actions, and remove experience entries related to the avoidable action set from the second candidate experience set; Action clustering is performed on the second candidate experience set after elimination to obtain multiple cluster subsets; For each of the cluster subsets, an average utility value is calculated, and the subsets are sorted according to the average utility value. The preferred experience item is then determined based on the sorting result.

4. The agent policy self-evolution method according to claim 2, characterized in that, The step of updating the utility value of the target experience item based on the result feedback includes: For each step in the process of completing the current user intent, the current reward is determined based on the result feedback, and the utility value of the target experience item is updated in real time based on the current reward to obtain the updated utility value; After the current user intent is completed, a cumulative reward is determined based on the current reward for each step in the process of completing the current user intent, and the updated utility value for each step in the process of completing the current user intent is corrected based on the cumulative reward.

5. The agent policy self-evolution method according to claim 4, characterized in that, The agent policy self-evolution method further includes: For each step in the process of completing the current user intent, the utility value of the original default policy corresponding to the current state data is obtained. When the corrected updated utility value exceeds the sum of the utility value of the original default policy and the first preset threshold, and the number of accesses to the policy corresponding to the target action instruction reaches the minimum sample size, the policy corresponding to the target action instruction is configured as the new default policy corresponding to the current state data. When the current reward corresponding to the new default strategy is less than the second preset threshold N times consecutively, the default strategy corresponding to the current state data is restored to the original default strategy.

6. The agent policy self-evolution method according to claim 4, characterized in that, The feedback results include at least two of the following: explicit evaluation, emotional shift, changes in vital signs, conversation duration, and follow-up interval. Determining the current reward based on the feedback results includes: Determine the weights of at least two of the following dimensions: explicit evaluation, emotional shift, changes in vital signs, conversation duration, and follow-up interval. Normalize at least two of the explicit ratings, the emotional shift, the changes in vital signs, the duration of the conversation, and the follow-up interval to obtain normalized scores for each of the explicit ratings, the emotional shift, the changes in vital signs, the duration of the conversation, and the follow-up interval. The current reward is determined based on the normalized score and the corresponding dimensional weight.

7. The agent policy self-evolution method according to claim 1, characterized in that, The agent policy self-evolution method further includes: When the result feedback includes an explicit instruction from the target user, an explicit instruction experience entry is created in the intent closed-loop source database.

8. The agent policy self-evolution method according to claim 7, characterized in that, The explicit instruction experience entry is configured with a retrieval bias, which is used to give the explicit instruction experience entry a higher retrieval ranking priority than the remaining experience entries in the intent closed-loop source database, given the same utility value. The explicit indication indicates that the initial utility value of the experience entry is greater than the initial utility value of the remaining experience entries in the intentional closed-loop source database; The learning rate of the explicitly indicated experience entry is lower than the learning rate of the remaining experience entries in the intended closed-loop source database.

9. The agent policy self-evolution method according to claim 1, characterized in that, The intent closed-loop source database includes a user intent closed-loop source database and an agent intent closed-loop source database. The user intent closed-loop source database and the agent intent closed-loop source database are not physically isolated, and are distinguished by the source type of the same database to achieve dual source.

10. The agent policy self-evolution method according to claim 1, characterized in that, The agent policy self-evolution method further includes: When the first candidate experience set is obtained through third-party experience accumulation, the utility value of the target experience item is not updated, and the result feedback is stored as an associated memory in the target experience item.

11. A self-evolving agent policy system based on dual-source experience, characterized in that, The system executes the agent policy self-evolution method according to any one of claims 1 to 10.

12. An electronic device, characterized in that, The electronic device includes a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, it implements the agent policy self-evolution method according to any one of claims 1 to 10.

13. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the agent policy self-evolution method according to any one of claims 1 to 10.

Images