Training method and device of intelligent agent, equipment, medium and program product
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- SHANGHAI TASHI ZHIHANG TECHNOLOGY CO LTD
- Filing Date
- 2026-05-14
- Publication Date
- 2026-07-21
AI Technical Summary
In existing technologies, the training efficiency of intelligent agents is low and the results are poor, mainly because the differences in the quality of teaching data, state distribution and time series length make it difficult for intelligent agents to learn stable operating strategies.
By acquiring teaching trajectory information, a value assessment model is used to evaluate the value of each frame of teaching image. Combined with the remaining task progress, a loss function is constructed for agent training. High-quality training samples are selected to achieve reinforcement learning of key states and steps, and to reduce the interference of redundant information.
It significantly improves the training efficiency and policy stability of intelligent agents, enhances the execution accuracy and generalization ability of complex tasks, and performs particularly well in the manipulation of flexible objects.
Smart Images
Figure CN122197949B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of artificial intelligence technology, and in particular to a training method, apparatus, device, medium, and program product for an intelligent agent. Background Technology
[0002] With the development of robot imitation learning, offline reinforcement learning, and embodied intelligence training technologies, constructing training datasets based on human teaching trajectory information has become an important technical approach for completing complex operational tasks. In long-sequence, multi-stage tasks such as folding flexible objects, the execution process of teaching trajectory information is complex, and state changes are continuous. Different operators have differences in action style, execution speed, and step sequence, resulting in significant differences in the collected teaching data in terms of duration, timing of key states, and rhythm of image changes.
[0003] In related technologies, when training an agent directly using teaching trajectory information, it is easily affected by factors such as the quality of teaching data, state distribution, and differences in time sequence length, making it difficult for the agent to effectively learn stable operating strategies, thus resulting in low training efficiency and poor training effect. Summary of the Invention
[0004] This application provides a training method, apparatus, device, medium, and program product for an intelligent agent.
[0005] In a first aspect, embodiments of this application provide a training method for an intelligent agent, applied to an electronic device. The method includes: acquiring teaching trajectory information of a first task, wherein the teaching trajectory information includes multiple teaching images used to indicate the execution process of the first task; inputting the teaching trajectory information into a value assessment model, and using the value assessment model to assess the value of each teaching image in the teaching trajectory information to obtain a value score corresponding to each teaching image in the teaching trajectory information; using the teaching trajectory information and the value scores corresponding to each teaching image in the teaching trajectory information as training data to train a first intelligent agent to obtain a second intelligent agent, wherein the second intelligent agent is used to execute the first task... The task involves the following steps: First, the training process of the value assessment model includes: acquiring N sample teaching trajectory information for the first task and the label value score corresponding to each frame of the sample teaching image in the N sample teaching trajectory information. The label value score corresponding to the j-th frame of the M-frame sample teaching image in the i-th sample teaching trajectory information is related to the remaining task progress corresponding to the j-th frame sample teaching image. N, M, i, and j are all positive integers, and i ≤ N, j ≤ M. Based on the N sample teaching trajectory information and the label value score corresponding to each frame of the sample teaching image in the N sample teaching trajectory information, the value assessment model to be trained is trained to obtain the trained value assessment model.
[0006] By performing frame-by-frame value evaluation on the teaching trajectory information, and using the teaching images and corresponding value scores together as training data to train the agent, the impact of differences in teaching data in terms of quality, state distribution, and time series length on the training process can be reduced, enabling the agent to learn more stable operating strategies and solving the problems of low training efficiency and poor training effect in related technologies.
[0007] In addition, the value assessment model is trained under supervision based on the remaining task progress, which enables the value score output by the value assessment model to objectively reflect the actual progress of the task, thereby improving the accuracy and stability of value assessment and further improving the training efficiency, policy stability and task execution capability of the agent.
[0008] In some implementations, the label value score corresponding to the j-th frame sample teaching image is negatively correlated with the remaining task progress corresponding to the j-th frame sample teaching image.
[0009] By setting the value score to be negatively correlated with the remaining task progress, the frame image that is closer to the task completion will receive a higher value score. During training, the learning of key closing steps will be automatically strengthened, which will further improve the agent's fitting accuracy to the task target state and improve the completion rate of complex tasks.
[0010] In some implementations, the N sample teaching trajectory information satisfies the following conditions: the image similarity between the last frame of the N sample teaching trajectory information and a preset reference image is greater than an image similarity threshold, wherein the reference image is the image corresponding to the environmental state when the first task is completed; and / or, the number of frames of the N sample teaching trajectory information is within a preset frame number range; and / or, the duration of the N sample teaching trajectory information is within a preset duration range.
[0011] By verifying the completion status, duration, and frame count of the sample teaching trajectory information, the effectiveness and standardization of the training samples are ensured, the interference of invalid and abnormal samples on the training of the value assessment model is reduced, and the reliability of the model's output value score is improved.
[0012] In some implementations, a value assessment model to be trained is trained based on N sample teaching trajectory information and the label value scores corresponding to each frame of sample teaching images in the N sample teaching trajectory information to obtain a trained value assessment model. This includes: inputting the N sample teaching trajectory information into the value assessment model to be trained to obtain the predicted value scores of each frame of sample teaching images in the N sample teaching trajectory information output by the value assessment model; and adjusting the model parameters of the value assessment model according to the difference between the predicted value scores of each frame of sample teaching images in the N sample teaching trajectory information and the corresponding label value scores to obtain a trained value assessment model.
[0013] The value assessment model is trained using supervised learning, with the label value score as the supervision signal. This enables the model to accurately learn the mapping relationship between image features and task value, ensuring that the value assessment results are stable and reproducible.
[0014] In some implementations, the teaching trajectory information and the value scores corresponding to each frame of the teaching image in the teaching trajectory information are used as training data to train the first agent to obtain the second agent. This includes: acquiring multiple predicted images output by the first agent during the execution of the first task, wherein the multiple predicted images indicate the execution process of the first task predicted by the first agent; constructing a loss function based on the degree of difference between the multiple predicted images and the corresponding teaching images of the multiple predicted images in the teaching trajectory information and the value scores corresponding to the multiple teaching images of the multiple predicted images in the teaching trajectory information; and adjusting the model parameters of the first agent based on the loss function to obtain the second agent.
[0015] Based on the difference between the predicted image and the teaching image, and combined with the value score to construct a loss function, weighted training is achieved. This allows the agent to prioritize learning high-value keyframe information, weaken the interference of low-value redundant information, accelerate the training convergence speed, and improve the stability of the policy and the accuracy of task execution.
[0016] In some implementations, a loss function is constructed based on the degree of difference between the multiple predicted images and the corresponding teaching images in the teaching trajectory information, and the value scores of the multiple teaching images corresponding to the multiple predicted images in the teaching trajectory information. This includes: using the value scores of the teaching images corresponding to the multiple predicted images in the teaching trajectory information as weight coefficients, and performing a weighted summation of the degree of difference between the multiple predicted images and their corresponding teaching images in the teaching trajectory information to obtain the loss function.
[0017] The loss function is constructed by weighting the image differences with value scores as weights. The weighting mechanism is simple, efficient and easy to implement. It can maximize the guiding role of high-value frames in model updates and further improve the training effect of the agent and its ability to adapt to complex tasks.
[0018] In some implementations, the first task includes folding the flexible object; the remaining task progress includes the remaining number of folds of the flexible object.
[0019] The solution is adapted to flexible object folding tasks, with the value score directly related to the remaining number of folds. This accurately matches the long-term, multi-stage characteristics of flexible object operations, effectively solving the problems of large differences in teaching data and difficulty in model learning when operating flexible items such as clothing and bedding, and improving the stability of robot operations.
[0020] Secondly, this application provides a training device for an intelligent agent, comprising: an acquisition module for acquiring teaching trajectory information of a first task, the teaching trajectory information being used to indicate the execution process of the first task; a value evaluation module for inputting the teaching trajectory information into a value evaluation model, and evaluating the value of each frame of teaching images in the teaching trajectory information through the value evaluation model to obtain the value score corresponding to each frame of teaching images in the teaching trajectory information; and a training module for using the teaching trajectory information and the value scores corresponding to each frame of teaching images in the teaching trajectory information as training data to train a first intelligent agent to obtain a second intelligent agent, wherein the second intelligent agent is used to execute the first task; wherein the value evaluation model is a model trained based on sample data; the sample data includes N sample teaching trajectory information of the first task, and the label value score corresponding to each frame of sample teaching images in the N sample teaching trajectory information; the label value score corresponding to the j-th frame of sample teaching images in the M-th frame of sample teaching images of the i-th sample teaching trajectory information is related to the remaining task progress corresponding to the j-th frame of sample teaching images, where N, M, i, and j are all positive integers, and i≤N, j≤M.
[0021] The beneficial effects achievable in the second aspect can be referred to in the beneficial effects of the method provided in any embodiment of the first aspect, and will not be repeated here.
[0022] Thirdly, this application provides an electronic device, comprising: a memory for storing instructions; and at least one processor for executing the instructions to cause the device to implement the methods provided in the first aspect and any possible implementation of the first aspect. The beneficial effects achievable in the third aspect can be referred to the beneficial effects of the methods provided in any embodiment of the first aspect, and will not be repeated here.
[0023] Fourthly, this application provides a computer-readable storage medium storing instructions that, when executed by a device, cause a computer to implement the methods provided in the first aspect and any possible implementation of the first aspect. The beneficial effects achievable in the fourth aspect can be found in the beneficial effects of the methods provided in any embodiment of the first aspect, and will not be repeated here.
[0024] Fifthly, this application provides a computer program product that stores instructions that, when executed on a device, cause the device to implement the methods provided in the first aspect and any possible implementation of the first aspect. The beneficial effects achievable in the fifth aspect can be found in the beneficial effects of the methods provided in any embodiment of the first aspect, and will not be repeated here. Attached Figure Description
[0025] Figure 1 According to some embodiments of this application, a flowchart of a method for training an intelligent agent is shown.
[0026] Figure 2 According to some embodiments of this application, a flowchart illustrating the training process of an intelligent agent is shown.
[0027] Figure 3 According to some embodiments of this application, a schematic diagram of the structure of a training device for an intelligent agent is shown.
[0028] Figure 4 According to some embodiments of this application, a schematic diagram of the structure of an electronic device is shown. Detailed Implementation
[0029] The illustrative embodiments of this application include, but are not limited to, methods, apparatuses, devices, media, and program products for training intelligent agents.
[0030] The electronic devices in this application embodiment can also be referred to as terminals, user equipment (UE), mobile stations (MS), mobile terminals (MT), etc. Electronic devices may include, but are not limited to, servers, mobile phones, smart TVs, wearable devices, tablets, computers with wireless transceiver capabilities, virtual reality (VR) terminal devices, augmented reality (AR) terminal devices, wireless terminals in industrial control, wireless terminals in transportation safety, wireless terminals in smart cities, and so on.
[0031] To address the technical problems in the background art, this application provides a training method for an intelligent agent. During the training of the agent using teaching trajectory information, a value evaluation model can be used to evaluate the value of each frame of the teaching trajectory information, obtaining a value score corresponding to each frame. Then, based on each frame of the teaching trajectory information and its corresponding value score, the agent to be trained is trained using training data. In this way, the value evaluation model can quantify the value of each frame of the teaching trajectory information frame by frame, accurately distinguishing the contribution of different frames to the agent's training. During training, reinforcement learning of key states and key steps and filtering of low-value redundant information are achieved based on the value scores. This effectively offsets the negative impacts of fluctuations in teaching trajectory duration, uneven rhythm, and inconsistent key frame positions caused by operator differences on the training process, significantly improving the training efficiency and convergence speed of the agent, and enhancing the stability of the agent's strategy, the execution accuracy of complex tasks, and its generalization ability.
[0032] The intelligent agent in this application embodiment can be software (such as intelligent voice assistant, game assistant), hardware (such as robot), or a system (such as autonomous driving system) with a neural network model deployed.
[0033] The training method for the intelligent agent provided in this application will be described in detail below with reference to specific embodiments.
[0034] Figure 1 According to some embodiments of this application, a flowchart of a method for training an intelligent agent is shown. The execution entities for each step of this process are all electronic devices, which will not be described in detail below.
[0035] like Figure 1 As shown, the process includes:
[0036] S101: Obtain the teaching trajectory information of the first task, wherein the teaching trajectory information includes multiple teaching images, which are used to indicate the execution process of the first task.
[0037] The first task includes, but is not limited to, tasks such as folding flexible objects, folding flexible objects (such as clothing, bedding (such as sheets, duvet covers, pillowcases, etc.), flexible leather, etc.), autonomous driving tasks, and game decision-making tasks. For ease of understanding, the following explanation will use the clothing folding task as an example.
[0038] The teaching trajectory information can be the teaching video of the first task, or a sequence of teaching image frames containing multiple teaching image frames.
[0039] S102: Input the teaching trajectory information into the value assessment model, and use the value assessment model to assess the value of each frame of teaching image in the teaching trajectory information to obtain the value score corresponding to each frame of teaching image in the teaching trajectory information.
[0040] After the electronic device acquires the teaching trajectory information of the first task, it can input the teaching trajectory information into the value assessment model. The value assessment model then evaluates the value of each frame of the teaching image in the teaching trajectory information to obtain the value score corresponding to each frame of the teaching image in the teaching trajectory information.
[0041] As can be understood, the value assessment model is a pre-trained model used to assess the value of each frame of the teaching trajectory information and output the corresponding value score for each frame. For the sake of coherence, the specific training process of the value assessment model will be described below.
[0042] It can be understood that the remaining task progress corresponding to each teaching image is the semantic remaining progress of completing the first task from that teaching image.
[0043] In some embodiments, the remaining task progress corresponding to each teaching image frame may be the number of remaining actions (or remaining steps) to complete the first task from the beginning of the teaching image, or it may be the ratio between the number of remaining actions to complete the first task from the beginning of the teaching image and the total number of actions (or total number of steps) of the first task.
[0044] For example, when the first task is a clothing folding task, the remaining task progress corresponding to each frame of the teaching image can be the ratio between the remaining number of folds corresponding to each frame of the image and the total number of folds for the clothing folding task.
[0045] Taking the folding action of clothing folding task as an example, which includes 6 actions, namely grasping, unfolding, flattening, aligning, folding, and tidying, for the teaching image where the clothing is in the unfolded state, since the remaining four actions of flattening, aligning, folding, and tidying are to be completed from this teaching image, the remaining task progress corresponding to this teaching image can be 4 / 6=2 / 3.
[0046] In some embodiments, the value score corresponding to each teaching image frame is negatively correlated with the remaining task progress corresponding to each teaching image frame. That is, the more remaining task progress a teaching image has, the lower its value score; conversely, the less remaining task progress a teaching image has, the higher its value score.
[0047] In some embodiments, the value score and the remaining task progress can satisfy the following formula (1).
[0048] (1)
[0049] Where W is the value score, R is the remaining task progress, and Q is the total task progress.
[0050] Taking the first task as clothing folding as an example, the execution process of the clothing folding task is as follows: the clothing is outside the operation area (initial state, denoted as state s0) → grab (denoted as action a1) → the clothing is within the operation area (denoted as state s1) → unfold (denoted as action a2) → the clothing is in the unfolded state (denoted as state s2) → flatten (denoted as action a3) → the clothing is in the flattened state (denoted as state s3) → align (denoted as action a4) → the clothing is in the aligned state (denoted as state s4) → fold (denoted as action a5) → the clothing is in the folded state (denoted as state s5) → arrange (denoted as action a6) → the clothing is in the folded state (denoted as state s6).
[0051] For example, the remaining task progress of the teaching image corresponding to state s0 is 1, and the value score of the teaching image can be 0. The remaining task progress of the teaching image corresponding to state s1 is 5 / 6, and the value score of the teaching image can be 1 / 6. The remaining task progress of the teaching image corresponding to state s2 is 2 / 3, and the value score of the teaching image can be 1 / 3. The remaining task progress of the teaching image corresponding to state s3 is 1 / 2, and the value score of the teaching image can be 1 / 2. The remaining task progress of the teaching image corresponding to state s4 is 1 / 3, and the value score of the teaching image can be 2 / 3. The remaining task progress of the teaching image corresponding to state s5 is 1 / 6, and the value score of the teaching image can be 5 / 6. The remaining task progress of the teaching image corresponding to state s6 is 0, and the value score of the teaching image can be 1.
[0052] It is understood that the remaining task progress in the embodiments of this application is the semantic actual remaining task progress, such as the ratio of the actual number of remaining actions to the actual total number of actions, rather than the remaining time progress of teaching trajectory information (such as teaching video) or the remaining number of frames of teaching trajectory information (such as teaching image frame sequence).
[0053] S103: Using the teaching trajectory information and the value scores corresponding to each frame of the teaching image in the teaching trajectory information as training data, train the first intelligent agent to be trained to obtain the second intelligent agent, wherein the second intelligent agent is used to perform the first task.
[0054] In some embodiments, the electronic device can acquire multiple predicted images output by the first agent during the execution of a first task, wherein the multiple predicted images indicate the execution process of the first task predicted by the first agent. Then, the electronic device can construct a loss function based on the degree of difference between the multiple predicted images and their corresponding teaching images in the teaching trajectory information, and the value scores corresponding to the multiple teaching images in the teaching trajectory information. Based on this loss function, the model parameters of the first agent are adjusted to obtain a second agent.
[0055] In some embodiments, the electronic device may use the value scores corresponding to the teaching images corresponding to the plurality of predicted images in the teaching trajectory information as weighting coefficients, and perform a weighted summation of the differences between the plurality of predicted images and their respective corresponding teaching images in the teaching trajectory information to obtain a loss function.
[0056] In some embodiments, the degree of difference between the predicted image and the corresponding teaching image can be the difference between the remaining task progress corresponding to the predicted image and the remaining task progress corresponding to the teaching image.
[0057] For example, the electronic device can calculate the above loss function using the following formula (2).
[0058] (2)
[0059] Where L represents the loss function, T represents the total number of frames in the teaching trajectory information, and W t I represents the value score corresponding to the teaching image in frame t. t Y represents the remaining task progress corresponding to the t-th frame of the teaching trajectory information. t This represents the remaining task progress corresponding to the predicted image of the t-th frame output by the first agent.
[0060] In other embodiments, the degree of difference between the predicted image and the corresponding teaching image can also be the image similarity between the predicted image and the teaching image.
[0061] In this embodiment, the teaching trajectory information of the first task is first obtained, and then the value evaluation model is used to evaluate the value of each frame of the teaching image in the teaching trajectory information frame by frame to obtain the value score corresponding to each frame. Finally, the teaching trajectory information and the value scores of each frame are used together as training data to complete the training of the agent to be trained.
[0062] Compared with existing technologies, this embodiment can perform frame-level value quantification of teaching data, accurately distinguish the contribution of different image frames to the learning of the agent, automatically strengthen the learning of key states and key steps during training, weaken the interference of low-value redundant information, effectively offset the training instability caused by operator differences, trajectory duration fluctuations, and uneven rhythm, significantly improve the training efficiency and convergence speed of the agent, and enable the trained agent to have higher policy stability, execution accuracy and generalization ability in long-term, multi-stage complex operation tasks.
[0063] The training process of the agent is described below.
[0064] Figure 2 According to some embodiments of this application, a flowchart illustrating the training process of an intelligent agent is shown. The execution entities for each step of this process are all electronic devices, which will not be described in detail below.
[0065] like Figure 2 As shown, the process includes:
[0066] S201: Obtain the N sample teaching trajectory information of the first task and the label value score corresponding to each frame of sample teaching image in the N sample teaching trajectory information. The label value score corresponding to the j-th frame of sample teaching image in the M frames of sample teaching image of the i-th sample teaching trajectory information is related to the remaining task progress corresponding to the j-th frame of sample teaching image. N, M, i, and j are all positive integers, and i≤N, j≤M.
[0067] In some embodiments, the N sample teaching trajectory information satisfies at least one of the following screening criteria:
[0068] (1) The image similarity between the last frame of the N sample teaching trajectory information and the preset reference image is greater than the image similarity threshold, wherein the reference image is the image corresponding to the environmental state when the first task is completed.
[0069] (2) The number of frames for teaching trajectory information of N samples is within the preset frame number range.
[0070] The preset frame rate range can be a pre-defined interval based on the standard execution flow of the first task, the data acquisition frame rate, and the statistical distribution of a large number of effective teaching trajectories. This range is used to exclude excessively short trajectories caused by acquisition interruptions, incomplete operations, or severe missing actions, as well as excessively long trajectories caused by redundant actions, invalid pauses, or repetitive operations. For example, in a clothing folding task, the preset frame rate range can be set to 100 to 1000 frames.
[0071] (3) The duration of the N sample teaching trajectory information is within the preset duration range.
[0072] The preset duration range can be a pre-defined time interval based on the standard execution time of the first task, the average effective teaching time, and the trajectory duration distribution statistics. This is used to ensure that the execution rhythm of the sample teaching trajectory conforms to the normal task flow and to eliminate invalid samples with abnormal durations. For example, in the clothing folding task, the preset duration range can be set to 10 seconds to 120 seconds.
[0073] In this embodiment of the application, the electronic device can use the above-mentioned screening conditions to select high-value sample teaching trajectory information that meets the conditions from N sample teaching trajectory information, and train the value assessment model.
[0074] Specifically, after the electronic device acquires N sample teaching trajectory information for the first task, the electronic device can calculate the image similarity between the last frame of the sample teaching image of each sample teaching trajectory information and the preset reference image corresponding to the environmental state when the first task is completed, and compare the image similarity with the preset image similarity threshold. If the image similarity is greater than the image similarity threshold, the sample teaching trajectory information is determined to be a valid sample, retained and used for training the subsequent value assessment model. If the image similarity is less than or equal to the image similarity threshold, the sample teaching trajectory information is determined to be an invalid sample, discarded and not used for training the value assessment model.
[0075] In addition, the electronic device can also count the number of frames and the duration of N sample teaching trajectory information respectively, compare the number of trajectory frames with a preset frame number range, and compare the trajectory duration with a preset duration range; if the number of trajectory frames is within the preset frame number range and the trajectory duration is within the preset duration range, then the sample teaching trajectory information is retained; if the number of trajectory frames or the trajectory duration exceeds the corresponding preset range, then the sample teaching trajectory information is determined to be an abnormal sample and is removed.
[0076] It is understandable that by applying at least one of the following constraints to the N sample teaching trajectory information used to train the value assessment model: final state image similarity (the similarity between the last frame of the N sample teaching trajectory information and the image corresponding to the environmental state when the first task is completed), trajectory frame number, and trajectory duration, the standardization, validity, and consistency of the sample set can be guaranteed from the source, providing a high-quality, low-noise training foundation for the value assessment model.
[0077] It is understandable that by verifying the similarity between the last frame of the sample teaching trajectory and the reference image of the task completion state, invalid samples that are not completed, have unqualified final states, or have failed to execute can be directly eliminated, thus avoiding erroneous final states from interfering with the learning direction of the value assessment model.
[0078] It is understandable that by limiting the number of trajectory frames and duration within a preset reasonable range, abnormally short trajectories caused by data acquisition interruption, premature termination, or missing operations, as well as abnormally long trajectories caused by redundant actions or invalid pauses, can be effectively eliminated, ensuring that the sample trajectories are complete, standardized, and effective teaching in accordance with the normal task execution process.
[0079] The sample set obtained through the above multiple screening conditions can significantly reduce the negative impact of abnormal and noisy data on the training of the value assessment model, enabling the model to learn the mapping relationship between state features and task progress more stably and accurately, improving the reliability, consistency and generalization ability of the value assessment model's output score, and thus providing more accurate value weights for the subsequent training of the agent, further improving the agent's training efficiency and task execution effect.
[0080] S202: Based on the teaching trajectory information of N samples and the label value scores corresponding to each frame of the teaching image in the teaching trajectory information of N samples, train the value evaluation model to be trained to obtain the trained value evaluation model.
[0081] In some embodiments, the valuation model may adopt one of the following model structures:
[0082] (1) Single-frame visual coding model based on convolutional neural network.
[0083] (2) Image semantic coding model based on visual Transformer.
[0084] (3) Multi-frame sequence models based on recurrent neural networks, temporal convolutional networks or Transformer temporal encoders.
[0085] (4) A multimodal model that integrates image observation, motion information, and robot state information.
[0086] (5) Regression model that outputs scalar value scores, ranking model that outputs progress level, or classification model.
[0087] In some embodiments, the electronic device can input N sample teaching trajectory information into the value assessment model to be trained, obtain the predicted value score of each frame of sample teaching image in the N sample teaching trajectory information output by the value assessment model, and then adjust the model parameters of the value assessment model according to the difference between the predicted value score of each frame of sample teaching image in the N sample teaching trajectory information and the corresponding label value score. In this way, the electronic device can perform multiple rounds of training on the value assessment model until the training conditions are met and the training ends, thus obtaining the trained value assessment model.
[0088] In some embodiments, when an electronic device trains a value assessment model, training may terminate when at least one of the following training conditions is met:
[0089] (1) Regression training conditions
[0090] Using the label value score corresponding to the sample teaching image as the supervision signal, the error between the predicted value score output by the value assessment model and the label value score is minimized, thereby achieving accurate regression of the value score.
[0091] For example, in the clothing folding task, the label value score corresponding to clothing in a flat state is 0.50, and the label value score corresponding to clothing in an aligned state is 0.67. When a teaching image of clothing in a flat state is input into the value assessment model, the value assessment model outputs a predicted value score; when a teaching image of clothing in an aligned state is input into the value assessment model, the value assessment model also outputs a predicted value score. During training, the electronic device calculates the difference between the predicted value score output by the value assessment model and the label value score, and continuously adjusts the parameters of the value assessment model to continuously reduce the above difference, ultimately minimizing the error between the predicted value score and the label value score, allowing the value assessment model to output accurate and stable value scores.
[0092] (2) Sorting training conditions
[0093] In the same teaching trajectory, the predicted value score corresponding to the state in the later stage of task execution is higher than the predicted value score corresponding to the state in the early stage of task execution, so as to ensure that the value scores maintain a reasonable order as the task progresses.
[0094] In the same clothing folding teaching trajectory information, the order of appearance of the states is as follows: clothing is in the operation area, clothing is laid flat, and clothing is folded. The predicted value score output by the value assessment model should satisfy the following condition: the score for clothing being in the operation area is less than the score for clothing being laid flat, and the score for clothing being laid flat is less than the score for clothing being folded, for example, 0.17, 0.50, and 0.83 respectively. This ensures that the closer the task is to completion, the higher the value score output by the value assessment model.
[0095] (3) Comparison of learning and training conditions
[0096] Teaching images with similar semantic progress states are placed closer together in the model's feature space; teaching images with significantly different semantic progress states are placed further apart in the feature space, thereby improving the model's ability to distinguish state progress.
[0097] For example, among multiple teaching trajectory information collected by different operators, images of clothing in an aligned state differ in shooting angle, lighting conditions, and clothing placement, but their semantic task progress is consistent. The value assessment model should make these images close in distance in the feature space. The initial state of clothing outside the operation area and the state of clothing in a folded state differ greatly in task progress. The value assessment model should keep these two types of images far apart in the feature space to accurately distinguish different task progress levels.
[0098] (4) Multi-task joint training conditions
[0099] Simultaneously, two sub-tasks are learned: first, to determine whether the final state of the teaching trajectory is a qualified state for task completion; second, to score the progress of each state during task execution, thereby improving the accuracy and robustness of value assessment through multi-task joint constraints.
[0100] For example, in the clothing folding task, the value assessment model performs two tasks simultaneously during training. The first task is to determine whether the clothing in the last frame of the teaching trajectory information is in a completed folding state and is considered acceptable. The second task is to output corresponding value scores for each intermediate state: clothing outside the operation area, clothing within the operation area, clothing unfolded, clothing flattened, clothing aligned, and clothing folded. The final state acceptance judgment and the process progress score complement and constrain each other, making the value scores output by the value assessment model more closely reflect the actual progress of the task.
[0101] A satisfactory state refers to the final state of the target object and environment after the first task has been completed completely and correctly according to preset standards, meeting the expectations of the task. For example, in the task of folding clothes, a satisfactory state means that the clothes are folded neatly, in a regular shape, without obvious misalignment or loose unfolding, and the overall folding effect meets the preset folding completion standards.
[0102] (5) Temporal Consistency Training Conditions
[0103] Temporal constraints are imposed on high-quality complete teaching trajectories so that the value scores within the same trajectory show a monotonically increasing trend or a phased increasing trend over time, ensuring that the value assessment results conform to the continuous evolution logic of task execution.
[0104] In a complete clothing folding demonstration trajectory, the states, in the execution order, are: clothing outside the operation area, clothing within the operation area, clothing unfolded, clothing laid flat, clothing aligned, clothing folded, and clothing folded completely. The value score output by the value assessment model should start from 0.00 and steadily increase, successively to 0.00, 0.17, 0.33, 0.50, 0.67, 0.83, and 1.00, maintaining a monotonous increase throughout without any score drop, oscillation, or error, consistent with the actual evolution logic of the task execution.
[0105] In this embodiment, a dedicated value assessment model is trained using label value scores that correlate sample teaching trajectory information with the remaining task progress. This model provides stable, objective, and highly semantically consistent frame-level value scores for subsequent agent training. By verifying the completion status of the sample teaching trajectories and standardizing their duration and frame count, abnormal, invalid, or incomplete training samples can be effectively removed, improving the learning reliability and output stability of the value assessment model. Employing multi-dimensional constraints such as regression, ranking, contrastive learning, multi-task collaboration, and temporal consistency training objectives allows the value assessment model to accurately learn the mapping relationship between state features and task progress. This ensures that the output value score exhibits a reasonable increasing trend as the task progresses and maintains a strict negative correlation with the remaining task progress, preventing value score fluctuations or errors. The resulting value assessment model is highly versatile and accurate, providing reliable weighting criteria for subsequent weighted imitation learning by the agent, further improving the convergence speed, policy stability, and performance on complex tasks.
[0106] Figure 3 According to some embodiments of this application, a schematic diagram of the structure of a training device for an intelligent agent is shown.
[0107] like Figure 3 As shown, the training device 30 for the intelligent agent includes an acquisition module 31, a value evaluation module 32, and a training module 33.
[0108] The acquisition module 31 is used to acquire the teaching trajectory information of the first task, which is used to indicate the execution process of the first task.
[0109] The value assessment module 32 is used to input the teaching trajectory information into the value assessment model, and to assess the value of each frame of teaching image in the teaching trajectory information through the value assessment model to obtain the value score corresponding to each frame of teaching image in the teaching trajectory information.
[0110] The training module 33 is used to train the first intelligent agent to obtain the second intelligent agent by using the teaching trajectory information and the value scores corresponding to each frame of the teaching trajectory information as training data. The second intelligent agent is used to perform the first task.
[0111] In some embodiments, the value assessment model is a model trained based on sample data; the sample data includes N sample teaching trajectory information for the first task, and the label value score corresponding to each frame of sample teaching image in the N sample teaching trajectory information; the label value score corresponding to the j-th frame of sample teaching image in the M frames of sample teaching image of the i-th sample teaching trajectory information is related to the remaining task progress corresponding to the j-th frame of sample teaching image, where N, M, i, and j are all positive integers, and i≤N, j≤M. The specific training process of the value assessment model can be found in [reference needed]. Figure 2 The relevant descriptions of the embodiments shown will not be repeated here.
[0112] It should be noted that the information interaction and execution process between the modules of the training device 30 for the aforementioned intelligent agent are based on the same concept as those in the method embodiments of this application, and therefore the resulting technical effects are the same. For details, please refer to the descriptions in the foregoing method embodiments of this application (e.g., Figure 1 The descriptions of the corresponding embodiments are omitted here.
[0113] The electronic devices provided in the above embodiments are described below.
[0114] For example, Figure 4 According to some embodiments of this application, a schematic diagram of the structure of an electronic device is shown.
[0115] The electronic device 100 may be the electronic device in the foregoing embodiments, and the electronic device 100 is used to implement the training method of the intelligent agent provided in the foregoing embodiments.
[0116] like Figure 4 As shown, the electronic device 100 includes one or more processors 101, system memory 102, non-volatile memory (NVM) 103, communication interface 104, input / output device 105, and system control logic unit 106 for coupling the processor 101, system memory 102, NVM 103, communication interface 104, and input / output device 105. Wherein:
[0117] Processor 101 may include one or more processing units, such as processing modules or processing circuits that may include a central processing unit (CPU), graphics processing unit (GPU), digital signal processor (DSP), micro-programmed control unit (MCU), artificial intelligence (AI) processor or field programmable gate array (FPGA), neural network processing unit (NPU), etc., and may include one or more single-core or multi-core processors.
[0118] System memory 102 is volatile memory, such as random-access memory (RAM), double data rate synchronous dynamic random access memory (DDR SDRAM), etc.
[0119] The non-volatile memory 103 may include one or more tangible, non-transitory computer-readable media for storing data and / or instructions. In some embodiments, the non-volatile memory 103 may include any suitable non-volatile memory such as flash memory and / or any suitable non-volatile storage device, such as a hard disk drive (HDD), compact disc (CD), digital versatile disc (DVD), solid-state drive (SSD), etc. In some embodiments, the non-volatile memory 103 may also be a removable storage medium, such as secure digital (SD) storage. In other embodiments, the non-volatile memory 103 may be used to store instructions for training methods of the intelligent agent provided in the foregoing embodiments.
[0120] Specifically, system memory 102 and non-volatile memory 103 may each include a temporary copy and a permanent copy of instruction 107. Instruction 107 may include, when executed by at least one of processors 101, causing electronic device 100 to implement the training method for the intelligent agent provided in the embodiments of this application.
[0121] The communication interface 104 may include a transceiver for providing a wired or wireless communication interface for the electronic device 100, thereby enabling communication with any other suitable device via one or more networks. In some embodiments, the communication interface 104 may be integrated into other components of the electronic device 100, for example, the communication interface 104 may be integrated into the processor 101. In some embodiments, the electronic device 100 may communicate with other devices through the communication interface 104, for example, the electronic device 100 may obtain relevant data from other devices through the communication interface 104.
[0122] Input / output device 105 can be an input device such as a keyboard or mouse, or an output device such as a monitor. Users can interact with electronic device 100 through input / output device 105.
[0123] The system control logic unit 106 may include any suitable interface controller to provide any suitable interface to other modules of the electronic device 100. For example, in some embodiments, the system control logic unit 106 may include one or more memory controllers to provide an interface to the system memory 102 and the non-volatile memory 103.
[0124] In some embodiments, at least one of the processors 101 may be packaged together with the logic of one or more controllers for the system control logic unit 106 to form a system in package (SiP). In other embodiments, at least one of the processors 101 may also be integrated on the same chip with the logic of one or more controllers for the system control logic unit 106 to form a system-on-chip (SoC).
[0125] Understandable. Figure 4 The structure of the electronic device 100 shown is only one example; in other embodiments, the electronic device 100 may include more... Figure 4 It can have more or fewer components, or combine some components, or split some components, or arrange the components differently. Figure 4 The components can be implemented in hardware, software, or a combination of both.
[0126] It is understood that electronic device 100 can be any device configured on the vehicle, including but not limited to mobile phones, tablets, laptops, super mobile personal computers, netbooks, personal digital assistants, or servers, etc.
[0127] This application also provides a computer-readable storage medium storing one or more programs, which, when executed by an electronic device, enable the electronic device to implement the training method for the intelligent agent provided in the foregoing embodiments.
[0128] This application also provides a program product that stores instructions. When these instructions are executed on an electronic device, the electronic device can implement the training methods for intelligent agents provided in the foregoing embodiments.
[0129] Various embodiments of the mechanisms disclosed in this application can be implemented in hardware, software, firmware, or combinations of these implementation methods. Embodiments of this application can be implemented as computer programs or program code executable on a programmable system, the programmable system including at least one processor, a storage system (including volatile and non-volatile memory and / or storage elements), at least one input device, and at least one output device.
[0130] Program code can be applied to input instructions to execute the functions described in this application and generate output information. The output information can be applied to one or more output devices in a known manner. For the purposes of this application, the processing system includes any system having a processor such as, for example, a digital signal processor, a microcontroller, an application-specific integrated circuit, or a microprocessor.
[0131] The program code can be implemented using a high-level procedural language or an object-oriented programming language to communicate with the processing system. Assembly language or machine language can also be used when needed. In fact, the mechanisms described in this application are not limited to any particular programming language. In either case, the language can be a compiled language or an interpreted language.
[0132] In some cases, the disclosed embodiments may be implemented in hardware, firmware, software, or any combination thereof. The disclosed embodiments may also be implemented as instructions carried on or stored thereon by one or more transient or non-transitory machine-readable (e.g., computer-readable) storage media, which may be read and executed by one or more processors. For example, the instructions may be distributed via a network or through other computer-readable media. Therefore, machine-readable media can include any mechanism for storing or transmitting information in a machine-readable (e.g., computer-readable) form, including but not limited to floppy disks, optical disks, CD-ROMs, compact disc-read-only memory (CD-ROMs), magneto-optical disks, read-only memory (ROM), random-access memory (RAM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), magnetic cards or optical cards, flash memory, or tangible machine-readable storage for transmitting information (e.g., carrier waves, infrared signals, digital signals, etc.) using the Internet in the form of electrical, optical, acoustic, or other forms of propagation signals. Therefore, machine-readable media includes any type of machine-readable medium suitable for storing or transmitting electronic instructions or information in a machine-readable (e.g., computer-readable) form.
[0133] In the accompanying drawings, some structural or methodological features may be shown in a specific arrangement and / or order. However, it should be understood that such a specific arrangement and / or order may not be necessary. Rather, in some embodiments, these features may be arranged in a manner and / or order different from that shown in the illustrative drawings. Furthermore, the inclusion of structural or methodological features in a particular figure does not imply that such features are required in all embodiments, and in some embodiments, these features may be omitted or may be combined with other features.
[0134] It should be noted that all units / modules mentioned in the device embodiments of this application are logical units / modules. Physically, a logical unit / module can be a physical unit / module, a part of a physical unit / module, or a combination of multiple physical units / modules. The physical implementation of these logical units / modules themselves is not the most important factor; the combination of functions implemented by these logical units / modules is the key to solving the technical problems proposed in this application. Furthermore, to highlight the innovative aspects of this application, the above-described device embodiments of this application have not introduced units / modules that are not closely related to solving the technical problems proposed in this application. This does not mean that the above-described device embodiments do not contain other units / modules.
[0135] It should be noted that in the examples and description of this patent, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one" does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.
[0136] Although this application has been illustrated and described with reference to certain preferred embodiments thereof, those skilled in the art will understand that various changes in form and detail may be made thereto without departing from the scope of this application.
Claims
1. A method for training an intelligent agent, characterized in that, Applied to electronic devices, the method includes: Obtain teaching trajectory information for the first task, wherein the teaching trajectory information includes multiple teaching images, which are used to indicate the execution process of the first task; The teaching trajectory information is input into the value assessment model, and the value assessment model is used to evaluate the value of each frame of teaching image in the teaching trajectory information to obtain the value score corresponding to each frame of teaching image in the teaching trajectory information. During the process of the first intelligent agent executing the first task, multiple predicted images output by the first intelligent agent are obtained, and the multiple predicted images indicate the execution process of the first task predicted by the first intelligent agent. A loss function is constructed based on the degree of difference between the multiple predicted images and the corresponding teaching images of the multiple predicted images in the teaching trajectory information, and the value scores corresponding to the multiple teaching images of the multiple predicted images in the teaching trajectory information. Based on the loss function, the model parameters of the first agent are adjusted to obtain a second agent, wherein the second agent is used to perform the first task; The training process of the value assessment model includes: Obtain N sample teaching trajectory information of the first task and the label value score corresponding to each frame of sample teaching image in the N sample teaching trajectory information. The label value score corresponding to the j-th frame of sample teaching image in the M frames of sample teaching image of the i-th sample teaching trajectory information is related to the remaining task progress corresponding to the j-th frame of sample teaching image. N, M, i, and j are all positive integers, and i≤N, j≤M. Based on the N sample teaching trajectory information and the label value scores corresponding to each frame of sample teaching images in the N sample teaching trajectory information, the value evaluation model to be trained is trained to obtain the trained value evaluation model.
2. The method for training an agent according to claim 1, characterized in that, The label value score corresponding to the j-th frame sample teaching image is negatively correlated with the remaining task progress corresponding to the j-th frame sample teaching image.
3. The method for training an intelligent agent according to claim 1, characterized in that, The N sample teaching trajectory information satisfies the following condition: The image similarity between the last frame of the N sample teaching trajectory information and a preset reference image is greater than an image similarity threshold, wherein the reference image is the image corresponding to the environmental state when the first task is completed; and / or, The number of frames for the N sample teaching trajectory information is within a preset frame range; and / or, The duration of the N sample teaching trajectory information is within a preset duration range.
4. The method for training an agent according to claim 1, characterized in that, The step involves training the value evaluation model to be trained based on the N sample teaching trajectory information and the label value scores corresponding to each frame of the sample teaching image in the N sample teaching trajectory information, to obtain the trained value evaluation model, including: The N sample teaching trajectory information is input into the value evaluation model to be trained, and the predicted value score of each frame of sample teaching image in the N sample teaching trajectory information output by the value evaluation model is obtained. Based on the difference between the predicted value score and the corresponding label value score of each frame of the teaching image in the N sample teaching trajectory information, the model parameters of the value assessment model are adjusted to obtain the trained value assessment model.
5. The method for training an agent according to claim 1, characterized in that, The loss function is constructed based on the degree of difference between the multiple predicted images and the corresponding teaching images of each of the multiple predicted images in the teaching trajectory information, and the value scores corresponding to the multiple teaching images of the multiple predicted images in the teaching trajectory information, including: The loss function is obtained by using the value scores corresponding to the teaching images corresponding to the multiple predicted images in the teaching trajectory information as weight coefficients, and by weighting and summing the differences between the multiple predicted images and their respective corresponding teaching images in the teaching trajectory information.
6. The method for training an agent according to any one of claims 1 to 5, characterized in that, The first task includes the folding of flexible objects; The remaining task progress includes the remaining number of folds of the flexible object.
7. A training device for an intelligent agent, characterized in that, include: An acquisition module is used to acquire teaching trajectory information of the first task, wherein the teaching trajectory information is used to indicate the execution process of the first task; The value assessment module is used to input the teaching trajectory information into the value assessment model, and to perform value assessment on each frame of teaching image in the teaching trajectory information through the value assessment model to obtain the value score corresponding to each frame of teaching image in the teaching trajectory information. A training module is used to acquire multiple predicted images output by the first agent during the execution of the first task, wherein the multiple predicted images indicate the execution process of the first task predicted by the first agent. Based on the degree of difference between the multiple predicted images and the corresponding teaching images of the multiple predicted images in the teaching trajectory information and the value scores corresponding to the multiple teaching images of the multiple predicted images in the teaching trajectory information, a loss function is constructed. Based on the loss function, the model parameters of the first agent are adjusted to obtain a second agent, wherein the second agent is used to execute the first task. The value assessment model is a model trained based on sample data; The sample data includes N sample teaching trajectory information of the first task, and the label value score corresponding to each frame of sample teaching image in the N sample teaching trajectory information; The label value score corresponding to the j-th sample teaching image in the M-frame sample teaching images of the i-th sample teaching trajectory information is related to the remaining task progress corresponding to the j-th sample teaching image. N, M, i, and j are all positive integers, and i≤N, j≤M.
8. An electronic device, characterized in that, Includes memory for storing instructions; At least one processor is configured to execute the instructions to cause the electronic device to implement the training method for the intelligent agent as described in any one of claims 1 to 6.
9. A computer-readable storage medium, characterized in that, The readable storage medium stores instructions that, when executed on a computer, cause the computer to perform the training method for the intelligent agent according to any one of claims 1 to 6.
10. A computer program product, characterized in that, The computer program product stores instructions, which, when executed on the device, cause the device to perform the training method for the intelligent agent as described in any one of claims 1 to 6.