Robotic learning method and system based on agile iteration aggregation

Abstract

The application belongs to the technical field of robots, and particularly relates to a robot learning method based on agile iteration aggregation, which comprises the following steps: according to an initial expert demonstration data set, environment state features and corresponding action labels are extracted, a robot action strategy network is pre-trained using a behavior cloning algorithm, and an initial action strategy network is obtained; the robot autonomously executes a current action and detects the current action; when the action is inconsistent, an interactive device is intervened and control right is taken over, the current action of the robot is corrected, and corrected trajectory data is stored; the corrected trajectory data is processed through a delay compensation truncation mechanism and a trajectory standardization resampling algorithm, the processed corrected trajectory data is dynamically weighted and aggregated, and an aggregated data set is obtained; the data is input into the initial action strategy network for training, and through iterative training, a final action strategy network is obtained; and the application completes robust training of complex operation skills of the robot with extremely low labor cost and high efficiency.

Description

Technical Field

[0001] This invention belongs to the field of robot intelligent control and machine learning technology, specifically relating to a robot learning method, system, device and medium based on agile iterative aggregation. Background Technology

[0002] Imitation learning (IL) is a mainstream paradigm for robots to acquire operational skills by observing expert demonstration data. It enables robots to quickly master complex operational tasks without complex programming and extensive trial and error. Among these methods, behavioral cloning (BC) is the most direct implementation, transforming policy learning into a supervised regression problem from state to action. By fitting expert state-action samples, it achieves end-to-end policy modeling, offering advantages such as simplicity and high training efficiency. It is widely used in robot grasping, assembly, and mobile operations.

[0003] However, traditional behavioral cloning has an inherent distribution shift defect in actual deployment: small errors generated by policy execution will accumulate over time, causing the robot to enter a state space not covered by the expert demonstration, which in turn leads to error amplification and task execution failure, making it difficult to guarantee the stability and reliability of the operation.

[0004] To mitigate the distribution shift problem, existing technologies have introduced interactive imitation learning (such as DAgger and CEIL), which incorporates biased states and expert corrections during policy execution into the training set through online human intervention and correction feedback. This iterative aggregation of data reduces the difference in state distribution between training and deployment. However, in highly practical scenarios with extremely small samples (initially only 10–20 data points), existing interactive methods still face several unresolved technical bottlenecks:

[0005] First, the data misbinding caused by human reaction delays is not addressed. Humans have inherent cognitive and physical delays in the process of observing robot errors, making decisions, and implementing takeover interventions. Existing methods directly use correction data containing delays for training, which leads to the incorrect association between "error states that cause failure" and "human recovery actions", seriously contaminating the few-sample training data and causing policy learning bias.

[0006] Second, the varying trajectory lengths lead to optimization instability. The timing and duration of human intervention for different errors vary significantly, resulting in highly variable lengths of the collected correction trajectory sequences. In the process of multiple rounds of data aggregation with extremely small samples, the input of variable-length sequences causes the gradient updates of temporal strategy networks such as RNN and LSTM to oscillate violently, significantly reducing the model's convergence and generalization ability.

[0007] Third, the rigidity of feedback weights leads to learning imbalance. Existing methods equally weight all human correction data without distinguishing the value of intervention information: early interventions often target fundamental policy defects, resulting in high effective information density; later interventions are mostly fine-tuning of actions, which easily introduces noise. The lack of a dynamic weight adjustment mechanism makes it difficult for the model to reasonably balance expert guidance and autonomous exploration, limiting the learning efficiency and final performance of the strategy under conditions of few samples.

[0008] In summary, existing imitation learning techniques struggle to overcome problems such as distribution shift, delay interference, sequence instability, and feedback weight mismatch under extremely small sample constraints, thus failing to meet the skill learning and deployment needs of robots in highly reliable, low-sample, and highly interactive scenarios. Summary of the Invention

[0009] The present invention aims to solve at least one of the technical problems existing in the prior art.

[0010] To this end, this invention proposes a robot learning method, system, device and medium based on agile iterative aggregation, which can complete robust training of robot complex operation skills with extremely low human cost and high efficiency.

[0011] According to a robot learning method based on agile iterative aggregation according to the present invention, the method includes the following steps:

[0012] S1. Based on multiple initial expert demonstration datasets, extract environmental state features and corresponding action labels, and use the behavior cloning algorithm to pre-train the robot action policy network to obtain the initial action policy network.

[0013] S2, the robot autonomously performs the current action in the environment and detects the current action;

[0014] When the robot's actions are inconsistent with the preset actions, the interactive device will intervene and take over control to correct the robot's current actions and store the corrected trajectory data.

[0015] S3, the corrected trajectory data is processed by a delay compensation truncation mechanism and a trajectory standardization resampling algorithm to obtain standardized corrected trajectory data;

[0016] S4. Dynamically weighted aggregate the standardized correction trajectory data to obtain the aggregated dataset.

[0017] S5. Input the aggregated dataset into the initial action policy network for training, and then return to step S2 for iterative training until the robot's actions are consistent with the preset actions, thus obtaining the final action policy network.

[0018] Furthermore, specifically, step S3 includes the following steps:

[0019] S31, acquire the correction trajectory data, and truncate the correction trajectory data for the initial preset number of steps using a time window processor;

[0020] S32, the length of the truncated corrected trajectory data is resampled and aligned to the standard sequence length using an interpolation algorithm, and a hard retention strategy for the critical end state is executed to obtain the standardized corrected trajectory data.

[0021] Furthermore, specifically, step S4 includes the following steps:

[0022] S41, Calculate the correction frequency factor F based on the number of corrections and the total number of autonomous executions by the robot. cor ;

[0023] S42, according to the correction frequency factor F cor The feedback weight α of the standardized correction trajectory data is dynamically acquired;

[0024] S43, the standardized corrected trajectory and the corresponding weight α are input into the replay buffer for weighted aggregation to form the aggregated dataset.

[0025] Furthermore, specifically, step S42 includes:

[0026] .

[0027] Furthermore, specifically, in step S5, a loss function is calculated based on the feedback weight α of the standardized corrected trajectory data to update the network parameters of the trained action policy network;

[0028] The formula for calculating the loss function is as follows:

[0029]

[0030] in, For the network parameters of the action policy network, The total number of training samples, For the first Environmental state characteristics corresponding to each sample For the first Action labels corresponding to each sample For the action policy network in parameters Down Output Action The conditional probability, For sample weights, when the first When a sample comes from standardized corrected trajectory data When the first When the samples were derived from the initial expert demonstration data, .

[0031] Furthermore, specifically, the correction frequency factor F cor The calculation formula is:

[0032]

[0033] Where, N cor N represents the total number of corrections. pos N represents the number of times the robot has successfully executed autonomously. neg This represents the number of times the robot failed to execute autonomously.

[0034] Preferably, the initial number of expert demonstration data is 10-20.

[0035] A robot learning system based on agile iterative aggregation, the system comprising:

[0036] The robot itself;

[0037] An edge computing device configured to perform the intelligent robot handling control method as described above;

[0038] The edge computing device includes:

[0039] The initial policy training module extracts environmental state features and corresponding action labels from multiple initial expert demonstration datasets, and uses the behavior cloning algorithm to pre-train the robot action policy network to obtain the initial action policy network.

[0040] The reactive data acquisition module allows the robot to autonomously perform its current actions in the environment and detect those actions.

[0041] When the robot's actions are inconsistent with the preset actions, the interactive device will intervene and take over control to correct the robot's current actions and store the corrected trajectory data.

[0042] The agile data processing module processes the corrected trajectory data through a delay compensation truncation mechanism and a trajectory standardization resampling algorithm to obtain standardized corrected trajectory data.

[0043] The dynamic weighted aggregation module performs dynamic weighted aggregation on the standardized correction trajectory data to obtain the aggregated dataset.

[0044] The policy update module inputs the aggregated dataset into the initial action policy network for training. Through iterative training, the robot's actions are kept consistent with the preset actions, thus obtaining the final action policy network.

[0045] A computer device, comprising:

[0046] processor;

[0047] Memory, used to store executable instructions;

[0048] The processor is used to read the executable instructions from the memory and execute the executable instructions to implement the intelligent robot handling control method as described above.

[0049] A computer-readable storage medium storing a computer program that, when executed by a processor, causes the processor to implement the intelligent robot handling control method described above.

[0050] The beneficial effects of the robot learning method based on agile iterative aggregation of the present invention are:

[0051] This invention maximizes the effectiveness of each correction by using delay compensation, lightweight standardization, and dynamic weighting mechanisms for interactive correction trajectory data, under the constraint of very limited initial demonstration data. This allows for robust training of complex robot operation skills with extremely low human cost and high efficiency. Attached Figure Description

[0052] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments.

[0053] Figure 1 This is a schematic diagram of the method flow of Embodiment 1 of the present invention.

[0054] Figure 2 This is a schematic diagram of the overall performance thermodynamic matrix of the test according to Embodiment 1 of the present invention.

[0055] Figure 3 This is a comparative schematic diagram of the trajectory standardization resampling algorithm introduced in Embodiment 1 of the present invention.

[0056] Figure 4 This is a comparative schematic diagram of the delay compensation truncation mechanism introduced in Embodiment 1 of the present invention.

[0057] Figure 5 The introduction of the correction frequency factor F in Embodiment 1 of this invention cor A comparison diagram of dynamic weighting.

[0058] Figure 6 This is a comparative diagram illustrating the use of an interactive device to correct the robot's movements in Embodiment 1 of the present invention.

[0059] Figure 7 This is a schematic diagram of the system structure of Embodiment 2 of the present invention.

[0060] Figure 8This is a schematic diagram of the computer device structure according to Embodiment 3 of the present invention.

[0061] In the figure, 20 is the robot body; 21 is the edge computing device; 211 is the initial policy training module; 212 is the reactive data acquisition module; 213 is the agile data processing module; 214 is the dynamic weighted aggregation module; 215 is the policy update module; 10 is the computer equipment; 1002 is the processor; 1004 is the memory; and 1006 is the transmission device. Detailed Implementation

[0062] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.

[0063] Example 1:

[0064] This application provides a robot learning method based on agile iterative aggregation, such as... Figure 1 As shown, the method includes the following steps:

[0065] S1. Based on multiple initial expert demonstration datasets, extract environmental state features and corresponding action labels, and use the behavior cloning algorithm to pre-train the robot action policy network to obtain the initial action policy network; in the specific implementation method of this embodiment, the action policy network is a convolutional neural network LSTM.

[0066] In this embodiment, the initial number of expert demonstration data points is 10-20, significantly reducing the reliance on large-scale expert data under extremely small sample constraints. In baseline operation tasks such as unplugging plugs and pressing buttons, the success rate achieves a qualitative leap compared to traditional interactive learning methods, going from no capability to a success rate exceeding 70% or even 90%.

[0067] S2, the robot autonomously performs the current action in the environment and detects the current action; when the robot's action is inconsistent with the preset action, the interactive device intervenes and takes over the control, corrects the robot's current action, and stores the corrected trajectory data; otherwise, the strategy is executed normally.

[0068] S3 processes the corrected trajectory data through a delay compensation truncation mechanism and a trajectory standardization resampling algorithm to obtain standardized corrected trajectory data.

[0069] S4 performs dynamic weighted aggregation on the standardized correction trajectory data to obtain the aggregated dataset.

[0070] S5. Input the aggregated dataset into the initial action policy network for training, and then return to step S2 for iterative training until the robot's actions are consistent with the preset actions, thus obtaining the final action policy network.

[0071] In this embodiment, in step S2, the robot autonomously performs its current action in the environment, which is then detected by a human expert. When the robot's action deviates from the preset action, such as when the robot's action deviates from expectations or an error occurs, an interactive device intervenes and takes over control to correct the robot's current action. This involves providing a correction trajectory that guides the robot back to the correct state, and this correction trajectory includes a state-action pair sequence. This embodiment allows the robot to autonomously operate to an out-of-distribution state before intervention is applied. Compared to a fully automated demonstration provided by humans, this protocol can collect higher density and more targeted high-quality boundary state correction trajectories.

[0072] In this embodiment, step S3 specifically includes the following steps:

[0073] S31, acquire correction trajectory data, and truncate the correction trajectory data for the initial preset number of steps through the time window processor; compared with directly adding unprocessed human feedback, the delay compensation truncation mechanism, by setting a preset number of steps (such as 10 steps) for front-end data truncation through the time window processor, can maximize the elimination of erroneous "state-action" mapping during delayed reaction, completely severing the association between the edge state that leads to failure and panic correction actions, resulting in the most significant improvement in success rate.

[0074] S32 uses an interpolation algorithm to resample and align the length of the truncated corrected trajectory data to the standard sequence length, and executes a hard retention strategy for critical states at the ends to obtain standardized corrected trajectory data. The trajectory standardization resampling algorithm overcomes the gradient explosion or vanishing problem caused by varying timing and duration of human intervention at different stages. Compared to direct truncation or mirror extension, it can more stably guide the convergence of recurrent action policy networks (such as LSTM), improving the training stability of multi-round iterative data aggregation. Furthermore, by executing a hard retention strategy for critical states at the ends—that is, by directly anchoring and retaining them instead of performing smooth interpolation—it prevents distortion of critical precision actions caused by interpolation.

[0075] In this embodiment, step S4 specifically includes the following steps:

[0076] S41, Calculate the correction frequency factor F based on the number of corrections and the total number of autonomous executions by the robot. cor The calculation formula is:

[0077]

[0078] Where, N corN represents the total number of corrections. pos N represents the number of times the robot has successfully executed autonomously. neg This represents the number of times the robot failed to execute autonomously.

[0079] S42, based on the correction frequency factor F cor The feedback weight α of the standardized correction trajectory data is dynamically acquired, specifically including: .

[0080] S43, the standardized corrected trajectory and the corresponding weight α are input into the replay buffer for weighted aggregation to form the aggregated dataset.

[0081] It should be noted that this invention is based on the correction frequency factor F cor Dynamic weighting effectively avoids the overfitting problem caused by fixed and uniform weights, allowing the policy network to focus its attention on specific high-value failure nodes (such as alignment and grasping moments), thus achieving "targeted treatment" based on human feedback.

[0082] In this embodiment, in step S5, a loss function is calculated based on the feedback weight α of the standardized corrected trajectory data to update the network parameters of the trained action policy network; further, the loss function calculation formula is:

[0083] in, For the network parameters of the action policy network, The total number of training samples, For the first Environmental state characteristics corresponding to each sample For the first Action labels corresponding to each sample For the action policy network in parameters Down Output Action The conditional probability, For sample weights, when the first When a sample comes from standardized corrected trajectory data When the first When the samples were derived from the initial expert demonstration data, The action policy network is backpropagated based on the loss function to update the network parameters.

[0084] Let's take the robot performing the "Unplug Charger" task as an example to illustrate this:

[0085] When the robot autonomously performed the task of unplugging the charger, due to minimal initial data distribution shift, the gripper failed to align precisely when approaching the charger, deviating 2 cm from the plug. A human expert noticed the alignment error and took over control by pressing the takeover button on the interactive device (denoted as T0). Using the operating handle, the gripper was moved back to the correct position, aligned, and the charger unplugged (denoted as T45). The corrected trajectory data, consisting of 45 state-action pairs, was maintained from T0 to T45. The time window processor automatically truncated the first 10 steps of this 45-step trajectory, retaining only the 35 corrected trajectory steps from step 11 to 45, setting the standard action sequence length to 150 steps. These 35 corrected trajectory steps from step 11 to 45 were stretched to 150 steps using linear interpolation. Specifically, for the final two states, "gripper closed" and "pull outwards," the system did not perform smooth interpolation but directly anchored and retained them to ensure that key motion features were not lost. During this process, the robot autonomously attempted 100 times, with human intervention correcting 15 times. The correction frequency factor F was calculated. cor =0.15, corresponding to a feedback weight α=1.2. The standardized correction trajectory data from 150 steps and the feedback weight α are aggregated and input into the action policy network for training. When calculating the loss function, the gradient generated by the standardized correction trajectory data is amplified by a factor of 1.2, forcing the action policy network to more deeply "remember" this alignment correction. After the update, the robot's alignment accuracy at this node is significantly improved.

[0086] Test Description: To verify this embodiment, rigorous comparative experiments were conducted on six standard desktop robot operation tasks (RLBench simulation environment) that included various physical constraints and long-range operations. All comparison methods were limited to training under the extremely challenging condition of 20 initial expert demonstration data (Few-shot).

[0087] Figure 2 The overall performance heatmap visually compares the success rates of this embodiment (Agile Iterative Aggregation, AIA) with five existing mainstream imitation learning methods across six complex tasks. The numerical values ​​and color intensity in the matrix collectively represent the task success rate (%). As shown in the graph, the performance of this embodiment comprehensively surpasses and exceeds existing technologies across all task dimensions. Particularly in tasks relying on high-precision action correction (such as PB and UC), this embodiment effectively eliminates data contamination caused by human feedback latency and performs precise weighting.

[0088] Figures 3 to 6 The figure above shows the verification of the necessity of the ablation experiment for each step of this implementation (ablation experiment).

[0089] Figure 3This diagram illustrates the comparison after introducing the trajectory standardization resampling algorithm in this embodiment. Specifically, when using the trajectory standardization resampling strategy for trajectory processing in this embodiment, the task success rate is 81%. As a comparison, the task success rate is 65% when using a linear interpolation plus linear truncation strategy; 47% when using a mirror expansion plus linear truncation strategy; 72% when using a linear interpolation plus directional truncation strategy; and 58% when using a linear interpolation plus uniform sampling strategy. This demonstrates that compared to trajectory processing methods such as direct truncation, mirror expansion, or simple sampling, the trajectory standardization resampling strategy used in this embodiment can better maintain the continuity and consistency of the trajectory, thereby more stably guiding model convergence and improving the task success rate.

[0090] Figure 4 This diagram illustrates the comparison after introducing the delay compensation truncation mechanism in this embodiment. Specifically, this embodiment employs the delay compensation truncation mechanism, and when the truncation length is set to 10 steps, the task success rate reaches 80%. As a comparative example, when the truncation length is set to 3, 5, 7, 12, and 15 steps, the task success rates are 55%, 65%, 75%, 78%, and 72%, respectively. The above results show that directly introducing human feedback or setting the compensation length unreasonably can easily lead to delay mapping errors, thereby affecting model performance. The delay compensation truncation mechanism used in this embodiment can retain effective correction information while minimizing erroneous mapping caused by feedback delay, thus achieving a more significant improvement in success rate.

[0091] Figure 5 This embodiment introduces a correction frequency factor F. cor A comparison diagram after the dynamic weighting mechanism. Specifically, this embodiment uses a correction frequency factor F. cor When using the dynamic weighting strategy, the task success rate is 67%. As a comparison, the baseline scheme has a task success rate of 22%, the time decay weighting scheme has a task success rate of 40%, the sample proportion weighting scheme has a task success rate of 32%, the fixed weight scheme based on success rate has a task success rate of 55%, and the sliding window scheme has a task success rate of 30%. This demonstrates that the proposed strategy based on the correction frequency factor F... cor The dynamic weighting mechanism can adaptively adjust the sample weights according to the actual changes in the interactive correction signal. Compared with fixed weights or static rule schemes, it can effectively avoid overfitting and further improve the model training effect.

[0092] Figure 6This illustration shows a comparison of robot motion correction after the introduction of an interactive device in this embodiment. Specifically, in the PB task, when this embodiment intervenes and takes over control through the interactive device to correct the robot's current motion, the task success rate is 85%; as a comparison, the task success rate is 60% when using real-time guidance. In the TLS task, the task success rate of this embodiment is 75%, while the task success rate of the comparison is 65%. This demonstrates that, compared to the active demonstration or real-time guidance method in the comparison, this embodiment, by directly intervening in the correction process of the robot's current motion through the interactive device, can provide more targeted and high-quality correction signals when the robot is in an edge-of-state or failure-prone state, thereby improving the success rate of robot task execution.

[0093] In summary, this invention, with the limitation of very little initial demonstration data, maximizes the effectiveness of each correction by using delay compensation, lightweight standardization, and dynamic weighting mechanisms for interactive correction trajectory data. This allows for robust training of complex robot operation skills with extremely low human cost and high efficiency.

[0094] Example 2:

[0095] This application provides a robot learning system based on agile iterative aggregation, such as... Figure 7 As shown, the system includes:

[0096] Robot body 20;

[0097] Edge computing device 21 is configured to perform the intelligent robot handling control method described above.

[0098] Furthermore, edge computing devices include:

[0099] The initial policy training module 211 extracts environmental state features and corresponding action labels based on multiple initial expert demonstration datasets, and uses the behavior cloning algorithm to pre-train the robot action policy network to obtain the initial action policy network.

[0100] The reactive data acquisition module 212 enables the robot to autonomously perform its current action in the environment and detect the current action.

[0101] When the robot's actions deviate from the preset actions, the system intervenes through an interactive device to take over control, correct the robot's current actions, and store the corrected trajectory data.

[0102] The agile data processing module 213 processes the corrected trajectory data through a delay compensation truncation mechanism and a trajectory standardization resampling algorithm to obtain standardized corrected trajectory data.

[0103] The dynamic weighted aggregation module 214 performs dynamic weighted aggregation on the standardized correction trajectory data to obtain the aggregated dataset.

[0104] The policy update module 215 inputs the aggregated dataset into the initial action policy network for training. Through iterative training, the robot's actions are consistent with the preset actions, and the final action policy network is obtained.

[0105] The foregoing Figure 1 The various variations and specific examples of the intelligent robot handling control method in Embodiment 1 are also applicable to the intelligent robot handling control device in this embodiment. Through the foregoing detailed description of the intelligent robot handling control method, those skilled in the art can clearly understand the implementation method of the intelligent robot handling control device in this embodiment. Therefore, for the sake of brevity, it will not be described in detail here.

[0106] Example 3:

[0107] This application provides a computer device including a processor and a memory. The memory stores at least one instruction or at least one program, which is loaded and executed by the processor to implement an intelligent robot handling control method as provided in the above method embodiments.

[0108] Figure 8 A schematic diagram of the hardware structure of a device for implementing the intelligent robot handling control method provided in the embodiments of this application is shown. The device can participate in or include the apparatus or system provided in the embodiments of this application. Figure 8 As shown, the computer device 10 may include one or more processors 1002 (the processor may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.), a memory 1004 for storing data, and a transmission device 1006 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the I / O interface), a network interface, a power supply, and / or a camera. Those skilled in the art will understand that... Figure 8 The structure shown is for illustrative purposes only and does not limit the structure of the aforementioned electronic device. For example, computer device 10 may also include... Figure 8 The more or fewer components shown, or having the same Figure 8 The different configurations shown.

[0109] It should be noted that the aforementioned one or more processors and / or other data processing circuits are generally referred to herein as "data processing circuits". These data processing circuits may be embodied, in whole or in part, in software, hardware, firmware, or any other combination thereof. Furthermore, the data processing circuit may be a single, independent processing module, or may be integrated, in whole or in part, into any other element within the computer device 10 (or mobile device). As involved in the embodiments of this application, the data processing circuit serves as a processor control mechanism (e.g., selection of a variable resistor termination path connected to an interface).

[0110] The memory 1004 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the intelligent robot handling control method in this embodiment of the application. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory 1004, thereby implementing the aforementioned method. The memory 1004 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 1004 may further include memory remotely located relative to the processor, and these remote memories can be connected to the computer device 10 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0111] The transmission device 1006 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the communication provider of the computer device 10. In one example, the transmission device 1006 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 1006 may be a Radio Frequency (RF) module, used for wireless communication with the Internet.

[0112] The display may be, for example, a touchscreen liquid crystal display (LCD) that allows the user to interact with the user interface of the computer device 10 (or mobile device).

[0113] Example 4:

[0114] This application also provides a computer-readable storage medium, which can be disposed in a server to store at least one instruction or at least one program related to implementing an intelligent robot handling control method in the method embodiment. The at least one instruction or the at least one program is loaded and executed by the processor to implement the intelligent robot handling control method provided in the above method embodiment.

[0115] Optionally, in this embodiment, the storage medium may be located at at least one of the multiple network servers in a computer network. Optionally, in this embodiment, the storage medium may include, but is not limited to, various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0116] Example 5:

[0117] This invention also provides a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform an intelligent robot handling control method provided in the various optional embodiments described above.

[0118] It should be noted that the order of the embodiments described above is merely for descriptive purposes and does not represent the superiority or inferiority of the embodiments. Furthermore, the above description focuses on specific embodiments of this application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps described in the claims can be performed in a different order than that shown in the embodiments and still achieve the desired results. Additionally, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired results. In some implementations, multitasking and parallel processing are also possible or may be advantageous.

[0119] The various embodiments in this application are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the device, equipment, and storage medium embodiments are basically similar to the method embodiments, so the descriptions are relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0120] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0121] Based on the above-described preferred embodiments of the present invention, and through the foregoing description, those skilled in the art can make various changes and modifications without departing from the inventive concept. The technical scope of this invention is not limited to the contents of the specification, but must be determined according to the scope of the claims.

Claims

1. A robot learning method based on agile iterative aggregation, characterized in that, The method includes the following steps: S1. Based on multiple initial expert demonstration datasets, extract environmental state features and corresponding action labels, and use the behavior cloning algorithm to pre-train the robot action policy network to obtain the initial action policy network. S2, the robot autonomously performs the current action in the environment and detects the current action; When the robot's actions are inconsistent with the preset actions, the interactive device will intervene and take over control to correct the robot's current actions and store the corrected trajectory data. S3, the corrected trajectory data is processed by a delay compensation truncation mechanism and a trajectory standardization resampling algorithm to obtain standardized corrected trajectory data; S4. Dynamically weighted aggregate the standardized correction trajectory data to obtain the aggregated dataset. S5. Input the aggregated dataset into the initial action policy network for training, and then return to step S2 for iterative training until the robot's actions are consistent with the preset actions, thus obtaining the final action policy network.

2. The robot learning method based on agile iterative aggregation according to claim 1, characterized in that, Step S3 specifically includes the following steps: S31, acquire the correction trajectory data, and truncate the correction trajectory data for the initial preset number of steps using a time window processor; S32, the length of the truncated corrected trajectory data is resampled and aligned to the standard sequence length using an interpolation algorithm, and a hard retention strategy for the critical end state is executed to obtain the standardized corrected trajectory data.

3. The robot learning method based on agile iterative aggregation according to claim 1, characterized in that, Step S4 specifically includes the following steps: S41, calculate a correction frequency factor F based on the number of corrections and the total number of times the robot autonomously executes cor ; S42, according to the correction frequency factor F cor dynamically obtaining a correction weight a of the normalized correction trajectory data S43, the standardized corrected trajectory and the corresponding weight α are input into the replay buffer for weighted aggregation to form the aggregated dataset.

4. The robot learning method based on agile iterative aggregation according to claim 3, characterized in that, Step S42 specifically includes: 。 5. The robot learning method based on agile iterative aggregation according to claim 1, characterized in that, In step S5, a loss function is calculated based on the feedback weight α of the standardized corrected trajectory data to update the network parameters of the trained action policy network. The formula for calculating the loss function is as follows: in, For the network parameters of the action policy network, The total number of training samples, For the first Environmental state characteristics corresponding to each sample For the first Action labels corresponding to each sample For the action policy network in parameters Down Output Action The conditional probability, For sample weights, when the first When a sample comes from standardized corrected trajectory data When the first When the samples were derived from the initial expert demonstration data, .

6. The robot learning method based on agile iterative aggregation according to claim 1, characterized in that, The correction frequency factor F cor The calculation formula is: Where, N cor N represents the total number of corrections. pos N represents the number of times the robot has successfully executed autonomously. neg This represents the number of times the robot failed to execute autonomously.

7. The robot learning method based on agile iterative aggregation according to claim 1, characterized in that, The initial expert demonstration data consists of 10-20 records.

8. A robot learning system based on agile iterative aggregation, characterized in that, The system includes: Robot body (20); An edge computing device (21) configured to perform the intelligent robot handling control method as described in any one of claims 1 to 7; The edge computing device includes: The initial policy training module (211) extracts environmental state features and corresponding action labels based on multiple initial expert demonstration datasets, and uses the behavior cloning algorithm to pre-train the robot action policy network to obtain the initial action policy network. The reactive data acquisition module (212) enables the robot to autonomously perform the current action in the environment and detect the current action; When the robot's actions are inconsistent with the preset actions, the interactive device will intervene and take over control to correct the robot's current actions and store the corrected trajectory data. The agile data processing module (213) processes the corrected trajectory data through a delay compensation truncation mechanism and a trajectory standardization resampling algorithm to obtain standardized corrected trajectory data; The dynamic weighted aggregation module (214) performs dynamic weighted aggregation on the standardized correction trajectory data to obtain the aggregated dataset. The policy update module (215) inputs the aggregated dataset into the initial action policy network for training. Through iterative training, the robot's actions are consistent with the preset actions, and the final action policy network is obtained.

9. A computer device, characterized in that, include: processor; Memory, used to store executable instructions; The processor is configured to read the executable instructions from the memory and execute the executable instructions to implement the intelligent robot handling control method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, causes the processor to implement the intelligent robot handling control method as described in any one of claims 1 to 7.

Images