Charging gun plugging task execution method and device, robot, and storage medium
By using a policy network trained offline and adjusted online, the generalization problem of charging gun plugging and unplugging tasks in unstructured environments was solved, enabling adaptive plugging and unplugging task execution for new models of charging guns and vehicles.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- UBTECH ROBOTICS CORP LTD
- Filing Date
- 2024-12-24
- Publication Date
- 2026-05-26
AI Technical Summary
Existing algorithms for charging gun plugging and unplugging tasks lack generalization ability in open, unstructured environments and cannot successfully handle plugging and unplugging tasks for charging gun models and target vehicles that have not been pre-set.
The pre-trained policy network is obtained through offline training, and behavior is cloned using historical data. Then, the network parameters are adjusted through online training to obtain the target policy network, thereby realizing the online adaptive charging gun plugging and unplugging task.
It can adaptively handle the plugging and unplugging tasks of new models of charging guns and vehicles in real-world environments, improving the success rate and adaptability of task execution.
Smart Images
Figure CN119886663B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of computer motion control, and in particular to a method, apparatus, robot, and storage medium for performing a charging gun plugging and unplugging task. Background Technology
[0002] Existing charging gun plugging / unplugging tasks are all performed in structured environments, operating a single charging gun and a single vehicle model. However, charging guns are diverse, and actual usage environments are complex. Existing algorithms are highly specific but lack generalization ability. In open, unstructured environments (such as parking lots), deployed humanoid robots cannot successfully handle charging gun plugging / unplugging tasks where the charging gun model and target vehicle have not been pre-defined. Summary of the Invention
[0003] In view of this, the purpose of the present invention is to overcome the shortcomings of the prior art and provide a method, device, robot and storage medium for performing charging gun plugging and unplugging tasks. This method is used to clone behavior through offline training to obtain a pre-trained policy network, and then learn new objects that fall outside the offline training distribution through online training to obtain a target policy network. The target policy network can then be used to realize online adaptive charging gun plugging and unplugging tasks.
[0004] This invention provides the following technical solution:
[0005] In a first aspect, the present invention proposes a method for performing a charging gun plugging and unplugging task, comprising:
[0006] Acquire historical initial observation images, historical movement operation primitives, and historical end images of the robot when it performs historical charging gun plugging and unplugging tasks;
[0007] The initial policy network is trained offline based on the historical initial observation image, the historical movement operation primitives, and the historical end image to obtain a pre-trained policy network.
[0008] The robot acquires the current initial observation image when it performs the current charging gun plugging and unplugging task, and predicts the current movement operation primitive based on the pre-trained policy network.
[0009] The pre-trained policy network is trained online based on the current mobile operation primitive to obtain the target policy network;
[0010] The total control amount corresponding to the new charging gun insertion / removal task is obtained according to the target policy network, and the robot is controlled to execute the new charging gun insertion / removal task according to the total control amount.
[0011] In one embodiment, the step of training the pre-trained policy network online based on the current mobile operation primitive to obtain the target policy network includes:
[0012] The robot is controlled to execute the current charging gun plugging / unplugging task according to the current movement operation primitive, and the current completed image corresponding to the current charging gun plugging / unplugging task is obtained;
[0013] The network parameters of the pre-trained policy network are adjusted based on the current end image to obtain the target policy network.
[0014] In one embodiment, adjusting the network parameters of the pre-trained policy network based on the current end image to obtain the target policy network includes:
[0015] The current task execution status is obtained based on the currently completed image using a visual language model;
[0016] The current reward value is determined based on the current task execution status.
[0017] The network parameters of the pre-trained policy network are adjusted based on the current reward value to obtain the target policy network.
[0018] In one embodiment, determining the current reward value based on the current task execution status includes:
[0019] If the current task execution status is successful, then the current reward value is the first value;
[0020] If the current task execution status is failure, then the current reward value is the second value;
[0021] If the current task execution status exceeds the limit, then the current reward value is the third value.
[0022] In one embodiment, the step of offline training of the initial policy network based on the historical initial observation image, the historical movement operation primitives, and the historical end image to obtain a pre-trained policy network includes:
[0023] Offline training data is obtained based on the historical initial observation image, the historical movement operation primitive, and the historical end image;
[0024] The initial policy network is trained offline based on the offline training data to obtain the pre-trained policy network.
[0025] In one embodiment, obtaining offline training data based on the historical initial observation image, the historical movement operation primitive, and the historical end image includes:
[0026] The execution status of historical tasks can be obtained based on the historical end image.
[0027] The historical reward value is determined based on the historical task execution status.
[0028] The offline training data is determined based on the historical movement primitives, the historical initial observation image, the historical end image, and the historical reward value.
[0029] In one embodiment, obtaining the total control quantity corresponding to the new charging gun plugging / unplugging task based on the target policy network includes:
[0030] Obtain the new initial observation image corresponding to the new charging gun plugging / unplugging task;
[0031] The target policy network is used to predict the new initial observation image to obtain a new movement operation primitive, and the control quantity is obtained based on the new movement operation primitive.
[0032] Secondly, the present invention provides a charging gun plugging / unplugging task execution device, comprising:
[0033] The first acquisition module is used to acquire historical initial observation images, historical movement operation primitives, and historical end images when the robot performs historical charging gun plugging and unplugging tasks.
[0034] The offline module is used to train the initial policy network offline based on the historical initial observation image, the historical movement operation primitives, and the historical end image to obtain a pre-trained policy network.
[0035] The second acquisition module is used to acquire the current initial observation image when the robot performs the current charging gun plugging and unplugging task, and predict the current movement operation primitive based on the current initial observation image according to the pre-trained policy network.
[0036] An online module is used to train the pre-trained policy network online based on the current mobile operation primitive to obtain the target policy network;
[0037] The execution module is used to obtain the total control amount corresponding to the new charging gun insertion / removal task based on the target policy network, and control the robot to execute the new charging gun insertion / removal task based on the total control amount.
[0038] Thirdly, the present invention proposes a robot including a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, it implements the charging gun plugging and unplugging task execution method as described in the first aspect.
[0039] Fourthly, the present invention provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the charging gun plugging and unplugging task execution method as described in the first aspect.
[0040] This invention discloses a method, apparatus, robot, and storage medium for executing charging gun insertion / removal tasks. The method involves acquiring historical initial observation images, historical movement operation primitives, and historical end images of the robot executing previous charging gun insertion / removal tasks. An initial policy network is trained offline based on these images to obtain a pre-trained policy network. A current initial observation image is acquired when the robot executes the current charging gun insertion / removal task. The current initial observation image is predicted using the pre-trained policy network to obtain a current movement operation primitive. The pre-trained policy network is then trained online based on the current movement operation primitive to obtain a target policy network. A control quantity corresponding to the new charging gun insertion / removal task is obtained from the target policy network, and the robot is controlled to execute the new task based on this control quantity. In this way, offline training of the initial policy network on historical data allows for learning using behavior cloning to obtain a pre-trained policy network. Further online training of the pre-trained network strategy on insertion / removal tasks in a real-world environment allows for learning a comprehensive strategy for new objects falling outside the offline training distribution. This ensures that the target policy network obtained after online training can adapt to new charging gun insertion / removal tasks online. Attached Figure Description
[0041] To more clearly illustrate the technical solution of the present invention, the accompanying drawings used in the embodiments will be briefly described below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope of protection of the present invention. In the various drawings, similar components are numbered similarly.
[0042] Figure 1 A flowchart illustrating the charging gun plugging / unplugging task execution method proposed in this embodiment is shown.
[0043] Figure 2 Another flowchart illustrating the charging gun plugging / unplugging task execution method proposed in this embodiment is shown.
[0044] Figure 3 A schematic diagram of another process for the charging gun plugging and unplugging task execution method proposed in this embodiment is shown;
[0045] Figure 4 This illustration shows another flowchart of the charging gun plugging and unplugging task execution method proposed in this embodiment;
[0046] Figure 5 A schematic diagram of the charging gun plugging and unplugging task execution device proposed in this embodiment is shown.
[0047] Explanation of reference numerals in the attached diagram:
[0048] 500 - Charging gun plugging / unplugging task execution device; 501 - First acquisition module; 502 - Offline module; 503 - Second acquisition module; 504 - Online module; 505 - Execution module. Detailed Implementation
[0049] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.
[0050] The components of the embodiments of the invention described and illustrated herein can typically be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention.
[0051] In the following, the terms “comprising,” “having,” and their cognates, which may be used in various embodiments of the invention, are intended only to indicate a particular feature, number, step, operation, element, component, or combination thereof, and should not be construed as excluding, firstly, the presence of one or more other features, numbers, steps, operations, elements, components, or combinations thereof, or adding the possibility of one or more features, numbers, steps, operations, elements, components, or combinations thereof.
[0052] Furthermore, the terms "first," "second," and "third" are used only to distinguish descriptions and should not be interpreted as indicating or implying relative importance.
[0053] Unless otherwise specified, all terms used herein (including technical and scientific terms) shall have the same meaning as commonly understood by one of ordinary skill in the art to which the various embodiments of the invention pertain. Terms (such as those defined in commonly used dictionaries) shall be interpreted as having the same meaning as in their contextual meaning in the relevant technical field and shall not be interpreted as having an idealized or overly formal meaning, unless clearly defined in the various embodiments of the invention.
[0054] Example 1
[0055] This disclosure provides a method for executing a charging gun plugging and unplugging task. The method involves cloning behavior through offline training to obtain a pre-trained policy network, and then learning new objects that fall outside the offline training distribution through online training to obtain a target policy network. This target policy network can then be used to achieve an online adaptive charging gun plugging and unplugging task.
[0056] Please see Figure 1 A method for performing a charging gun plugging and unplugging task includes steps S101 to S105, and each step is described in detail below.
[0057] Step S101: Obtain the historical initial observation image, historical movement operation primitive, and historical end image when the robot performs the historical charging gun plugging and unplugging task.
[0058] In this embodiment, the initial historical observation image is the image of the charging port area where the charging gun has not yet been inserted when the robot performs the historical charging gun insertion and removal task; the historical movement operation primitive is a series of actions performed by the robot when performing the historical charging gun insertion and removal task; the historical end image is an image of the charging port area taken again after the series of actions of the historical charging gun insertion and removal task are completed, which can be used to determine whether the charging gun is inserted into the charging port.
[0059] Among them, the historical initial observation image I ′ s For RGBD images, image segmentation and object recognition are needed to determine the center of the charging port, and image depth information is used to determine the depth. Based on the port center and depth information, the required charging gun model can be identified. (End of history, image I) ′ e As an RGB image, it can be directly fed into a large model to obtain the corresponding historical task execution status. Each movement operation primitive contains the instructions and information required for the robot to perform a specific action, such as the instruction or information to select the required model of charging gun.
[0060] It should be noted that the movement operation primitive is in open-loop form. When the robot is performing an action, it does not adjust its actions based on real-time feedback from the environment, but instead executes the action according to the trajectory corresponding to the pre-set movement operation primitive.
[0061] Step S102: The initial policy network is trained offline based on the historical initial observation image, the historical movement operation primitive, and the historical end image to obtain a pre-trained policy network.
[0062] In this embodiment, behavior cloning is performed based on historical initial observation images, historical movement operation primitives, and historical end images to train the initial policy network offline, resulting in a pre-trained policy network. The goal of offline training is to predict appropriate movement operation primitives based on the input initial observation images.
[0063] Please see Figure 2 In one specific embodiment, step S102 includes steps S1021 to S1022, and each step is described in detail below.
[0064] Step S1021: Obtain offline training data based on the historical initial observation image, the historical movement operation primitive, and the historical end image.
[0065] In this embodiment, offline training data is obtained based on historical initial observation images, historical movement operation primitives, and historical end images, which are used to train the initial policy network.
[0066] In one specific embodiment, step S1021 includes: obtaining historical task execution status based on the historical end image; determining historical reward value based on the historical task execution status; and determining the offline training data based on the historical movement operation primitive, the historical initial observation image, the historical end image, and the historical reward value.
[0067] In this embodiment, the historical task execution status is determined based on the historical end image. The historical task execution status refers to the insertion and removal execution status after the robot performs the historical charging gun insertion and removal task according to the historical movement operation primitives.
[0068] The historical task execution status includes whether the task was successful, failed, or exceeded force or speed limits. Whether force or speed limits were exceeded needs to be determined based on preset force and joint speed thresholds. A historical movement operation primitive is a sequence of N constrained movement operation primitives.
[0069] Historical reward values are determined based on the historical task execution. Specifically, if the historical task execution resulted in the charging gun being successfully inserted into the charging port, the historical reward value can be 1; if the historical task execution resulted in the charging gun failing to be inserted into the charging port, the historical reward value is 0; if the historical task execution resulted in the force or speed exceeding the limit, the historical reward value is -1. By assigning different reward values to different task execution results, the system can distinguish the severity of the behavior.
[0070] Move history primitive Historical Initial Observation Image I ′ s End of History Image I ′ e This data, along with the historical reward value R′, forms a set of offline training data.
[0071] Step S1022: Perform offline training on the initial policy network based on the offline training data to obtain the pre-trained policy network.
[0072] In this embodiment, the initial policy network is trained offline based on the information in the offline training data to obtain a pre-trained policy network, thereby training the pre-trained policy network to select actions that maximize reward value.
[0073] Step S103: Obtain the current initial observation image when the robot performs the current charging gun plugging and unplugging task, and predict the current initial observation image according to the pre-trained policy network to obtain the current movement operation primitive.
[0074] In this embodiment, please refer to Figure 3 In a real-world environment, acquire the initial observation image I of the robot performing the current charging gun plugging / unplugging task. s The current initial observation image I S Input to the pre-trained policy network π θ Make a prediction to obtain the current movement operation primitive. Current movement primitive at this time For the pre-trained policy network π θ Based on the current initial observation image I s The generated prediction results.
[0075] Step S104: Train the pre-trained policy network online according to the current mobile operation primitive to obtain the target policy network.
[0076] In this embodiment, during the actual execution of the task, the robot may encounter new situations or challenges that may not have appeared in the offline training data. To address these new challenges, new data can be collected and used to further train and optimize the target policy network.
[0077] Therefore, the pre-trained policy network can be trained online using the current mobile operation primitives predicted by the pre-trained policy network. This allows the network parameters of the pre-trained policy network to be improved according to the task requirements of the new environment, so as to obtain a target policy network that is more adapted to the new environment.
[0078] Please see Figure 4 In one specific embodiment, step S104 includes steps S1041 to S1042, and each step is described in detail below.
[0079] Step S1041: Control the robot to execute the current charging gun plugging / unplugging task according to the current movement operation primitive, and obtain the current end image corresponding to the current charging gun plugging / unplugging task.
[0080] In this embodiment, the robot uses the pre-trained policy network π θ The output prediction result, i.e., the current movement operation primitive. Execute the current charging gun plugging / unplugging task and obtain the current end image after the current charging gun plugging / unplugging task is executed. The current end image is a picture taken again of the charging area after the current charging gun plugging / unplugging task is executed, which is used to determine whether the current charging gun plugging / unplugging task is executed successfully.
[0081] Step S1042: Adjust the network parameters of the pre-trained policy network according to the current end image to obtain the target policy network.
[0082] In this embodiment, feedback is obtained based on the current ending image to adjust the network parameters of the pre-trained policy network until a stable state is reached, thus obtaining the target policy network.
[0083] In one specific embodiment, step S1042 includes: obtaining the current task execution status based on the current completed image using a visual language model; determining the current reward value based on the current task execution status; and adjusting the network parameters of the pre-trained policy network based on the current reward value to obtain the target policy network.
[0084] In this embodiment, a Visual Language Model (LVLMs) is used to determine whether the charging gun has been successfully inserted into the charging port after the current charging gun insertion / removal task is executed, based on the current completed image. This insertion status is then used as the current task execution status. The current reward value is determined based on this status, and the network parameters of the pre-trained policy network are adjusted accordingly. Through multiple iterations and adjustments, the pre-trained policy network gradually learns how to select the optimal action based on the current situation, thereby maximizing long-term rewards. When the performance of the pre-trained policy network reaches a certain preset standard or converges, the target policy network is obtained. This target policy network is more likely to predict actions that lead to high reward values in future new tasks.
[0085] In one specific embodiment, if the current task execution status is successful, the current reward value is a first value; if the current task execution status is unsuccessful, the current reward value is a second value; if the current task execution status exceeds the limit, the current reward value is a third value.
[0086] In this embodiment, if the current task execution status is successful, the current reward value is a first value, such as 1; if the current task execution status is unsuccessful, the current reward value is a second value, such as 0; if the current task execution status is that the force or speed exceeds the limit, the current reward value is a third value, such as -1.
[0087] When a task is successfully executed, giving a positive reward (such as 1) explicitly tells the policy network that this behavior is expected by the system and should therefore be adopted more frequently in the future. Conversely, when a task fails or exceeds a limit, giving a zero or negative reward (such as 0 or -1) indicates that this behavior is not expected and the policy network should avoid repeating this behavior in the future.
[0088] Step S105: Obtain the total control amount corresponding to the new charging gun insertion / removal task according to the target policy network, and control the robot to execute the new charging gun insertion / removal task according to the total control amount.
[0089] In this embodiment, the total control required to perform the new charging gun insertion / removal task is predicted using an online-trained target policy network, and the robot is controlled to perform the new charging gun insertion / removal task based on the total control.
[0090] In one specific embodiment, step S105 includes: acquiring a new initial observation image corresponding to the new charging gun insertion / removal task; predicting the new initial observation image through the target policy network to obtain a new movement operation primitive, and obtaining the total control amount based on the new movement operation primitive.
[0091] In this embodiment, a new initial observation image is acquired when the robot performs a new charging gun plugging and unplugging task, and the new initial observation image is input into the target policy network for prediction to obtain a new movement operation primitive. Based on the new movement operation primitive, the total control required for the robot to perform the new charging gun plugging and unplugging task is obtained.
[0092] In this system, the robot's base coordinate system remains stationary when inserting the charging gun. The control system includes an admittance control module and a reinforcement learning module. The parameters of the admittance control module do not change during the reinforcement learning process. When the robot performs a new charging gun insertion / removal task, the admittance control module outputs a first control variable, and the target policy network in the reinforcement learning module predicts and outputs a second control variable. The sum of the first and second control variables is used as the total control to execute the new charging gun insertion / removal task. The total control includes six components: end-effector velocity and end-effector angular velocity {v}. x ,v y ,v z ,,v roll ,v pitcy ,v yaw}
[0093] The charging gun insertion / removal task execution method proposed in this embodiment acquires historical initial observation images, historical movement operation primitives, and historical end images of the robot when performing historical charging gun insertion / removal tasks. Based on the historical initial observation images, historical movement operation primitives, and historical end images, an initial policy network is trained offline to obtain a pre-trained policy network. The current initial observation image of the robot when performing the current charging gun insertion / removal task is acquired, and the current movement operation primitive is predicted based on the pre-trained policy network. The pre-trained policy network is then trained online based on the current movement operation primitive to obtain a target policy network. The control quantity corresponding to the new charging gun insertion / removal task is obtained based on the target policy network, and the robot is controlled to perform the new charging gun insertion / removal task based on the control quantity. In this way, through offline training of historical data using the initial policy network, behavioral cloning can be used for learning to obtain the pre-trained policy network. Furthermore, the pre-trained network strategy is used for online training on insertion / removal tasks in a real environment, thereby learning a comprehensive strategy for new objects that fall outside the offline training distribution, thus ensuring that the target policy network obtained after online training can adapt to new charging gun insertion / removal tasks online.
[0094] Example 2
[0095] Furthermore, this disclosure provides a charging gun plugging / unplugging task execution device 500. Please refer to [link / reference]. Figure 5 The device includes:
[0096] The first acquisition module 501 is used to acquire the historical initial observation image, historical movement operation primitives and historical end image when the robot performs the historical charging gun plugging and unplugging task.
[0097] Offline module 502 is used to train the initial policy network offline based on the historical initial observation image, the historical movement operation primitives and the historical end image to obtain a pre-trained policy network;
[0098] The second acquisition module 503 is used to acquire the current initial observation image when the robot performs the current charging gun plugging and unplugging task, and predict the current initial observation image according to the pre-trained policy network to obtain the current movement operation primitive.
[0099] Online module 504 is used to train the pre-trained policy network online based on the current mobile operation primitive to obtain the target policy network;
[0100] The execution module 505 is used to obtain the total control amount corresponding to the new charging gun insertion / removal task according to the target policy network, and control the robot to execute the new charging gun insertion / removal task according to the total control amount.
[0101] Optionally, the online module 504 is used to control the robot to execute the current charging gun insertion / removal task according to the current movement operation primitive, and to obtain the current end image corresponding to the current charging gun insertion / removal task; and to adjust the network parameters of the pre-trained policy network according to the current end image to obtain the target policy network.
[0102] Optionally, the online module 504 is used to obtain the current task execution status based on the current completed image using a visual language model; determine the current reward value based on the current task execution status; and adjust the network parameters of the pre-trained policy network based on the current reward value to obtain the target policy network.
[0103] Optionally, the online module 504 is configured to: if the current task execution status is successful, then the current reward value is a first value; if the current task execution status is unsuccessful, then the current reward value is a second value; if the current task execution status exceeds the limit, then the current reward value is a third value.
[0104] Optionally, the offline module 502 is used to obtain offline training data based on the historical initial observation image, the historical movement operation primitive, and the historical end image; and to perform offline training on the initial policy network based on the offline training data to obtain the pre-trained policy network.
[0105] Optionally, the offline module 502 is used to obtain the historical task execution status based on the historical end image; determine the historical reward value based on the historical task execution status; and determine the offline training data based on the historical movement operation primitive, the historical initial observation image, the historical end image, and the historical reward value.
[0106] Optionally, the execution module 505 is used to acquire a new initial observation image corresponding to the new charging gun insertion / removal task; predict the new initial observation image through the target policy network to obtain a new movement operation primitive, and obtain the control total amount based on the new movement operation primitive.
[0107] The apparatus provided in this embodiment can execute the steps of the charging gun plugging and unplugging task execution method provided in Embodiment 1. To avoid repetition, it will not be described again.
[0108] The charging gun insertion / removal task execution device and robot proposed in this embodiment acquire historical initial observation images, historical movement operation primitives, and historical end images of the robot when performing historical charging gun insertion / removal tasks. Based on the historical initial observation images, historical movement operation primitives, and historical end images, an initial policy network is trained offline to obtain a pre-trained policy network. The current initial observation image of the robot when performing the current charging gun insertion / removal task is acquired, and the current movement operation primitive is predicted based on the pre-trained policy network. The pre-trained policy network is then trained online based on the current movement operation primitive to obtain a target policy network. The control quantity corresponding to the new charging gun insertion / removal task is obtained based on the target policy network, and the robot is controlled to perform the new charging gun insertion / removal task based on the control quantity. Thus, by offline training of historical data using the initial policy network, behavioral cloning can be used for learning to obtain the pre-trained policy network. Furthermore, the pre-trained network strategy is used for online training on insertion / removal tasks in a real environment, thereby learning a comprehensive strategy for new objects that fall outside the offline training distribution. This ensures that the target policy network obtained after online training can adapt to new charging gun insertion / removal tasks online.
[0109] Example 3
[0110] Furthermore, this disclosure provides a robot including a memory and a processor. The memory stores a computer program, which, when executed by the processor, implements the charging gun plugging and unplugging task execution method described in Embodiment 1.
[0111] The robot provided in this embodiment can execute the steps of the charging gun plugging and unplugging task execution method provided in Embodiment 1. To avoid repetition, the steps will not be repeated.
[0112] Example 4
[0113] This disclosure provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the charging gun plugging / unplugging task execution method described in Embodiment 1.
[0114] In this embodiment, the computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, etc.
[0115] The computer-readable storage medium provided in this embodiment can implement the charging gun plugging and unplugging task execution method provided in Embodiment 1. To avoid repetition, it will not be described again here.
[0116] In all examples shown and described herein, any specific values should be interpreted as merely exemplary and not as limitations; therefore, other examples of exemplary embodiments may have different values.
[0117] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.
[0118] The above-described embodiments are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention.
Claims
1. A method for performing a charging gun plugging and unplugging task, characterized in that, include: Acquire historical initial observation images and historical movement operation primitives of the robot when performing historical charging gun plugging and unplugging tasks. And the end of history image; The initial policy network is trained offline based on the historical initial observation image, the historical movement operation primitives, and the historical end image to obtain a pre-trained policy network; the historical movement operation primitives are a sequence of N constrained movement operation primitives. The robot acquires the current initial observation image when performing the current charging gun plugging / unplugging task, and predicts the current movement operation primitive based on the current initial observation image using the pre-trained policy network. ; The robot is controlled to execute the current charging gun plugging / unplugging task according to the current movement operation primitive, and to obtain the current end image corresponding to the current charging gun plugging / unplugging task; the current end image is an image taken again of the charging area after the current charging gun plugging / unplugging task is executed, which is used to determine whether the current charging gun plugging / unplugging task is executed successfully. Adjust the network parameters of the pre-trained policy network based on the current end image to obtain the target policy network; The total control amount corresponding to the new charging gun insertion / removal task is obtained according to the target policy network, and the robot is controlled to execute the new charging gun insertion / removal task according to the total control amount.
2. The charging gun plugging / unplugging task execution method according to claim 1, characterized in that, The step of adjusting the network parameters of the pre-trained policy network based on the current end image to obtain the target policy network includes: The current task execution status is obtained based on the currently completed image using a visual language model; The current reward value is determined based on the current task execution status. The network parameters of the pre-trained policy network are adjusted based on the current reward value to obtain the target policy network.
3. The charging gun plugging / unplugging task execution method according to claim 2, characterized in that, The step of determining the current reward value based on the current task execution status includes: If the current task execution status is successful, then the current reward value is the first value; If the current task execution status is failure, then the current reward value is the second value; If the current task execution status exceeds the limit, then the current reward value is the third value.
4. The charging gun plugging / unplugging task execution method according to claim 1, characterized in that, The step of offline training of the initial policy network based on the historical initial observation image, the historical movement operation primitive, and the historical end image to obtain a pre-trained policy network includes: Offline training data is obtained based on the historical initial observation image, the historical movement operation primitive, and the historical end image; The initial policy network is trained offline based on the offline training data to obtain the pre-trained policy network.
5. The charging gun plugging / unplugging task execution method according to claim 4, characterized in that, The step of obtaining offline training data based on the historical initial observation image, the historical movement operation primitive, and the historical end image includes: The execution status of historical tasks can be obtained based on the historical end image. The historical reward value is determined based on the historical task execution status. The offline training data is determined based on the historical movement primitives, the historical initial observation image, the historical end image, and the historical reward value.
6. The charging gun plugging / unplugging task execution method according to claim 1, characterized in that, The step of obtaining the total control quantity corresponding to the new charging gun plugging / unplugging task based on the target policy network includes: Obtain the new initial observation image corresponding to the new charging gun plugging / unplugging task; The target policy network is used to predict the new initial observation image to obtain a new movement operation primitive, and the control quantity is obtained based on the new movement operation primitive.
7. A charging gun plugging / unplugging task execution device, characterized in that, include: The first acquisition module is used to acquire historical initial observation images and historical movement operation primitives when the robot performs historical charging gun insertion and removal tasks. And the end of history image; The offline module is used to train the initial policy network offline based on the historical initial observation image, the historical movement operation primitives, and the historical end image to obtain a pre-trained policy network; the historical movement operation primitives are a sequence of N constrained movement operation primitives. The second acquisition module is used to acquire the current initial observation image when the robot performs the current charging gun plugging / unplugging task, and predict the current movement operation primitive based on the current initial observation image according to the pre-trained policy network. ; The online module is used to control the robot to execute the current charging gun plugging / unplugging task according to the current movement operation primitive, and to obtain the current end image corresponding to the current charging gun plugging / unplugging task; adjust the network parameters of the pre-trained policy network according to the current end image to obtain the target policy network; the current end image is an image taken again of the charging area after the current charging gun plugging / unplugging task is executed, and is used to determine whether the current charging gun plugging / unplugging task is executed successfully. The execution module is used to obtain the total control amount corresponding to the new charging gun insertion / removal task based on the target policy network, and control the robot to execute the new charging gun insertion / removal task based on the total control amount.
8. A robot, characterized in that, It includes a memory and a processor, the memory storing a computer program that, when executed by the processor, implements the charging gun plugging and unplugging task execution method as described in any one of claims 1 to 6.
9. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed by a processor, implements the charging gun plugging and unplugging task execution method as described in any one of claims 1 to 6.