Method, apparatus, device, robot, and medium for controlling a robot

Abstract

Embodiments of the present disclosure relate to methods, devices, equipment, robots, and media for controlling robots. The method for controlling a robot according to an embodiment of the present disclosure includes determining reference motion parameters and reference control forces corresponding to a real-time image captured by the robot at a first moment based on the real-time image. The method also includes determining a target position and target control force of the robot at a second moment after the first moment based on the reference motion parameters and the reference control force. The method also includes determining a target action of the robot at a second moment based on the target position and the target control force. In this way, the robot can imitate and learn kinematic actions as well as the actual contact force and torque generated by its end with the environment during movement, thereby improving the accuracy and robustness of robot control.

Description

Technical Field

[0001] Embodiments of the present disclosure relate generally to the field of automation, and more particularly to methods, apparatuses, devices, robots, and media for controlling robots. Background Art

[0002] In recent years, academia and industry have become increasingly interested in the research of robot operational capabilities, and the form of robots has gradually evolved from fixed robotic arms and mobile unmanned vehicles to more complex mobile operational configurations.

[0003] Mobile manipulation combines two fundamental robotic capabilities: mobility and object manipulation. These capabilities significantly expand the real-world applications of robots compared to static manipulation. For example, mobile manipulation enables robots to complete tasks involving manipulation of large workspaces. Summary of the Invention

[0004] An embodiment of the present disclosure provides a technical solution for controlling a robot.

[0005] In a first aspect of the present disclosure, a method for controlling a robot is provided. The method includes determining, based on a real-time image captured by the robot at a first moment, reference motion parameters and reference control forces corresponding to the real-time image. The method also includes determining, based on the reference motion parameters and the reference control forces, a target position and target control forces of the robot at a second moment subsequent to the first moment. The method also includes determining, based on the target position and target control forces, a target motion of the robot at a second moment.

[0006] According to a second aspect of the present disclosure, a device for controlling a robot is provided. The device includes a retrieval module configured to determine, based on a real-time image captured by the robot at a first moment, reference motion parameters and a reference control force corresponding to the real-time image. The device also includes a prediction module configured to determine, based on the reference motion parameters and the reference control force, a target position and target control force of the robot at a second moment subsequent to the first moment. The device also includes a determination module configured to determine, based on the target position and target control force, a target action of the robot at the second moment.

[0007] According to a third aspect of the present disclosure, an electronic device is provided, comprising a processor and a memory coupled to the processor, wherein the memory has instructions stored therein, and when the instructions are executed by the processor, the electronic device executes the method according to the first aspect of the present disclosure.

[0008] According to a fourth aspect of the present disclosure, a robot is provided, comprising a robotic arm, a chassis, and the electronic device according to the third aspect of the present disclosure.

[0009] According to a fifth aspect of the present disclosure, a computer-readable storage medium is provided, wherein computer-executable instructions are stored on the computer-readable storage medium, wherein the computer-executable instructions are executed by a processor to implement the method according to the first aspect of the present disclosure.

[0010] Please note that the invention summary is provided to introduce a series of concepts in a simplified form, which will be further described in the detailed description below. The invention summary is not intended to identify key features or essential features of the present disclosure, nor is it intended to limit the scope of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The above and other objects, features and advantages of the present disclosure will become more apparent through a more detailed description of the embodiments of the present disclosure in conjunction with the accompanying drawings, in which:

[0012] Figure 1 is a schematic diagram illustrating an example environment in which methods and / or processes according to embodiments of the present disclosure may be implemented;

[0013] Figure 2 is a flowchart illustrating a method for controlling a robot according to an embodiment of the present disclosure;

[0014] Figure 3 is a diagram illustrating a process 300 of building a reference data database according to an embodiment of the present disclosure;

[0015] Figure 4 is a diagram illustrating a robot control process according to an embodiment of the present disclosure.

[0016] Figure 5 illustrates an example of a target operation performed using robot control according to an embodiment of the present disclosure;

[0017] Figure 6 An example process of performing a target operation using robot control according to an embodiment of the present disclosure is illustrated;

[0018] Figure 7 is a block diagram illustrating an apparatus for controlling a robot according to an embodiment of the present disclosure; and

[0019] Figure 8 is a schematic block diagram illustrating an example device that can be used to implement embodiments according to the present disclosure.

[0020] Throughout the drawings, same or similar reference numbers generally refer to same or similar elements. DETAILED DESCRIPTION

[0021] It should be understood that before using the technical solutions disclosed in the various embodiments of the present disclosure, the type, scope of use, usage scenarios, etc. of the personal information (for example, captured images) involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0022] It should be understood that the data involved in the technical solutions of the present disclosure (including but not limited to the data itself, the acquisition or use of the data) should comply with the requirements of relevant laws, regulations and relevant provisions.

[0023] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are illustrated in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should also be understood that the drawings and embodiments of the present disclosure are for illustrative purposes only and are not intended to limit the scope of protection of the present disclosure.

[0024] In the description of the embodiments of the present disclosure, the term "including" and its variations should be understood as open inclusion, that is, "including but not limited to". The term "based on" should be understood as "based at least in part on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The terms "first", "second", etc. can refer to different or the same objects, unless explicitly indicated to be different.

[0025] As mentioned above, more and more researchers are focusing on the manipulation capabilities of robots, and the types of robots are becoming increasingly diverse. Robotic mobility and object manipulation are among their most important characteristics, enabling them to perform many complex manipulation tasks, including contact manipulation.

[0026] In fact, robots' touch-handling capabilities have already brought about significant changes. For example, in the manufacturing industry, automated production and assembly using robots has become the norm. Using robots to perform touch-handling tasks not only improves production efficiency and product quality, but also reduces the risks and workload of manual operations. Consequently, an increasing number of companies are incorporating robotic technology to improve their production processes, with significant results. Beyond manufacturing, robots' touch-handling capabilities are also playing a vital role in various fields, such as healthcare and logistics.

[0027] However, while robots bring convenience, they also bring challenges. Various uncertainties in the robot's performance of contact manipulation tasks (such as jitter caused by improper operation) bring safety issues to the robot. In addition, due to the high-dimensional spatial configuration of mobile manipulation robots, it is difficult to generate control signals to complete the control of the robot. These problems cannot be well solved by conventional control methods. This is because conventional control methods either do not consider the force information of the robot during movement, or for any scenario, they need to generate trajectories by understanding the geometric constraints of the scene, rather than imitation learning.

[0028] To address at least some of the aforementioned and other potential issues, embodiments of the present disclosure provide a technical solution for controlling a robot. This solution includes determining, based on a real-time image captured by the robot at a first moment, reference motion parameters and reference control forces corresponding to the real-time image. The solution also includes determining, based on the reference motion parameters and reference control forces, a target position and target control force for the robot at a second moment subsequent to the first moment. The solution also includes determining, based on the target position and target control force, a target action for the robot at a second moment.

[0029] The technical solution for controlling a robot according to an embodiment of the present disclosure is intended to enable the robot not only to imitate and learn kinematic movements, but also to imitate the actual contact forces and torques generated by its end with the environment during movement, thereby improving the accuracy and robustness of robot control and ensuring a reasonable computational cost.

[0030] Refer to the following Figures 1 to 8 To illustrate the basic principles and several example implementations of the present disclosure. It should be understood that these exemplary embodiments are provided only to enable those skilled in the art to better understand and implement the embodiments of the present disclosure, and are not intended to limit the scope of the present disclosure in any way. In addition, in the following description, certain embodiments will be discussed with reference to the robot control process of a robot opening a cabinet drawer. However, it should be understood that this is only for the purpose of enabling those skilled in the art to better understand the principles and concepts of the embodiments of the present disclosure, and is not intended to limit the scope of the present disclosure in any way.

[0031] Figure 1 1 is a diagram illustrating an example environment 100 in which methods and / or processes according to embodiments of the present disclosure may be implemented. Figure 1 As shown in FIG, the example environment 100 includes a database 110 and a robot 120 coupled to each other (eg, via a wire or a network), and a target location 130. It should be understood that Figure 1 The arrangement shown in is merely exemplary, and embodiments of the present disclosure may also include other different arrangements.

[0032] According to an embodiment of the present disclosure, the database 110 may store prior data for performing contact operations of the robot 120. The prior data for performing contact operations of the robot 120 may include multiple trajectories for multiple tasks, and each trajectory may include an observation image, motion parameters, control force, and termination identification at each time step (also referred to as a moment) during the execution of the contact operation of the robot 120. By way of example, these trajectories are obtained, for example, in the process of performing a certain contact operation (for example, opening a cabinet drawer) by the robotic arm 121 of the expert handheld robot 120. Below, the detailed process of obtaining the trajectory will be further described. It should be understood that the method of obtaining the trajectory is not limited to the above method, and the trajectory can also be obtained in other different ways.

[0033] The observation images stored in the database 110 are also referred to as reference images, and can be, for example, red, green, and blue (RGB) images captured at each time step using the camera 123 on the robotic arm 121 of the robot 120, and can describe the surrounding environment of the robot at that time step. In other words, one observation image can be captured at each time step using the robot 120, and multiple observation images captured in this way can be stored in the database. The motion parameters can be the kinematic behavior of the robot 120 in the SE(3) space, indicating the translation and rotation of the robot 120 in the SE(3) space. The control force can be a six-dimensional force applied to the robot 120, which is a thrust or pull in the x, y, and z directions and a torque around the x, y, and z axes. In the example, the control force can be a six-dimensional force applied to the joints on the robotic arm 121 of the robot 120. In the following, the motion parameters and control forces stored in the database 110 are also referred to as reference motion parameters and reference control forces, respectively. The reference motion parameters and reference control forces may be sensed and recorded by one or more sensors (such as, but not limited to, motion sensors and force sensors) disposed on the robot 120. The termination marker, also referred to as a reference termination marker, may indicate the termination of the contact operation of the robot 120.

[0034] According to an embodiment of the present disclosure, at each time step during a contact operation of the robot 120, corresponding reference motion parameters and reference control forces can be recorded while capturing a reference image, and a corresponding reference termination identifier can be determined. In other words, the reference images stored in the database 110 have corresponding reference motion parameters, reference control forces, and reference termination identifiers.

[0035] It should be understood that the methods of capturing reference images, the number and type of reference images, and the contents described by the reference images described herein are merely exemplary, and that other different capturing methods, numbers and types of images, and contents of descriptions may exist. In addition, the sensors that record the reference motion parameters and reference control forces at each time step during the contact operation of the robot 120 are not limited to being disposed on the robot 120, but may also be sensors external to the robot 120. In addition, as Figure 1 As shown in , the database 110 can be arranged separately from the robot 120. However, this is merely exemplary, and the database 110 can be arranged inside the robot 120, which is not limited in the present disclosure. Moreover, the present disclosure does not limit the type of the database 110, and the database 110 can be, for example, a local database or a cloud database.

[0036] It should be understood that the reference image is only one form of representation of the reference data (i.e., visual representation), and other different representations are possible. For example, the reference data can be represented in the form of a reference vector (converting the reference image into a vector or a vector set), or can be represented in the form of reference audio, reference video, etc. In addition, Figure 1 The database 110 shown in FIG. 1 is merely an exemplary representation of prior knowledge for executing contact operations of the robot 120, and embodiments of the present disclosure are not limited to such exemplary representations. For example, prior knowledge for executing contact operations of the robot 120 can be obtained directly from an expert in real time (e.g., via a network or line), or reference data can be obtained by simulating contact operations of other robots, etc. The present disclosure is not limited in this regard.

[0037] According to an embodiment of the present disclosure, Figure 1 The robot 120 exemplarily shown in the figure may include a robotic arm 121 and a chassis 122, and the robotic arm 121 may include a camera 123. By way of example, the robotic arm 121 may be, for example, a six-degree-of-freedom robotic arm, the chassis 122 may be, for example, a differential chassis, and the camera 123 may be an RGB camera. It should be understood that for ease of illustration, only these three components are described here, and the robot 120 may also include more or fewer components, for example, one or more sensors. Based on data from the database 110, the robot 120 may move to a target position 130 to perform a target operation. Hereinafter, the robot control process according to an embodiment of the present disclosure will be described in further detail.

[0038] like Figure 1As exemplarily shown in FIG, the target location 130 may include a cabinet, and the target operation in this example may be opening a drawer of the cabinet. It should be understood that the present disclosure is not limited to the target operation in this example, and that target operations that can be performed using the robot control process according to an embodiment of the present disclosure may also include, for example, turning on a faucet, turning on a washing machine, etc. In order to accurately and robustly perform these target operations as described above, a method for controlling the robot 120 according to an embodiment of the present disclosure is proposed, and such a robot control process will be further described below.

[0039] Combined with the above Figure 1 A schematic diagram of an example environment 100 is depicted in which methods and / or processes according to embodiments of the present disclosure may be implemented. Figure 2 1 and 2 are flowcharts of a method 200 for controlling a robot according to an embodiment of the present disclosure. As described above with respect to the database 110 and the robot 120, the method 200 for controlling a robot according to an embodiment of the present disclosure can be executed entirely within the robot 120 or in a distributed manner with the aid of external components, and the present disclosure does not impose any limitations on this.

[0040] Figure 2 is a flowchart illustrating a method 200 for controlling a robot according to an embodiment of the present disclosure. To improve the accuracy and robustness of robot control while ensuring reasonable computational costs, the method 200 for controlling a robot according to an embodiment of the present disclosure is proposed. The reference data retrieval and target data prediction processes according to the embodiments of the present disclosure are described in further detail below. It should be understood that the method 200 and other methods of the present disclosure can be executed by the robot's own control module or by an external control device.

[0041] At 210 , based on the real-time image captured by the robot 120 at the first moment, reference motion parameters and reference control forces corresponding to the real-time image are determined. In the robot control process according to embodiments of the present disclosure, the robot 120 (for example, but not limited to, using the camera 123 on the robotic arm 121) may capture a real-time image at each time step. This real-time image may, for example, depict the robot 120's surroundings at that time step. Next, reference data corresponding to the real-time image captured at that time step may be determined. The process of determining the corresponding reference data will be described in further detail below. As described above, each reference image in the database 110 has corresponding reference motion parameters and reference control forces. In one example, after determining the reference image corresponding to the real-time image captured at that time step, the reference motion parameters and reference control forces corresponding to the reference image may be determined. In another example, the real-time image captured at that time step may be sent to an expert, who may then return corresponding reference motion parameters and reference control forces based on the real-time image. It should be understood that the present disclosure does not limit the number and type of real-time images captured by the robot at each time step, nor does it limit the content depicted by the real-time image.

[0042] At 220, based on the reference motion parameters and the reference control force, a target position and target control force for robot 120 at a second time point after the first time point are determined. According to an embodiment of the present disclosure, after determining a reference image corresponding to the real-time image captured at the current time step at 210, according to an embodiment of the present disclosure, a reference control force corresponding to the reference image can be determined as the target control force for robot 120 at the next time step, and the target position of the robot at the next time step can be determined based on the reference motion parameters corresponding to the reference image. The target data prediction process according to an embodiment of the present disclosure will be described in further detail below.

[0043] At 230, a target action for the robot 120 at the second time step is determined based on the target posture and target control force. According to an embodiment of the present disclosure, the target action to be performed by the robot 120 at the next time step can be determined based on the target posture and target control force for the robot 120 at the next time step determined at 220. In this way, based on the target action determined at each time step, the robot 120 can execute the determined target action time-by-time until the target operation (e.g., opening a cabinet drawer) is completed.

[0044] The method for controlling a robot according to an embodiment of the present disclosure enables the robot 120 to not only imitate and learn kinematic movements, but also imitate and learn the actual contact forces and torques generated by the end of the robot 120 (for example, the robotic arm 121 of the robot 120) and the environment during movement, thereby improving the accuracy and robustness of the robot control.

[0045] Figure 3 FIG3 is a diagram illustrating a process 300 of building a reference data database according to an embodiment of the present disclosure. It should be understood that the description of building a reference data database or collecting reference data is merely exemplary, and there may be other different building or collecting methods.

[0046] According to an embodiment of the present disclosure, a process in which the robot arm 121 of the expert handheld robot 120 performs a certain contact operation (e.g., opening a drawer of a cabinet) is shown at 310. At each time step in the process, the image sensor can capture an observation image describing the surrounding environment of the robot 110 at that time. e As a reference image, the motion sensor and force sensor can record the motion parameters a of the robot 110 at this time. e and control force F e As reference motion parameters and reference control forces. In this way, multiple reference images and reference motion parameters and reference control forces corresponding to each reference image can be obtained. Alternatively or additionally, after capturing each observation image o e At the same time, a reference termination marker T corresponding to the e By way of example, the image sensor may be, for example, a camera on the robotic arm 121 of the robot 120. It should be understood that the image sensor, motion sensor, and force sensor described herein may be disposed on the robot 120 or may be a sensor external to the robot 120, and that these sensors are merely exemplary, and that other different sensors may also be utilized to perform the aforementioned reference data collection process.

[0047] After the reference data at 310 is collected, multiple tracks may be formed as exemplarily shown at 320. According to an embodiment of the present disclosure, each of the multiple tracks for multiple tasks (such as, but not limited to, opening a cabinet drawer, turning on a faucet, and turning on a washing machine, etc.) may include a reference image. e And the corresponding reference motion parameter a e , reference control force F e and reference end marker T e It should be understood that the multiple traces shown at 320 are exemplary, and each trace may include more or less information, and the embodiments of the present disclosure are not limited thereto.

[0048] The multiple trajectories exemplarily shown at 320 may be stored in the database 110 to form a reference data database. According to an embodiment of the present disclosure, the robot 120 may determine the real-time image taken in real time from the reference data database. t The corresponding reference image oe The following will further describe the determination of the corresponding reference image o e By utilizing the prior knowledge in the reference data database for executing the contact operation of the robot 120, the robot 120 can imitate and learn the kinematic motions and the actual contact forces and torques generated by its end with the environment during the movement, thereby improving the accuracy and robustness of the robot control.

[0049] Figure 4 is a diagram illustrating a robot control process 400 according to an embodiment of the present disclosure. It should be understood that Figure 4 The various sub-processes shown in are merely exemplary, and the robot control process 400 according to an embodiment of the present disclosure may include more or fewer sub-processes.

[0050] According to an embodiment of the present disclosure, the real-time image captured by the robot 120 at the first moment can be t 410 Encoded as real-time image o t Real-time visual representation of 410 t .like Figure 4 As shown in FIG, the real-time image captured by the robot 120 at the current time step is o t 410 can be encoded into a real-time visual representation via encoding unit 420' t By integrating real-time visual representation of t and a plurality of reference visual representations Z from the database 110 e Each reference visual representation z in e Compare and get real-time visual representation t Visual representation Z with multiple references e Each reference visual representation z in e The similarity between the reference images stored in the database 110 e can be encoded as a reference visual representation z via encoding unit 420 e , where z e ∈Z e . Real-time visual representation t and the reference visual representation z e Can be real-time image o t 410 and reference image o e The encoding unit 420 and the encoding unit 420' can be pre-trained visual encoders. It should be understood that the encoding unit 420 and the encoding unit 420' can be the same encoder or different encoders, and the embodiment of the present disclosure does not limit the type of encoder. The following formula (1) shows the real-time visual representation z t With the reference visual representation z e Calculation of similarity between:

[0051]

[0052] According to an embodiment of the present disclosure, the cosine distance can be used to calculate the real-time visual representation z t With the reference visual representation z e The similarity between them.

[0053] According to an embodiment of the present disclosure, based on real-time visual representation z t Visual representation Z with multiple references e Each reference visual representation z in e similarity between them, and determining a reference visual representation z that meets a predetermined similarity threshold. e In the example, for example, you can choose to have a real-time visual representation of z t The reference visual representation z with the highest similarity e , and the reference visual representation z e Reference image before encoding o e Determined as the real-time image o captured by the robot 120 at the current time step t 410. It should be noted that the embodiments of the present disclosure are not limited to the above selection and determination process. For example, a reference image corresponding to the real-time visual representation z t The similarity of the second reference visual representation z e The following equation (2) shows the relationship between the real-time visual representation z t The reference visual representation z with the highest similarity e Determination of:

[0054]

[0055] According to an embodiment of the present disclosure, a reference visual representation z that satisfies a predetermined similarity threshold is determined. e The corresponding reference motion parameter a e and the reference control force F e As mentioned above, each reference image o in the database 110 e With the corresponding reference motion parameter a e and the reference control force F e Therefore, after determining the reference visual representation z that satisfies the predetermined similarity threshold, e Afterwards, we can determine the difference between the reference visual representation z e Reference image before encoding o e The corresponding reference motion parameter a e and the reference control force F e , and the retrieval unit 430 of the robot 120 retrieves the reference motion parameter a e and the reference control force Fe Based on the reference motion parameter a e and the reference control force F e , the target position of robot 120 in the next time step can be determined and the target control force The target data prediction process according to an embodiment of the present disclosure will be described in further detail below.

[0056] According to an embodiment of the present disclosure, the determined reference motion parameter a of the robot 120 is e and the reference control force F e Determined as the real-time motion parameter a of the robot 120 at the first moment t and target control at the second moment The determined reference motion parameter a of the robot 120 e can be determined as the real-time motion parameter a of the robot 120 at the current time step t , and the determined reference control force F of the robot 120 e can be determined as the target control force of the robot 120 at the next time step

[0057] According to an embodiment of the present disclosure, based on the real-time motion parameter a of the robot 120 at the first moment t and real-time pose Determine the target position of the robot at the second moment Based on the real-time motion parameter a of the robot 120 at the current time step t and real-time pose The target position of robot 120 in the next time step can be obtained Where ○ is the group action in SE(3) space. It should be understood that the real-time pose of the robot 120 at the current time step is The sensing may be performed by its own sensor or by an external sensor, and the embodiments of the present disclosure are not limited thereto.

[0058] Due to the uncertainty caused by positioning and the lack of accuracy caused by motion prediction, the above target poses May be inaccurate. In order to make the target pose As accurately as possible, the embodiment of the present disclosure can use the posture adjustment plan to adjust the target posture The target posture adjustment process according to the embodiment of the present disclosure will be described in further detail below.

[0059] According to an embodiment of the present disclosure, the real-time control force F of the robot 120 at the first moment is obtained. tThe real-time control force F of the robot 120 at the current time step can be monitored by one or more sensors such as force sensors. t It should be understood that monitoring the real-time control force F t The one or more sensors may be sensors of the robot itself or sensors arranged outside the robot. Based on the real-time control force F of the robot 120 at the current time step t And the target control force at the next time step Can generate target pose The adjustment unit 440 of the robot 120 uses the generated posture adjustment plan to adjust the target posture Adjust the target posture. Generation of pose adjustment plan.

[0060] According to an embodiment of the present disclosure, based on the real-time control force F of the robot at the first moment t and the target control force at the second moment The difference between them generates the target pose The following equation (3) shows the attitude adjustment plan for the admittance term ΔP t+1 Calculation:

[0061]

[0062] where K p and K d are two gain matrices corresponding to the stiffness and damping of the admittance controller, respectively. In other words, a virtual spring-damper system is established between the end of the robot 120 and the target object (such as a cabinet) at the target position 130. The admittance term ΔP calculated above is t+1 It can be the predicted target pose The adjustment amount.

[0063] According to an embodiment of the present disclosure, the posture adjustment plan can be used to adjust the target posture of the robot 120 at the second moment. Adjust the predicted target pose according to the instructions The posture adjustment plan of the adjustment amount can be calculated using the admittance term ΔP t+1 The predicted target pose Make adjustments.

[0064] According to an embodiment of the present disclosure, based on the adjusted target posture of the robot 120 at the second moment Generate control commands for the robot 120. After utilizing the admittance compensation according to an embodiment of the present disclosure, the command unit of the robot 120 is based on the adjusted target pose at the next time step. To generate control commands u for whole body control t The generated control command u t The robot 120 can be controlled by the robot arm 121 and the chassis 122. The following equation (4) expresses the whole body control of the robot 120 as a trajectory optimization problem:

[0065]

[0066] Where x is the state variable of the system, including the joint states of the joints on the robotic arm 121 of the robot 120 and the status x of chassis 122 b =[x,y,γ]∈SE(2). The control vector u includes the velocity control for the robot arm 121 and the chassis 122. t is the initial state. f(·) is the transfer function, which is given by the system kinematic equation. The following equation (5) shows the defined loss function:

[0067] L(x,u)=L base +L ee +u T Ru (5)

[0068] Among them L ee and L base are the tracking errors of the robot arm 121 and the chassis 122 respectively.

[0069]

[0070] According to an embodiment of the present disclosure, the image corresponding to the reference image is determined from the database 100. e The corresponding reference end mark T e As described above, in determining the real-time image captured by the robot 120 at the current time step o t The corresponding reference image o e Afterwards, it can be determined that the e The corresponding reference motion parameter a e , reference control force F e and reference end marker T e The reference ends with T e It can be determined as the target termination representation T of the robot 120 at the next time step t , which can indicate whether to terminate the control of the robot 120 in the next time step, where T t ∈{0,1}.

[0071] According to an embodiment of the present disclosure, in response to the target termination identifier T t Instructs to terminate control, stops controlling the robot 120, and responds to the termination mark Tt Indicates that the control is not terminated and the robot 120 is controlled to perform the target action at the next time step. t =1, it can be terminated, otherwise it can be based on the reference motion parameter a of the robot 120 at the current time step e and the reference control force F at the next time step e , determine the target pose of robot 120 in the next time step and target control Then, the target action of the robot 120 in the next time step is determined.

[0072] According to an embodiment of the present disclosure, it is possible to determine whether the number of times the robot 120 is controlled has reached a predetermined number threshold. In response to the number reaching the predetermined number threshold, the control of the robot 120 can be stopped, and in response to the number not reaching the predetermined number threshold, it can be determined that the robot 120 will terminate at the target termination representation T at the next time step. t Whether to indicate termination control.

[0073] The following is an exemplary diagram of a robot control algorithm according to an embodiment of the present disclosure:

[0074]

[0075] Wherein Visual Encoder() is an image (such as a real-time image o t or reference image o e ) for visual encoding. getCurrentState() is to get the current state, that is, the real-time image o t , real-time control force F t and real-time pose Retrive() is, for example, retrieving data from a database, Admittance() is admittance, and WholeBodyControl() is whole body control.

[0076] Figure 5 An example of a target operation performed by robot control according to an embodiment of the present disclosure is illustrated. Figure 5 As shown in (1), the robot 120 is in the process of opening a cabinet drawer. Figure 5 As shown in (2), the robot 120 has opened the top drawer of the cabinet. Figure 5 (3) and (4) show that the robot 120 is performing the opening of the left door and the right door of the cabinet. In addition, Figure 5 (5) and (6) of FIG. 1 respectively show that the robot 120 is turning on the faucet and turning on the washing machine. It should be noted that the target operations that can be performed by the robot control according to the embodiment of the present disclosure are not limited to Figure 5The examples shown in , and may also include other different target operations.

[0077] Figure 6 FIGURE 1 illustrates an example process of performing a target operation using a robot control according to an embodiment of the present disclosure. Figure 6 As shown in (1), the robot 120 slowly approaches the washing machine at the target position 130 and opens the washing machine. Figure 6 The order of the six images shown in (1) from left to right corresponds to the order of the time steps in which the robot 120 performs the operation of opening the washing machine. In other words, Figure 6 (1) shows that the robot 120 performs the target operation of opening the washing machine step by step.

[0078] Similarly, Figure 6 (2), (3) and (4) respectively show the robot 120 performing the target operations of opening the left door, drawer and right door of the cabinet step by step. Figure 6 (5) and (6) respectively show the robot 120 performing the target operations of turning on the faucet and opening the top drawer of the cabinet step by step. It should be noted that Figure 6 The number of time frames shown in the various example processes is only exemplary.

[0079] like Figure 6 As shown in the various examples in FIG. 1 , the fine control of the differential chassis and mechanical joints according to the embodiments of the present disclosure enables the robot 120 to have higher flexibility and adaptability to complete many complex contact tasks.

[0080] Figure 6 Each embodiment process in the present invention has shown its practicality and convenience. Such robot control enables

[0081] Figure 7 is a block diagram illustrating an apparatus 700 for controlling a robot according to an embodiment of the present disclosure. Figure 7 As shown in , the device 700 includes a retrieval module, which is configured to determine the reference motion parameters and reference control force corresponding to the real-time image captured by the robot at the first moment. The device 700 also includes a prediction module, which determines the target posture and target control force of the robot at a second moment after the first moment based on the reference motion parameters and the reference control force. The device 700 also includes a prediction module and a determination module, which determines the target action of the robot at the second moment based on the target posture and the target control force. The device 800 may also include other modules to implement the method and / or process according to the embodiments of the present disclosure, which will not be repeated here for the sake of brevity.

[0082] It should be understood that the apparatus 800 of the present disclosure can achieve at least some of the advantages achievable by the methods or processes described above. For example, it can simulate and learn kinematic motions, and can also simulate and learn the actual contact forces and torques generated by the end-point of the robot 120 and the environment during motion, thereby improving the accuracy and robustness of robot control while ensuring reasonable computational costs.

[0083] Figure 8 FIG1 shows a block diagram of an electronic device 800 according to some embodiments of the present disclosure. The device 800 may be a device or apparatus described in an embodiment of the present disclosure. Figure 8 As shown, the device 800 includes a central processing unit (CPU) and / or a graphics processing unit (GPU) 801, which can perform various appropriate actions and processes according to computer program instructions stored in a read-only memory (ROM) 802 or computer program instructions loaded from a storage unit 808 into a random access memory (RAM) 803. Various programs and data required for the operation of the device 800 can also be stored in the RAM 803. The CPU / GPU 801, ROM 802, and RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804. Although not shown in FIG. Figure 8 As shown in FIG, device 800 may further include a co-processor.

[0084] Various components in device 800 are connected to I / O interface 805, including an input unit 806, such as a keyboard, mouse, etc.; an output unit 807, such as various types of displays, speakers, etc.; a storage unit 808, such as a magnetic disk, optical disk, etc.; and a communication unit 809, such as a network card, modem, wireless communication transceiver, etc. The communication unit 809 allows device 800 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0085] The various methods or processes described above may be performed by the CPU / GPU 801. For example, in some embodiments, the methods may be implemented as a computer software program tangibly embodied in a machine-readable medium, such as the storage unit 808. In some embodiments, part or all of the computer program may be loaded and / or installed onto the device 800 via the ROM 802 and / or the communication unit 809. When the computer program is loaded into the RAM 803 and executed by the CPU / GPU 801, one or more steps or actions in the methods or processes described above may be performed.

[0086] In some embodiments, the methods and processes described above may be implemented as a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for executing various aspects of the present disclosure.

[0087] Computer-readable storage medium can be a tangible device that can keep and store the instructions used by the instruction execution device.Computer-readable storage medium can be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device or any suitable combination thereof.More specific examples (non-exhaustive list) of computer-readable storage medium include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, for example, a punch card or a convex structure in a groove having instructions stored thereon, and any suitable combination thereof.Computer-readable storage medium used herein is not interpreted as a transient signal itself, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagated by waveguides or other transmission media (for example, light pulses by fiber optic cables), or electrical signals transmitted by wires.

[0088] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions to be stored in the computer-readable storage medium in each computing / processing device.

[0089] The computer program instructions for performing the disclosed operation can be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data or source code or the object code written in any combination of one or more programming languages, programming languages include object-oriented programming languages, and conventional procedural programming languages.Computer-readable program instructions can be performed completely on a user's computer, partially on a user's computer, performed as an independent software package, partly on a user's computer and partly on a remote computer, or performed completely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer by any type of network-including local area network (LAN) or wide area network (WAN), or can be connected to an external computer (such as utilizing an internet service provider to connect by the internet). In certain embodiments, by utilizing the state information of computer-readable program instructions to carry out personalized customization electronic circuits, such as programmable logic circuits, field programmable gate arrays (FPGAs) or programmable logic arrays (PLA), this electronic circuit can perform computer-readable program instructions, thereby realizing various aspects of the present disclosure.

[0090] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processing unit of the computer or other programmable data processing device, a device is generated that implements the functions / actions specified in one or more blocks in the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium, where these instructions cause the computer, programmable data processing device, and / or other device to operate in a specific manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing various aspects of the functions / actions specified in one or more blocks in the flowchart and / or block diagram.

[0091] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device, so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0092] The flow charts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices, methods and computer program products according to multiple embodiments of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and the part of the module, program segment or instruction contains one or more executable instructions for realizing the prescribed logical function. In some alternative implementations, the functions marked in the box can also occur in a sequence different from that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart, can be implemented by a special hardware-based system that performs the prescribed function or action, or can be implemented by a combination of special hardware and computer instructions.

[0093] While various embodiments of the present disclosure have been described above, the above descriptions are intended to be illustrative and non-exhaustive, and are not intended to limit the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is selected to best explain the principles of the various embodiments, their practical applications, or technical improvements to existing technologies, or to enable others skilled in the art to understand the various embodiments disclosed herein.

Claims

1. A method for controlling a robot, comprising: determining, based on a real-time image captured by the robot at a first moment, a reference motion parameter and a reference control force corresponding to the real-time image; Determining a target position and a target control force of the robot at a second moment after the first moment based on the reference motion parameters and the reference control force; obtaining a real-time control force of the robot at the first moment from a force sensor of the robot; generating a posture adjustment plan for the target posture based on a difference between the real-time control force and the target control force of the robot at the second moment, the posture adjustment plan indicating an adjustment amount for the target posture of the robot at the second moment; The adjustment unit of the robot adjusts the target posture using the posture adjustment plan; as well as Based on the adjusted target posture and the target control force, a target action of the robot at the second moment is determined.

2. The method according to claim 1, determining the reference motion parameter and the reference control force corresponding to the real-time image comprises: determining a reference image corresponding to the real-time image from a database; as well as The reference motion parameter and the reference control force of the robot corresponding to the reference image are obtained.

3. The method according to claim 2, further comprising: encoding the real-time image captured by the robot at the first moment into a real-time visual representation of the real-time image; obtaining a similarity between the real-time visual representation and each of the plurality of reference visual representations by comparing the real-time visual representation with each of the plurality of reference visual representations from the database; as well as Based on the similarity between the real-time visual representation and each of the plurality of reference visual representations, a reference visual representation that satisfies a predetermined similarity threshold is determined.

4. The method according to claim 3, wherein: The plurality of reference visual representations from the database are obtained by encoding a plurality of reference images captured using the robot, and Each reference image of the plurality of reference images corresponds to a reference motion parameter and a reference control force of the robot.

5. The method according to claim 3, wherein determining the target posture and the target control force of the robot at the second moment after the first moment comprises: determining the reference motion parameters and the reference control forces of the robot corresponding to the reference visual representation that satisfies the predetermined similarity threshold; as well as Based on the reference motion parameters and the reference control force, the target posture and the target control force of the robot at the second moment are determined.

6. The method according to claim 5, wherein determining the target posture and the target control force of the robot at the second moment after the first moment further comprises: Determining the determined reference motion parameters and reference control force of the robot as the real-time motion parameters of the robot at the first moment and the target control force at the second moment; as well as Based on the real-time motion parameters and real-time posture of the robot at the first moment, the target posture of the robot at the second moment is determined.

7. The method according to claim 1, wherein: The adjustment amount is associated with the stiffness and damping of a virtual spring-damper system between the robot and a target position of a target operation to be performed.

8. The method according to claim 1, further comprising: generating a control command for the robot based on the adjusted target pose of the robot at the second moment; as well as Based on the control command, the robot is controlled to perform the target action at the second moment.

9. The method according to claim 8, wherein controlling the robot comprises: Based on the control command, at least one of a robotic arm and a chassis of the robot is controlled.

10. The method according to claim 2, further comprising: determining a reference end mark of the robot corresponding to the reference image from the database; Determining the reference termination mark as the target termination mark of the robot at the second moment; In response to the target termination indicator indicating termination of control, stopping control of the robot; as well as In response to the target termination flag indicating that control is not to be terminated, the robot is controlled to perform the target action at the second moment.

11. The method according to claim 10, further comprising: determining whether the number of times the robot is controlled has reached a predetermined number threshold; In response to the number of times reaching the predetermined number threshold, stopping the control of the robot; as well as In response to the number of times not reaching the predetermined number threshold, it is determined whether the target termination flag indicates termination of control.

12. A device for controlling a robot, comprising: a retrieval module configured to determine, based on the real-time image captured by the robot at a first moment, a reference motion parameter and a reference control force corresponding to the real-time image; a prediction module configured to determine a target posture and a target control force of the robot at a second moment after the first moment based on the reference motion parameters and the reference control force; a force acquisition module, configured to obtain the real-time control force of the robot at the first moment from the force sensor of the robot; a plan generation module configured to generate a posture adjustment plan for the target posture based on a difference between the real-time control force and the target control force of the robot at the second moment, the posture adjustment plan indicating an adjustment amount for the target posture of the robot at the second moment; an adjustment module, configured to adjust the target posture using the posture adjustment plan through an adjustment unit of the robot; as well as A determination module is configured to determine a target action of the robot at the second moment based on the adjusted target posture and the target control force.

13. An electronic device comprising: processor; as well as A memory coupled to the processor, the memory having instructions stored therein, wherein when the instructions are executed by the processor, the electronic device performs the method according to any one of claims 1 to 11.

14. A robot comprising: robotic arm; chassis; as well as The electronic device according to claim 13. 15 . A computer-readable storage medium having computer-executable instructions stored thereon, wherein the computer-executable instructions are executed by a processor to implement the method according to claim 1 .

Images