Systems and Methods for Controlling a Robot, Electronic Devices, and Computer-Readable Media

Abstract

A system and method, an electronic device, and a computer-readable medium for controlling the robot are provided. The method includes: determining first position information of a joint of the robot based on feedback data received from the robot (510), the feedback data indicating movement of the joint, a tool being attached to the joint for holding a first object; determining second position information of the joint based on sensing data received from a sensor (520), the sensing data indicating relative movement between the joint and a second object to be aligned with the first object; and generating a prediction of a target position to align the first object with the second object based on the first position information and the second position information (530). In this way, high-accuracy assembly can be achieved with a shorter servo time, such that the assembly cost can be reduced and the assembly efficiency can be improved.

Description

Technical Field

[0001] Embodiments of the present disclosure generally relate to a system and method for controlling a robot, an electronic device, and a computer-readable medium. Background Art

[0002] High-accuracy assembly (0.02 mm accuracy) is an important process in various applications. The number and complexity of assembly tasks using industrial robots have been increasing year by year. Conventionally, to meet the high-accuracy requirements, the system needs multiple cycles of measurement and movement to achieve the accuracy, which may result in a significant overhead in cycle time.

[0003] For example, when the robotic arm holding the workpiece to be assembled moves one step, the sensor can capture the position of the workpiece and the deviation between the workpiece and its counterpart, and transmit the sensed data to the controller. The controller can also trigger the movement of the robot based on the sensed data. Thus, multiple cycles of movement sensing servo can be executed. In addition to time consumption, the debugging time and cost will also increase because accurate calibration is required to reduce the number of sensing and movement cycles. Summary of the Invention

[0004] Embodiments of the present disclosure provide a system and method for controlling a robot and a computer-readable medium.

[0005] In a first aspect, a method is provided. The method includes: determining first position information of a joint of a robot based on feedback data received from the robot, the feedback data indicating a movement of the joint, and a tool being attached to the joint for holding a first object; determining second position information of the joint based on sensed data received from a sensor, the sensed data indicating a relative movement between the joint and a second object to be aligned with the first object; and generating a prediction of a target position to align the first object with the second object based on the first position information and the second position information.

[0006] In this way, high-accuracy assembly can be achieved with a shorter servo time, so that the assembly cost can be reduced and the assembly efficiency can be improved.

[0007] In some embodiments, determining the first position information includes: receiving feedback data from the robot within a first time period; obtaining a set of coordinate parameters of the robot in a first coordinate system of the robot from the feedback data; and determining the first position information based on the set of coordinate parameters.

[0008] In some embodiments, determining the second position information includes: receiving sensed data from a sensor during a second time period that at least partially overlaps with a first time period for receiving feedback data; obtaining a set of position relationships between a first object and a second object in a second coordinate system of the sensor from the sensed data; and determining the second position information based on the first set of position relationships.

[0009] As described above, data from both the robot side and the sensor side will be collected by the system and transformed into corresponding position information, which is necessary for subsequent estimation of the target position of the robot. Specifically, the feedback and sensed data sets captured within a certain time period are transmitted to the data collector, rather than only transmitting the sensed data related to a single operation of the robot, which is beneficial for predicting the target position because the prediction result will be more accurate and the prediction process will also be accelerated.

[0010] In some embodiments, generating a prediction includes: obtaining a first set of sampling parameters from the first position information, the first set of sampling parameters characterizing the reference positions of the joints at a prediction time point; obtaining a second set of sampling parameters from the second position information, the second set of sampling parameters characterizing the reference position information between the joints of the robot and a second object at a predetermined time point; and generating a prediction of the target position by fusing the first set of sampling parameters and the second set of sampling parameters based on a predetermined fusion mode associated with an expected operation to be performed by the robot.

[0011] In some embodiments, the predetermined fusion mode includes at least one of the following: a predictor mode, a filter mode, a summation mode, and a subtraction mode.

[0012] During the data fusion process, the position information of the robot and the position information of the object at the same acquisition time point can be regarded as data for predicting the target position. By predicting the target position based on data from different data sources, high-quality prediction results can be obtained more effectively. At the same time, multiple fusion modes corresponding to a certain operator can be predetermined based on the expected operation process of the robot. In this way, any exceptional robot operation procedures can be implemented and developed more easily.

[0013] In some embodiments, the method further includes generating a command for controlling the robot based at least in part on the prediction.

[0014] In some embodiments, generating the command includes: determining a set of recording time points during a third time period for recording feedback data, wherein a start time point of a first time period for receiving the feedback data is offset from a start time point of the third time period by a predetermined time delay; determining, based on a prediction, a predicted time point at which a first object reaches a target position; determining, based on the set of recording time points, the feedback data, the predicted time point, and a prediction of the target position, a predicted trajectory for the first object to move to the target position; and generating a command based on the predicted trajectory. In this way, the prediction result can be further optimized to smooth the movement path of the robot to the target position.

[0015] In a second aspect, a system is provided. The system includes a data collector coupled to a robot and configured to determine first position information of joints of the robot based on feedback data received from the robot, the feedback data indicating movement of the joints, a tool being attached to the joints for holding a first object, a sensor coupled thereto and configured to determine second position information of the joints based on sensing data received from the sensor, the sensing data indicating relative movement between the joints and a second object to be aligned with the first object. The system further includes a first estimator coupled to the data collector and configured to generate a prediction of a target position for aligning the first object with the second object based on the first position information and the second position information.

[0016] In some embodiments, the data collector is further configured to: receive feedback data from the robot during a first time period; obtain a set of coordinate parameters of the robot in a first coordinate system of the robot from the feedback data; and determine the first position information based on the set of coordinate parameters.

[0017] In some embodiments, the data collector is further configured to: receive sensing data from the sensor during a second time period, the second time period at least partially overlapping with the first time period for receiving the feedback data; obtain a set of position relationships between the first object and the second object in a second coordinate system of the sensor from the sensing data; and determine the second position information based on the set of position relationships.

[0018] In some embodiments, the first estimator is further configured to: obtain a first set of sampling parameters from the first position information, the first set of sampling parameters characterizing a reference position of the joints at a predicted time point; obtain a second set of sampling parameters from the second position information, the second set of sampling parameters characterizing reference position information between the joints of the robot and the second object at a predetermined time point; and generate a prediction of the target position by fusing the first set of sampling parameters and the second set of sampling parameters based on a predetermined fusion mode, the predetermined fusion mode being associated with an expected operation to be performed by the robot.

[0019] In some embodiments, the predetermined fusion mode includes at least one of the following: a predictor mode, a filter mode, a summation mode, and a subtraction mode.

[0020] In some embodiments, the system further includes a command generator that is coupled to the first estimator and configured to generate commands for controlling the robot based at least in part on the prediction.

[0021] In some embodiments, the command generator is further configured to: determine a set of recording time points during a third time period for recording feedback data, wherein a start time point of the first time period for receiving the feedback data is offset from a start time point of the third time period by a predetermined time delay; determine, based on the prediction, a predicted time point at which the first object reaches the target position; determine, based on the set of recording time points, the feedback data, the predicted time point, and the prediction of the target position, a predicted trajectory of the first object moving to the target position; and generate commands based on the predicted trajectory.

[0022] In a third aspect, there is provided an electronic device. The electronic device includes a processor; and a memory coupled to the processor and storing instructions for execution, which, when executed by the processor, cause the device to perform the method of the first aspect.

[0023] In a fourth aspect, there is provided a computer-readable medium. The computer-readable medium includes program instructions for causing an electronic device to at least perform the method of the first aspect.

[0024] It is to be understood that the Summary of the Invention is not intended to identify key or essential features of the embodiments of the present disclosure, nor is it intended to be used to limit the scope of the present disclosure. Other features of the present disclosure will become readily appreciated through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The above and other objects, features, and advantages of the present disclosure will become more apparent from the following more detailed description of example embodiments of the present disclosure in conjunction with the accompanying drawings, in which like reference numerals generally represent like components in the example embodiments of the present disclosure.

[0026] Figure 1 An example operating environment in which embodiments of the present disclosure may be implemented is shown;

[0027] Figure 2 A schematic diagram of a system for controlling a robot according to an embodiment of the present disclosure is shown;

[0028] Figures 3A to 3C An example process of data fusion according to an embodiment of the present disclosure is shown;

[0029] Figures 4A to 4B Shows according to an embodiment of the present disclosure Figures 3A to 3C An example result of the data fusion shown;

[0030] Figure 5 shows a flowchart according to an embodiment of the present disclosure, illustrating a method for controlling a robot; and

[0031] Figure 6 Figure 6 shows a block diagram of an example computer-readable medium according to some example embodiments of the present disclosure.

[0032] In the drawings, the same or similar reference signs are used to indicate the same or similar elements. Detailed Description

[0033] The present disclosure will now be discussed with reference to multiple example embodiments. It is to be understood that these embodiments are discussed only to enable those skilled in the art to better understand and thus implement the present disclosure, and not to imply any limitation on the scope of the subject matter.

[0034] As used herein, the term "comprising" and variations thereof should be read as open terms, which mean "including but not limited to". The term "based on" should be read as "at least partially based on". The terms "an embodiment" and "embodiments" should be read as "at least one embodiment". The term "another embodiment" should be read as "at least one other embodiment". The terms "first", "second", etc. may refer to different or the same objects. Other definitions (explicit and implicit) may be included below. Unless the context clearly indicates otherwise, the definitions of the terms are consistent throughout the description.

[0035] Unless otherwise specified or limited, the terms "mounted", "connected", "supported", and "coupled" and variations thereof are used broadly and cover direct and indirect mounting, connection, support, and coupling. In addition, "connected" and "coupled" are not limited to physical or mechanical connection or coupling. In the following description, the same reference numerals and labels are used to describe the same, similar, or corresponding parts in the drawings. Other definitions (explicit and implicit) may be included below.

[0036] Figure 1 Figure 1 shows an example operating environment 100 in which embodiments of the present disclosure may be implemented. As Figure 1 shown, the operating environment 100 may include a robot 110. The robot 110 may be fixed to the ground. The robot 110 may include a manipulator arm having joints 111, such as joints at the end of the manipulator arm. A tool 112 may be attached to the joint 111 to hold an object to be assembled. For example, the tool 112 may be formed as a pin, a clamp, or a glue gun. As Figure 1 shown, the tool 112 is formed as a clamp for gripping a workpiece. Hereinafter, the workpiece gripped by the tool 112 may be referred to as a first object 131.

[0037] On the assembly table 133, there is another workpiece, which may be hereinafter referred to as the second object 132. In the assembly process, the second object 132 can be considered as the counterpart of the first object 131. To assemble the first object and the second object together, an alignment process of the first object and the second object may be required. In this document, alignment may refer to a complete match or a partial match. For example, they may be placed on top of each other or adjacent to each other without overlapping. The position where the first object 131 is aligned with the second object 132 may be referred to as the target position. In some cases, the second object 132 may be stationary. However, for example, due to the magnetic force generated by a magnetic field, the second object 132 may also be movable.

[0038] Figure 1 The illustrated operating environment 100 may also include a plurality of sensors 121 and 122 and a controller 150 coupled to these sensors to collect sensed data from these sensors. The controller 150 may communicate with the sensors 121 and 122 via a wired or wireless communication module (not shown). For example, the sensor 121 (which may also be hereinafter referred to as the first sensor 121) may be disposed on the joint 111 of the robot 110. The sensor 122 (which may also be hereinafter referred to as the second sensor 122) may be disposed in the environment. Both the first sensor 121 and the second sensor 122 may observe the robot 110, the first object 131, and the second object 132. It should be understood that Figure 1 the number of sensors shown is given for purposes of illustration and does not imply any limitation. The operating environment 100 may include any suitable number of sensors.

[0039] As described above, high-accuracy assembly (0.02 mm accuracy) is an important process in various applications. The number and complexity of assembly tasks using industrial robots increase year by year. Conventionally, to meet the requirement of high accuracy, the system needs multiple cycles of measurement and movement to achieve the accuracy, which may result in a significant overhead in cycle time.

[0040] For example, when the robotic arm holding the workpiece to be assembled moves one step, the sensor can capture the position of the workpiece and the deviation of the workpiece from its counterpart, and transmit the sensed data to the controller. The controller can also trigger the movement of the robot based on the sensed data. Therefore, multiple cycles of movement sensing can be performed. In addition to time consumption, the debugging time and cost are also increased because accurate calibration is required to reduce the number of sensing and movement cycles.

[0041] Therefore, the present disclosure proposes a solution for high-accuracy assembly supported by a robot. By utilizing the feedback data and sensed data collected in the alignment process, the number of sensing and movement cycles can be reduced, and high-accuracy assembly can be achieved simultaneously.

[0042] The principles and embodiments of the present disclosure will be described below with reference to Figures 2 to 5 more detail. Figure 2 A schematic diagram of a system for controlling a robot according to an embodiment of the present disclosure is shown. For the purpose of discussion, system 200 will be described with reference to Figure 1 described. It should be understood that although system 200 has been described in the Figure 1 operating environment 100, the system 200 can equally be applied to other operating environments.

[0043] As Figure 2 shown, system 200 can include a data collector 210. The data collector 210 can be coupled to the robot controller 113. When the operating arm of the robot 110 starts to move, the robot controller 113 can record the position of the joint 111 during the movement. The recorded data (which may hereinafter be referred to as feedback data) can be transmitted from the robot controller 113 to the data collector 210.

[0044] The data collector 210 can determine first position information of the joint 111 of the robot 110 based on the received feedback data. The feedback data can indicate the movement of the joint 111 to which the tool 112 is attached for holding the first object 131.

[0045] In some embodiments, the feedback data can refer to a set of coordinate parameters of the joint 111 of the robot 110 in the coordinate system of the robot 110 recorded by the robot controller 113 during a first time period during the robot movement. For example, the parameters of the robot tool center point (TCP) can be considered as feedback data.

[0046] For an industrial robot, one or more tools can be installed on the robot to manipulate an object. To describe the tool in space, a coordinate system on the tool (i.e., the tool coordinate system (TCS)) can be defined, and the origin of the tool coordinate system can be considered as the TCP. In the TCS, usually 6 degrees of freedom or 6 pieces of information are required to completely define the pose of the end joint of the robot, because it can move along three directions in space and can also rotate around three directions. In some embodiments, the first data collector 111 can then determine the first position information of the robot based on the set of coordinate parameters.

[0047] Further, the data collector 210 may be coupled to the sensing group 120 including the first sensor 121 and the second sensor 122, and receive sensing data from at least one of them. The data collector 210 may determine the second position information of the joint 111 based on the sensing data received from any one of the first sensor 121 and the second sensor 122. For example, the sensing data of the second sensor 122 may be transmitted to the data collector 210. The sensing data may indicate the relative movement between the joint 111 and the second object 132 to be aligned with the first object 131.

[0048] For example, the sensing data may refer to a set of images captured by the sensor within a second time period. The set of images may reflect the set of position relationships between the first object 131 and the second object 132 in the coordinate system of the second sensor 122. The second time period may overlap with the first time period during which the feedback data is received by the data collector 210.

[0049] As Figure 2 shown, the system 200 may further include a first estimator 220. The first estimator 220 may be coupled to the data collector 210 and configured to generate a prediction of the target position based on the first position information and the second position information for aligning the first object 131 with the second object 132.

[0050] In some embodiments, the first position information and the second position information may be fused in the first estimator 220. For example, the first estimator 220 may obtain a first set of sampling parameters from the first position information and a second set of sampling parameters from the second position information. The first set of sampling parameters may characterize the reference position of the joint 111 at a predetermined time point, and the second set of sampling parameters may characterize the reference position relationship between the joint 111 of the robot 110 and the second object 132 at the same time point.

[0051] In some embodiments, the first estimator 220 may further generate a prediction of the target position by fusing the first set of sampling parameters and the second set of sampling parameters based on a fusion mode that may be associated with the expected operation to be performed by the robot.

[0052] Referring to Figure 3A and Figure 3B , the data fusion of the first set of sampling parameters from the first position information and the second set of sampling parameters from the second position information may be described in further detail.

[0053] Figure 3A An example of a curve 301 of a sampling signal associated with the second position information is shown, and Figure 3B an example of a curve 302 of a sampling signal associated with the first position information is shown. As Figure 3AAs shown, the second sampling parameter set may refer to the parameters at time points T3 and T8. That is, for example, the parameter set sampled at time point T3 of curve 301 may reflect the positional relationship between joint 111 of robot 110 and the second object 132 captured by the second sensor 122 at time point T3. To fuse the first parameter set and the second parameter set, the parameter sampled at T3 of curve 302 may be required. However, the parameter sampled at T3 of curve 302 may not be directly readable from curve 302. That is, the first sampling parameter set may not be directly obtainable from the first position information. Instead, the sampling parameter at time point T3 of curve 302 may be calculated based on the parameters sampled at other sampling time points (e.g., based on the parameters sampled at T2 and T4 of curve 302). The sampling parameter at time point T3 of curve 302 may reflect the position of the joint at time point T3. Then, the sampling parameter at time point T3 of curve 301 may be fused with the sampling parameter at time point T3 of curve 302 to generate a prediction of the target position of the robot. A similar procedure may be performed for data fusion of the parameters sampled at time point T8 of curves 301 and 302.

[0054] In some embodiments, the first estimator 220 further includes an operator pool to provide operators corresponding to fusion modes such as Kalman mode, predictor mode, filter mode, observer mode, summation mode, or subtraction mode. These fusion modes may be pre-configured for specific tasks assigned by the sensor model.

[0055] During the data fusion process, different position parameters determined by sensing data from different sensor groups at the same acquisition time point may be regarded as position information for predicting the target position. Since the sensing data provides different position parameters at multiple viewpoints, high-quality prediction results can be obtained more effectively. Meanwhile, multiple fusion modes corresponding to a certain operator may be predetermined based on the desired operation process of the robot. In this way, any exceptional robot operation procedures can be more easily implemented and developed.

[0056] Returning to Figure 2 , the system 200 may further include a command generator 240 coupled to the first estimator 220. The command generator 240 may be configured to generate commands for controlling the robot 110 based on the prediction of the target position. The command generator 240 may be coupled to the robot controller 113 to transmit the commands to the robot controller 113.

[0057] To further smooth the movement path of the first object 131 and ensure the cycle time for alignment, a predicted trajectory for guiding the first object from the current position to the target position can be determined. In some embodiments, the command generator 240 may determine a set of recording time points during a third time period for recording feedback data. The start time point of the first time period for receiving feedback data may deviate from the start time point of the third time period by a predetermined time delay.

[0058] The recording time points of the feedback data can be determined in the data collector 210. In addition, the data collector 210 may also record the reception time point of each feedback data and compensate for the predetermined time delay to the reception time point of the feedback data to obtain the recording time point of the feedback data. Moreover, the data collector 210 may also record the reception time point of each sensing data and compensate for the predetermined time delay to the reception time point of the sensing data to obtain the sensing time point of the sensing data. For example, if the time stamp T_record represents the time when the system 200 receives the data, the sampling time point T_record (at which the data is exactly sampled) of the data can be expressed as T_receive minus the time delay T_delay of the sensor.

[0059] Furthermore, the command generator 240 may determine the predicted time point for the first object to reach the target position based on the prediction of the target position. Based on the set of recording time points, the feedback data, the predicted time point, and the prediction of the target position, the predicted trajectory can be determined. The command generator 240 may generate a command based on the predicted trajectory. Through the predicted trajectory, the first object 131 can move to the target position.

[0060] In some embodiments, more than one sensor may transmit sensing data to the data collector 210, so multiple predictions of the target position may be generated at the first estimator 220. In this case, the multiple predictions may be considered preliminary predictions and may be further fused to generate a final prediction.

[0061] Alternatively, the system 200 may further include a second estimator 230. As shown in FIG. 3, another sensor in the sensor group (e.g., the first sensor 121) may transmit sensing data to the data collector 210. The data collector 210 may determine the third position information of the joint 111 of the robot 110 based on the sensing data received from the first sensor 121. The sensing data may be obtained by sensing the movement of the joint 111 of the robot 110. As an option, the information of the joint 111 of the robot 110 may also be obtained from the robot controller 113.

[0062] As described above, the first position information and the second position information obtained from the data collector 210 can be fused to generate a prediction of the target position of the robot 110. Similarly, the first estimator 220 can further generate a further prediction of the target position based on the first position information and the third position information, for example, by fusing the first position information and the third position information.

[0063] Further referring to Figure 3B and Figure 3C , the data fusion procedure for the first position information and the third position information can be similar to the data fusion procedure for the first position information and the second position information.

[0064] Figure 3C An example of the curve 303 of the sampling signal associated with the third position information is shown, and Figure 3B An example of the curve 302 of the sampling signal associated with the first position information is shown. As Figure 3C shown, the third set of sampling parameters can refer to the parameters at time points T4 and T10. That is, for example, the set of parameters sampled at time point T4 of the curve 303 can reflect the positional relationship between the second object 132 captured by the first sensor 121 at time point T4 and the joint 111 of the robot 110. To fuse the first set of parameters and the third set of parameters, the parameters sampled at T4 of the curve 302 may be required. However, the parameters sampled at T4 of the curve 302 may not be directly readable from the curve 302. That is, the first set of sampling parameters may not be directly obtained from the first position information. Instead, the sampling parameters at time point T4 of the curve 302 can be calculated based on the parameters sampled at other sampling time points (e.g., based on the parameters sampled at T2 and T5 of the curve 302). The sampling parameters at time point T4 of the curve 302 can reflect the position of the joint 111 at time point T4. Then, the sampling parameters at time point T4 of the curve 303 can be fused with the sampling parameters at time point T4 of the curve 302 to generate a further prediction of the target position of the robot 110. A similar procedure can be performed for the data fusion of the parameters sampled at time point T10 of the curves 303 and 302.

[0065] Figures 4A to 4B An example result of the data fusion according to an embodiment of the present disclosure is shown. Figures 3A to 3C The example result of the data fusion shown. Figure 4A The result of the data fusion of the first position information and the second position information is shown, and Figure 4B The result of the data fusion of the first position information and the third position information is shown. For example, Figure 4AThe value of the curve 401 shown represents the data fusion result of the parameters of the first position information and the second position information at the time point T3. Based on the curve 401 obtained by fusing the first position information and the second position information and the curve 402 obtained by fusing the first position information and the third position information, two preliminary predictions of the target position can be generated, that is, the value P1 in the curve 401 and the value P2 of the curve 402.

[0066] Referring back to FIG. 3, the system 200 may further include a second estimator 230. The second estimator 230 may be coupled to the first estimator 220 and generate a final prediction of the target position by fusing the prediction of the target position and the further prediction of the target position. For example, as Figure 4A and 4B shown, the preliminary prediction of the target position P1 and the preliminary prediction of the target position P2 may be fused, and the fused value may be regarded as the final prediction of the target position. In this case, the command generator 240 may generate a command for controlling the robot 110 based on this final prediction of the target position.

[0067] The data fusion for generating the final prediction may be regarded as a second fusion program. The preliminary prediction of the target position / orientation may be sent to the buffer together with the time stamp in the second fusion module in the second estimator. In most applications, the moving speed of the object is relatively low compared with the movement of the robot. Therefore, the regression of the target position and orientation can provide a more reliable prediction than the regression of the sensor data.

[0068] It is to be understood that Figure 2 the system 200 shown can be implemented with any hardware and software. For example, the system 200 can be implemented as Figure 1 the controller 150 shown. The system 200 can also be implemented as an integrated chip. The components of the system 200 can be regarded as entities capable of performing certain functions, such as data collectors, estimators, instruction generators, etc. The components in the system 200 can also be regarded as virtual modules capable of implementing certain functions.

[0069] In this way, high-accuracy assembly can be achieved with a shorter servo time, so that the assembly cost can be reduced and the assembly efficiency can be improved.

[0070] Figure 5 FIG. shows a flowchart according to an embodiment of the present disclosure, illustrating a method for controlling a robot. For the purpose of discussion, the method 500 will be described from Figure 2 the perspective of the system 200.

[0071] At 510, system 200 determines first position information of joint 111 of robot 110 based on feedback data received from robot 110. The feedback data may indicate movement of joint 111 to which tool 112 is attached for holding first object 131.

[0072] At 520, system 200 determines second position information of joint 111 based on sensing data received from sensor 122. The sensing data may indicate relative movement between joint 111 and second object 132 to be aligned with first object 131.

[0073] In some embodiments, determining the first position information includes: receiving feedback data from robot 110 during a first time period; obtaining a set of coordinate parameters of robot 110 in a first coordinate system of robot 110 from the feedback data; and determining the first position information based on the set of coordinate parameters.

[0074] In some embodiments, determining the second position information includes: receiving sensing data from sensor 122 during a second time period, the second time period at least partially overlapping with the first time period for receiving feedback data; obtaining a set of position relationships between first object 131 and second object 132 in a second coordinate system of sensor 122 from the sensing data; and determining the second position information based on the set of position relationships.

[0075] At 530, system 200 generates a prediction of a target position to align first object 131 with second object 132 based on the first position information and the second position information.

[0076] In some embodiments, generating the prediction includes: obtaining a first set of sampling parameters from the first position information, the first set of sampling parameters characterizing a reference position of the joint at a prediction time point; obtaining a second set of sampling parameters from the second position information, the second set of sampling parameters characterizing reference position information between a joint of the robot and the second object at a predetermined time point; and generating a prediction of the target position by fusing the first set of sampling parameters and the second set of sampling parameters based on a predetermined fusion mode, the predetermined fusion mode being associated with an expected operation to be performed by the robot.

[0077] In some embodiments, the predetermined fusion mode includes at least one of the following: a predictor mode, a filter mode, a summation mode, and a subtraction mode.

[0078] In some embodiments, system 200 may also generate a command for controlling the robot at least partially based on the prediction.

[0079] In some embodiments, generating a command includes: determining a set of recording time points during a third time period for recording feedback data, where a start time point of a first time period for receiving feedback data is offset from a start time point of the third time period by a predetermined time delay; determining, based on a prediction, a predicted time point at which a first object reaches a target position; determining, based on the set of recording time points, the feedback data, the predicted time point, and a prediction of the target position, a predicted trajectory for the first object to move to the target position; and generating a command based on the predicted trajectory.

[0080] Figure 6 is a simplified block diagram of a device 600 suitable for implementing embodiments of the present disclosure. The device 600 may be provided to implement Figure 6 the illustrated system 200. As Figure 6 illustrated, the device 600 may include a computer processor 610 coupled to a computer-readable memory unit 620, and the memory unit 620 includes instructions 622. When executed by the computer processor 610, the instructions 622 may implement a method for controlling a robot as described in the previous paragraph, and details will be omitted hereinafter.

[0081] In some embodiments of the present disclosure, a computer-readable medium for simulating at least one object in a manufacturing line is provided. The computer-readable medium has instructions stored thereon, and when executed on at least one processor, the instructions may cause the at least one processor to execute a method for controlling a robot as described in the previous paragraph, and details will be described hereinafter.

[0082] In general, various embodiments of the present disclosure may be implemented in hardware or in a dedicated circuit, software, logic, or any combination thereof. Some aspects may be implemented in hardware, while other aspects may be implemented in firmware or software that can be executed by a controller, a microprocessor, or other computing device. Although the various aspects of the embodiments of the present disclosure are illustrated and described as block diagrams, flowcharts, or using some other graphical representation, it should be understood that the blocks, devices, systems, techniques, or methods described herein may be implemented as non-limiting examples in hardware, software, firmware, a dedicated circuit or logic, general hardware or a controller, or other computing device, or some combination thereof.

[0083] The present disclosure also provides at least one computer program product tangibly stored on a non-transitory computer-readable storage medium. The computer program product includes computer-executable instructions, such as those included in program modules, which are executed in a device on a target real or virtual processor to perform the operations described above with reference to Figure 5The described processes or methods. Generally, a program module includes routines, programs, libraries, objects, classes, components, data structures, etc. that perform a particular task or implement a particular abstract data type. In various embodiments, the functionality of program modules can be combined or split among program modules as needed. The machine-executable instructions of program modules can be executed within local or distributed devices. In a distributed device, program modules can be located in local and remote storage media.

[0084] The program code for performing the methods of the present disclosure can be written in any combination of one or more programming languages. This program code can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device such that, when executed by the processor or controller, the program code causes the functions / operations specified in the flowchart and / or block diagram to be implemented. The program code can be executed entirely on the machine, partially on the machine, executed as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0085] The above program code can be implemented on a machine-readable medium, which can be any tangible medium that can contain or store a program used by or in connection with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of a machine-readable storage medium will include an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0086] Further, although operations are depicted in a particular order, this should not be construed as requiring that such operations be performed in the particular order shown or in a sequential order, or that all illustrated operations be performed, to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. Similarly, although a number of specific implementation details are included in the above discussion, these should not be construed as limitations on the scope of the present disclosure, but rather as descriptions of features that may be specific to particular embodiments. Certain features described in the context of separate embodiments can also be implemented in combination in a single embodiment. On the other hand, the various features described in the context of a single embodiment can also be implemented separately or in any suitable sub-combination in multiple embodiments.

[0087] Although the subject matter has been described in language specific to structural features and / or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. On the contrary, the above specific features and acts are disclosed as example forms of implementing the claims.

Claims

1. A method for controlling a robot, comprising: Determining first position information of a joint of the robot based on feedback data received from the robot, the feedback data indicating movement of the joint, and a tool being attached to the joint for holding a first object; Determining second position information of the joint based on sensing data received from a sensor, the sensing data indicating relative movement between the joint and a second object to be aligned with the first object; Generating a prediction of a target position based on the first position information and the second position information to align the first object with the second object, wherein the target position is the position at which the first object is aligned with the second object; And Generating a command for controlling the robot at least in part based on the prediction.

2. The method according to claim 1, wherein determining the first position information comprises: Receiving the feedback data from the robot within a first time period; Obtaining a set of coordinate parameters of the robot in a first coordinate system of the robot from the feedback data; And Determining the first position information based on the set of coordinate parameters.

3. The method according to claim 1, wherein determining the second position information comprises: Receiving the sensing data from the sensor within a second time period, the second time period at least partially overlapping with the first time period for receiving the feedback data; Obtaining a set of position relationships between the first object and the second object in a second coordinate system of the sensor from the sensing data; And Determining the second position information based on the first set of position relationships.

4. The method according to claim 1, wherein generating the prediction comprises: Obtaining a first set of sampling parameters characterizing a reference position of the joint at a predetermined time point from the first position information; Obtaining a second set of sampling parameters from the second position information, the second set of sampling parameters characterizing a reference position relationship between the joint of the robot and the second object at the predetermined time point; And Generating the prediction of the target position by fusing the first set of sampling parameters and the second set of sampling parameters based on a predetermined fusion mode, the predetermined fusion mode being associated with an expected operation to be performed by the robot.

5. The method according to claim 4, wherein the predetermined fusion mode comprises at least one of the following: Predictor mode, Filter mode, Summation mode, and Subtraction mode.

6. The method according to claim 1, wherein generating the command comprises: Determining a set of recording time points within a third time period for recording the feedback data, a start time point of the first time period for receiving the feedback data being offset from a start time point of the third time period by a predetermined time delay; Determining a predicted time point at which the first object reaches the target position based on the prediction; Determining a predicted trajectory through which the first object moves to the target position based on the set of recording time points, the feedback data, the predicted time point, and the prediction of the target position; And Generating the command based on the predicted trajectory.

7. A system (200) for controlling a robot (110), comprising: A data collector (210) coupled to the robot (110) and configured to determine first position information of joints (111) of the robot (110) based on feedback data received from the robot (110), the feedback data indicating movement of the joints (111), a tool (112) being attached to the joints for holding a first object (131), the data collector being coupled to a sensor (122) and configured to determine second position information of the joints (111) based on sensing data received from the sensor (122), the sensing data indicating relative movement between the joints (111) and a second object (132) to be aligned with the first object (131); A first estimator (220) coupled to the data collector (210) and configured to generate a prediction of a target position for aligning the first object (131) with the second object (132) based on the first position information and the second position information; And A command generator (240) coupled to the first estimator (220) and configured to generate a command for controlling the robot (110) at least partially based on the prediction.

8. The system according to claim 7, wherein the data collector (210) is further configured to: Receive the feedback data from the robot (110) within a first time period; Obtain a set of coordinate parameters of the robot (110) in a first coordinate system of the robot (110) from the feedback data; and Determine the first position information based on the set of coordinate parameters.

9. The system according to claim 7, wherein the data collector (210) is further configured to: Receive the sensing data from the sensor (122) within a second time period, the second time period at least partially overlapping with the first time period for receiving the feedback data; Obtain a set of position relationships between the first object (131) and the second object (132) in a second coordinate system of the sensor (122) from the sensing data; And Determine the second position information based on the first set of position relationships.

10. The system according to claim 7, wherein the first estimator (220) is further configured to: Obtain a first set of sampling parameters characterizing a reference position of the joints (111) at a predetermined time point from the first position information; Obtain a second set of sampling parameters from the second position information, the second set of sampling parameters characterizing a reference position relationship between the joints (111) of the robot (110) and the second object (132) at the predetermined time point; and Generate the prediction of the target position by fusing the first set of sampling parameters and the second set of sampling parameters based on a predetermined fusion mode, the predetermined fusion mode being associated with an expected operation to be performed by the robot (110).

11. The system according to claim 10, wherein the predetermined fusion mode includes at least one of the following: Predictor mode, Filter mode, Summation mode, and Subtraction mode.

12. The system according to claim 7, wherein the command generator (240) is further configured to: Determine a set of recording time points during a third time period for recording the feedback data, the start time point of a first time period for receiving the feedback data being offset from the start time point of the third time period by a predetermined time delay; Based on the prediction, determine a predicted time point at which the first object (131) reaches the target position; Based on the set of recording time points, the feedback data, the predicted time point, and the prediction of the target position, determine a predicted trajectory through which the first object (131) moves to the target position; And Generate the command based on the predicted trajectory.

13. An electronic device, comprising: A processor; And A memory, coupled to the processor and storing instructions for execution, which, when executed by the processor, cause the device to perform the method according to any one of claims 1 to 6.

14. A computer-readable medium comprising program instructions for causing an electronic device to at least perform the method according to any one of claims 1 to 6.

Images