Systems and methods for operating a robot, electronic devices, and computer-readable media
By collecting and fusion data in the robot system, the problems of sensor signals out of synchronization and time delay are solved, the accuracy and efficiency of robot operations are improved, the implementation of multi-sensor applications is simplified, and automation performance is improved.
Patent Information
- Application Number
- CN202080092977.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-01-22
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2040-01-22
AI Technical Summary
In sensor-based robotic applications, sensor signal out-synchronization and time delay problems lead to reduced stability and limited servo task performance.
By introducing a coupled data collector and estimator into the robot system, receiving robot feedback data and sensor sensing data, performing timestamp calibration and data fusion, generating more accurate target position predictions, thereby generating commands for operating the robot.
Improves the accuracy and efficiency of robot operation, provides a more flexible servo solution, simplifies the implementation of multi-sensor applications, reduces development time, and improves performance in the field of automation.
Smart Images

Figure CN114945447B_ABST
Abstract
Description
Technical Field
[0001] Embodiments of the present disclosure generally relate to systems and methods, electronic devices, and computer-readable media for operating a robot. Background Art
[0002] Sensor-based robotic applications play an important role in agile and flexible automation. Multiple sensors can be set to monitor the robot's posture and detect data related to the robot's position or the robot's movement. By using sensor-based robotic applications, debugging time can be reduced and production efficiency can be improved. At the same time, robot skills and product varieties can be deeply expanded.
[0003] Typically, in conditional use scenarios for sensor-based robotic applications, the robot's operation tasks can be divided into several processes, and each process uses only one sensor to regulate the robot's operation. For example, vision can be used to guide the robot to a position first, and then force controls can be applied to assemble parts.
[0004] In other conditional use scenarios, multiple sensors may be used simultaneously, which may require combining the outputs of different individual sensors. For example, the control signal is determined based on a weighted sum of the outputs of a force sensor and a visual sensor.
[0005] The above usage scenarios suffer from sensor signal desynchronization and sensor signal time delay issues, which may impair stability and limit the performance of servo tasks. Summary of the invention
[0006] Embodiments of the present disclosure provide a system and method for operating a robot and a computer-readable medium.
[0007] In a first aspect, a system for operating a robot is provided. The system includes a first data collector coupled to the robot and configured to determine first position information of the robot based on feedback data received from the robot, and a second data collector coupled to a sensor and configured to determine second position information of an object to be operated by the robot based on sensed data received from the sensor. The system also includes a first estimator coupled to the first data collector and the second data collector and configured to generate a first prediction of a target position of the object to be operated by the robot based on the first position information and the second position information, and a command generator coupled to the estimator and configured to generate a command for operating the robot based at least in part on the first prediction.
[0008] By means of the system proposed in the first aspect, robot operations can be performed more accurately and efficiently. For the user-side operator, the sensor-based servo can provide a more flexible solution. The system proposed in the first aspect can be used with any conventional robot that the user has already deployed. In this way, the user does not have to bear the cost caused by the modification of a large number of components. For manufacturers, the system can simplify the process of implementing multi-sensor based solutions and can reduce development time. At the same time, the capabilities of the automation field can be expanded and the performance of sensor-based applications can be improved.
[0009] In some embodiments, the first data collector is also configured to receive feedback data from the robot within a first time period; obtain a set of coordinate parameters of the joint end of the robot in the coordinate system of the robot from the feedback data of the robot; and determine the first position information based on the set of coordinate parameters.
[0010] In some embodiments, the second data collector is also configured to receive sensing data from the sensor within a second time period, and the second time period overlaps with the first time period for receiving feedback data from the robot; obtain a set of position relationships between the robot and the object in another coordinate system of the sensor from the sensing data of the sensor; and determine second position information based on the set of position relationships.
[0011] In the system of the first aspect, data from both the robot side and the sensor side are collected by the first stage of the system and transformed into corresponding position information, which is necessary for the subsequent estimation of the target position of the robot. In particular, a set of feedback / sensing data captured within a certain period of time can be transmitted to the data collector, rather than only transmitting sensing data related to a single operation of the robot, which is beneficial to the prediction of the target position because the predicted result will be more accurate and the prediction process will be accelerated.
[0012] In some embodiments, the first estimator is further configured to calibrate the first set of timestamps for the first location information and the second set of timestamps for the second information based on a predetermined time delay.
[0013] Usually, there is a time delay between the time point when the data collector receives the data and the sampling time point of the robot's feedback data or the sensing time point of the sensor's sensing data, so timestamp calibration is necessary before the prediction process. In this way, the accuracy of the prediction will be significantly improved.
[0014] In some embodiments, the first estimator is also configured to obtain a first set of sampling parameters characterizing a reference position of the robot at a predetermined point in time from the first position information; obtain a second set of sampling parameters characterizing a reference position relationship between the robot and the object at a predetermined point in time from the second position information; and generate a first prediction of the target position by fusing the first set of sampling parameters and the second set of sampling parameters based on a fusion pattern associated with an expected operation to be performed by the robot.
[0015] In some embodiments, the fusion mode includes at least one of a prediction mode, a filtering mode, a summation mode, and a subtraction mode.
[0016] In the data fusion process, the position information of the robot and the position information of the object at the same acquisition time point can be considered as data for predicting the target position. By predicting the target position based on data from different data sources, high-quality prediction results can be obtained more efficiently. At the same time, multiple fusion modes corresponding to specific operators can be pre-determined based on the expected operation process of the robot. In this way, any exceptional robot operation process can be more easily implemented and developed.
[0017] In some embodiments, the second data collector is also coupled to an additional sensor and is configured to determine third position information of the object based on additional sensing data received from the additional sensor, and the first estimator is further configured to generate a second prediction of the target position based on the first position information and the third position information.
[0018] In some embodiments, the system also includes a second estimator coupled to the first estimator and configured to generate a third prediction of the target location by fusing the first prediction of the target location and the second prediction of the target location.
[0019] The position information of the object determined based on the data received from more than one sensor can be fused with the position information of the robot, and a set of preliminary predictions of the target position can be obtained. Subsequently, a final prediction of the target position can be generated by fusing the set of preliminary predictions. In this way, the robot can be operated more accurately based on the two-stage prediction.
[0020] In some embodiments, the command generator is further configured to generate commands for operating the robot based on the third prediction.
[0021] In a second aspect, a method for operating a robot is provided. The method includes determining first position information of the robot based on feedback data received from the robot; determining second position information of an object to be operated by the robot based on sensory data received from a sensor; generating a first prediction of a target position at which the object is operated by the robot based on the first position information and the second position information; and generating a command for operating the robot based at least in part on the first prediction.
[0022] In some embodiments, determining the first position information includes receiving feedback data from the robot within a first time period; obtaining a set of coordinate parameters of the joint end of the robot in the coordinate system of the robot from the feedback data of the robot, and determining the first position information based on the set of coordinate parameters.
[0023] In some embodiments, determining the second position information includes receiving sensing data from the sensor within a second time period, the second time period overlapping with a first time period for receiving feedback data from the robot; acquiring a set of position relationships between the robot and the object in another coordinate system of the sensor from the sensing data of the sensor; and determining the second position information based on the set of position relationships.
[0024] In some embodiments, the method further comprises calibrating the first set of time stamps for the first location information and the second set of time stamps for the second information based on a predetermined time delay.
[0025] In some embodiments, generating a first prediction includes obtaining a first set of sampling parameters representing a reference position of the robot at a predetermined point in time from the first position information; obtaining a second set of sampling parameters representing a reference position relationship between the robot and the object at the predetermined point in time from the second position information; and generating a first prediction of the target position by fusing the first set of sampling parameters and the second set of sampling parameters based on a predetermined fusion pattern associated with an expected operation to be performed by the robot.
[0026] In some embodiments, the fusion mode includes at least one of a prediction mode, a filtering mode, a summation mode, and a subtraction mode.
[0027] In some embodiments, the method further comprises determining third position information of the object based on further sensory data received from further sensors, and generating a second prediction of the target position based on the first position information and the third position information.
[0028] In some embodiments, the method further includes generating a third prediction of the target position by fusing the first prediction of the target position and the second prediction of the target position.
[0029] In some embodiments, the method further includes generating commands for operating the robot based on the third prediction.
[0030] In a third aspect, an electronic device is provided. The electronic device comprises a processor; and a memory coupled to the processor and storing instructions for execution, wherein when the instructions are executed by the processor, the device executes the method of the second aspect.
[0031] In a fourth aspect, a computer-readable medium is provided, wherein the computer-readable medium includes program instructions for causing an electronic device to at least execute the method of the second aspect.
[0032] It should be understood that the invention summary is not intended to determine the key or essential features of the embodiments of the present disclosure, nor is it intended to be used to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS
[0033] The above and other objects, features and advantages of the present disclosure will become more apparent through a more detailed description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, wherein like reference numerals generally denote like components throughout the exemplary embodiments of the present disclosure.
[0034] Figure 1 An example operating environment is shown in which embodiments of the present disclosure may be implemented;
[0035] Figure 2 A schematic diagram of a system for operating a robot according to an embodiment of the present disclosure is shown;
[0036] Figure 3 A schematic diagram of a system for operating a robot according to an embodiment of the present disclosure is shown;
[0037] Figure 4A-4C An example process of data fusion according to an embodiment of the present disclosure is shown;
[0038] Figure 5A-Figure 5B The embodiment according to the present disclosure is shown Figure 4A-4C Example results of data fusion shown;
[0039] Figure 6 A flowchart showing a method for operating a robot according to an embodiment of the present disclosure; and
[0040] Figure 7 A block diagram of an example computer-readable medium is shown according to some example embodiments of the present disclosure.
[0041] Throughout the drawings, the same or similar reference numbers are used to refer to the same or similar elements. DETAILED DESCRIPTION
[0042] The present disclosure will now be discussed with reference to several example embodiments. It should be understood that these embodiments are discussed only to enable those skilled in the art to better understand and thus implement the present disclosure, rather than implying any limitation on the scope of the subject matter.
[0043] As used herein, the term "including" and its variations should be understood as open terms meaning "including but not limited to". The term "based on" should be understood as "based at least in part on". The terms "one embodiment" and "embodiment" should be understood as "at least one embodiment". The term "another embodiment" should be understood as "at least one other embodiment". The terms "first", "second", etc. may refer to different or the same objects. Other definitions (explicit and implicit) may be included below. Unless the context clearly indicates otherwise, the definitions of terms are consistent throughout the specification.
[0044] Unless otherwise specified or limited, the terms "mounted," "connected," "supported," and "coupled," and variations thereof, are used broadly and encompass both direct and indirect mountings, connections, supports, and couplings. Additionally, "connected" and "coupled" are not limited to physical or mechanical connections or couplings. In the following description, like reference numerals and labels are used to describe the same, similar, or corresponding parts of the figures. Additional definitions (explicit and implicit) may be included below.
[0045] As mentioned above, sensor-based robotic applications play an important role in agile and flexible automation. Multiple sensors can be set to monitor the posture of the robot and detect data related to the position of the robot or the movement of the robot. By using sensor-based robotic applications, debugging time can be reduced and production efficiency can be improved. At the same time, robot skills and product varieties can be deeply expanded.
[0046] Figure 1 An example operating environment in which embodiments of the present disclosure may be implemented is shown. Figure 1 As shown, operating environment 10 may include robot 210 and object 230. In some embodiments, object 230 may be considered a stationary object. In some other embodiments, object 230 may be movable relative to robot 210. Robot 210 may be operated to capture an object or perform a specific action on an object.
[0047] like Figure 1 The illustrated operating environment 10 may also include sensors 221, 222, and 223 and a computing device 140. The sensors 221, 222, and 223 may monitor the posture or position of the robot and the object, and / or the positional relationship between the robot and the object, and transmit the sensing data to the computing device 140. The robot 210 may also transmit its own feedback data to the computing device 140. It should be understood that Figure 1The number of sensors shown is given for illustrative purposes and does not imply any limitation. Operating environment 10 may include any suitable number of sensors.
[0048] In some embodiments, at least one of sensors 221, 222, and 223 may be a camera. Based on the data provided by the camera, computing device 140 may determine the position of the robot and the object, and estimate the position the robot should reach in a subsequent action.
[0049] Typically, in a conditional use scenario of a sensor-based robot application, the robot's operation tasks can be divided into several processes, and each process uses only one sensor (e.g., sensor 221) to regulate the robot's operation. For example, vision can be used to guide the robot to a position first, and then force controls can be applied to assemble parts.
[0050] In other conditional use scenarios, sensors 221, 222, and 223 may be used simultaneously, which may require combining the outputs of different individual sensors 221, 222, and 223. For example, the control signal is determined based on a weighted sum of the outputs of the force sensor and the vision sensor.
[0051] Although multi-sensor based servo systems are the future technology in robotic applications, which improves intelligence and adaptability in dynamic environments, there are some problems in either case. For example, sensors are not synchronized. Due to different sampling rates, data from different sensors are likely to be collected at different moments. Even in the case of "one sensor at a time", the sensors are not synchronized with the robot controller. In addition, all sensors have time delays. In some extreme cases, data may be lost during the communication process. This makes the above-mentioned asynchrony problem more serious. Moreover, each sensor can only estimate part of the state of the target object, which may be overlapping or separate. The conventional solution of generating commands for the robot by simply adding the output of each controller may not be sufficient in this case.
[0052] Therefore, the embodiments of the present disclosure propose a solution for operating a robot. A general framework suitable for operating a traditional robot can be provided, which can receive data from the robot and sensor side, and predict the target position of the robot by fusing data from different data sources.
[0053] Combine the following Figures 2 to 6 The principles and implementations of the present disclosure are described in detail. Figure 2 A schematic diagram of a system for operating a robot according to an embodiment of the present disclosure is shown. For the purpose of discussion, reference will be made to Figure 1Describe the system 100. It should be understood that although the system 100 has been Figure 1 The system 100 is described in the operating environment 10 of the present invention, but the system 100 can also be applied to other operating environments.
[0054] like Figure 2 As shown, the system 100 may include a first data collector 111 and a second data collector 112. The first data collector 111 may be coupled to the robot 210 and the second data collector 112 may be coupled to a set of sensors (sensors 221-223).
[0055] The robot 210 may transmit its own feedback data to the first data collector 111, and the first data collector 111 may determine the first position information of the robot based on the received feedback data. In some embodiments, the feedback data may refer to a set of coordinate parameters of the joint end of the robot in the coordinate system of the robot recorded by the robot controller in a first time period during the robot motion process. For example, the parameters of the robot tool center point (TCP) may be considered as feedback data.
[0056] For an industrial robot, one or more tools may be mounted on the robot to manipulate an object. In order to describe the tool in space, a coordinate system on the tool, i.e., a tool coordinate system (TCS), may be defined, and the origin of the tool coordinate system may be considered as the TCP. In TCS, typically 6 degrees of freedom or 6 pieces of information are required to fully define the posture of the robot end joint, since it may move in three directions in space and may also rotate around three directions. In some embodiments, the first data collector 111 may then determine the first position information of the robot based on the set of coordinate parameters.
[0057] In addition, the first data collector 111 may also include a robot model, which can record the position of the robot TCP and mark the data with a timestamp. The timestamp may be the original timestamp minus the time delay executed by the robot.
[0058] Reference again Figure 2 , the sensor 221 may transmit the sensing data to the second data collector 112 , and the second data collector 112 may determine second position information of the object 230 to be operated by the robot 210 based on the sensing data received from the sensor 221 .
[0059] For example, the sensing data may be referred to as a set of images captured by the sensor during a second time period. The set of images may reflect a set of positional relationships between the robot 210 and the object 230 in the coordinate system of the sensor 211. The second time period may overlap with the first time period, during which feedback data is recorded by the robot controller.
[0060] The second data collector 112 may determine the second position information based on the set of position relationships. In some embodiments, the second position information may refer to the position information of the object 230 relative to the robot 210. The second position information may also include the position information of the object 230 and the position information of the robot 210.
[0061] In addition, the second data collector 112 may also include a sensor model. The sensor model is the relationship between the sensed data and the robot motion. This module can transform the sensed data into a position estimate and mark the data with a timestamp. The timestamp can also be the original timestamp minus the time delay.
[0062] As described above, the first data collector and the second data collector can receive data from their individual data sources. The first data collector and the second data collector can be considered as the interface layer of the system 100. Generally, the interface layer is responsible for receiving and converting external information. The user can configure the interface type and related parameters. Further, the interface layer may include a command interface, which can be used to receive external commands and notify the downstream processing layer of the application to be executed. The sensor pool of the interface layer can be a set of user-defined sensor models. Each sensor model has a communication interface for receiving data, a model for transforming sensed data into position information, and a data buffer for recording marked sensed data. The timestamp is marked with a time delay, and the time delay is used to compensate for delays in the downstream processing layer.
[0063] As described above, data from both the robot side and the sensor side will be collected by the first layer of the system and transformed into corresponding position information, which is necessary for the subsequent estimation of the robot's target position. In particular, a set of feedback / sensing data captured over a certain period of time can be transmitted to the data collector instead of only transmitting sensing data related to a single operation of the robot, which is beneficial for the prediction of the target position because the predicted result will be more accurate and the prediction process will be faster.
[0064] like Figure 2 As shown, the system 100 may further include a first estimator 121. The first estimator may be coupled to the first data collector and the second data collector and configured to generate a first prediction of the target position based on the first position information and the second position information. The object 230 may be operated by the robot 230 at the target position.
[0065] In some embodiments, the first position information and the second position information may be fused in the first estimator 121. For example, the first estimator 121 may obtain a first set of sampling parameters from the first position information and obtain a second set of sampling parameters from the second position information. The first set of sampling parameters may characterize a reference position of the robot 210 at a predetermined time point, and the second set of sampling parameters may characterize a reference position relationship between the robot 210 and the object 230 at the same time point.
[0066] The first estimator 121 may also generate a first prediction of the target location by fusing the first set of sampled parameters and the second set of sampled parameters based on a fusion pattern, which may be associated with an intended operation to be performed by the robot.
[0067] Generally, there is a time delay between the time point when the data collector receives the data and the sampling time point of the feedback data of the robot or the sensing time point of the sensing data of the sensor, so before the prediction process, timestamp calibration is necessary. Compared with the sensing data of the sensor 221, the feedback data of the robot 210 generally has a higher feedback frequency. Therefore, a buffer is provided in the first estimator 121.
[0068] In some embodiments, the buffer can record the data of the sensor and the feedback data of the robot 210. After the user configures the time delay T_delay of each sensor, each sensor data can be timestamped when it is recorded. For example, if the timestamp T_record represents the time when the system 100 receives the data, the sampling time point T_record of the data (where the data is accurately sampled) can be expressed as T_receive minus the time delay T_delay of the sensor. Therefore, before data fusion, the first estimator 121 can calibrate the first set of timestamps of the first position information and the second set of timestamps of the second information based on the predetermined time delay. In this way, the accuracy of the prediction will be significantly improved.
[0069] refer to Figure 4A and Figure 4B , the data fusion of the first set of sampling parameters from the first position information and the second set of sampling parameters from the second position information can be further described in detail.
[0070] Figure 4A shows an example of a curve 401 of a sampled signal associated with second position information, Figure 4B An example of a curve 402 of a sampled signal associated with first position information is shown. Figure 4AAs shown, the second set of sampled parameters may refer to parameters at time points T3 and T8. That is, for example, a set of parameters sampled at time point T3 of curve 401 may reflect the positional relationship between robot 210 and object 230 captured by sensor 221 at time point T3. In order to fuse the first set of parameters and the second set of parameters, the parameters sampled at T3 of curve 402 may be required. However, the parameters sampled at T3 of curve 402 may not be directly read from curve 402. That is, the first set of sampled parameters may not be directly obtained from the first position information. Instead, the sampled parameters at time point T3 of curve 402 may be calculated based on parameters sampled at other sampling time points, for example, based on parameters sampled at T2 and T4 of curve 402. The sampled parameters at time point T3 of curve 402 may reflect the position of the robot at time point T3. Then, the sampled parameters at time point T3 of curve 401 may be fused with the sampled parameters at time point T3 of curve 402 to generate a prediction of the target position of the robot. A similar process may be performed for data fusion of parameters sampled at time point T8 for both curves 401 and 402 .
[0071] In some embodiments, the first estimator 121 further includes an operator pool to provide operators corresponding to fusion modes, such as Kalman mode, prediction mode, filter mode, observer mode, summation mode or subtraction mode. These fusion modes can be preconfigured for specific tasks assigned by the sensor model.
[0072] As described above, the data fusion performed in the first estimator 121 can be referred to as a first fusion process. The first fusion process is used to fuse one sensor data with the robot motion feedback. In most sensors in handheld applications, the sensor data is affected by the deviation between the position / orientation of the robot actuator and the position / orientation of the target. Therefore, the sensor module can first provide an estimate of the position / orientation error between the robot actuator and the target. If the sensor has a time delay, the buffer module can provide an estimate of the robot actuator position / orientation at the time when the sensor samples the data by regressing the robot motion feedback. By adding these sampled data, the system 100 can provide an estimate of the target position / orientation when the sensor samples the data.
[0073] Reference again Figure 2 , the system 100 may further include a command generator 130 coupled to the first estimator 121. The command generator 130 may be configured to generate a command for operating the robot 210 based at least in part on the first prediction of the target position.
[0074] In the data fusion process, the position information of the robot and the position information of the object at the same acquisition time point can be considered as data for predicting the target position. By predicting the target position based on data from different data sources, high-quality prediction results can be obtained more efficiently. At the same time, multiple fusion modes corresponding to specific operators can be pre-determined based on the expected operation process of the robot. In this way, any exceptional robot operation process can be more easily implemented and developed.
[0075] In some embodiments, more than one sensor may transmit sensing data to the second data collector 112, and thus multiple first predictions of the target location may be generated at the first estimator 121. In this case, the multiple first predictions may be considered preliminary predictions and may be further fused to generate a final prediction.
[0076] Figure 3 A schematic diagram of a system for operating a robot according to an embodiment of the present disclosure is shown. Figure 2 Compared to the above, a second estimator 122 is added. Other components can be Figure 2 Corresponding components shown are the same or similar.
[0077] like Figure 3 As shown, the sensor 222 may transmit the sensing data to the second data collector 112. The second data collector 112 may determine third position information of the object 230 based on the sensing data from the sensor 222.
[0078] As described above, the first position information obtained from the first data collector 111 and the second position information obtained from the second data collector 112 may be fused to generate a prediction of the target position of the robot 210. Similarly, the first estimator 121 may also generate a second prediction of the target position based on the first position information and the third position information, for example, by fusing the first position information and the third position information.
[0079] Further references Figure 4B and Figure 4C , the data fusion process of the first position information and the third position information may be similar to the data fusion process of the first position information and the second position information.
[0080] Figure 4C An example of a curve 403 showing a sampled signal associated with the third position information, Figure 4B An example of a curve 402 of a sampled signal associated with first position information is shown. Figure 4CAs shown, the third set of sampled parameters may refer to parameters at time points T4 and T10. That is, for example, a set of parameters sampled at time point T4 of curve 403 may reflect the positional relationship between robot 210 and object 230 captured by sensor 222 at time point T4. In order to fuse the first set of parameters and the third set of parameters, the parameters sampled at T4 of curve 402 may be required. However, the parameters sampled at T4 of curve 402 may not be directly read from curve 402. That is, the first set of sampled parameters may not be directly obtained from the first position information. Instead, the sampled parameters at time point T4 of curve 402 may be calculated based on the parameters sampled at other sampling time points, for example, based on the parameters sampled at T2 and T5 of curve 402. The sampled parameters at time point T4 of curve 402 may reflect the position of the robot at time point T4. Then, the sampled parameters at time point T4 of curve 403 may be fused with the sampled parameters at time point 4 of curve 402 to generate a further prediction of the target position of robot 210. A similar process may be performed for data fusion of parameters sampled at time point T10 of both curves 403 and 402 .
[0081] Figure 5A-Figure 5B The embodiment according to the present disclosure is shown Figure 4A-4C Example results of data fusion are shown. Figure 5A shows the result of data fusion of the first position information and the second position information, Figure 5B The data fusion result of the first position information and the third position information is shown. For example, Figure 5A The value of the curve 501 shown represents the result of data fusion of the parameters of the first position information and the second position information at time point T3. Based on the curve 501 obtained by fusing the first position information and the second position information and the curve 502 obtained by fusing the first position information and the third position information, two preliminary predictions of the target position can be generated, that is, the value P1 in the curve 501 and the value P2 of the curve 502.
[0082] Reference again Figure 3 , the system 100 may further include a second estimator 122. The second estimator 122 may be coupled to the first estimator 121 and generate a third prediction of the target position by fusing the first prediction of the target position and the second prediction of the target position. Figure 5A and Figure 5B As shown, the preliminary prediction of the target position P1 and the preliminary prediction of the target position P2 can be fused, and the fused value can be considered as the final prediction of the target position. In this case, the command generator 130 can generate a command for operating the robot 210 based on the final prediction of the target position.
[0083] The data fusion used to generate the final prediction can be considered as a second fusion process. The preliminary prediction of the target position / orientation can be sent to the buffer along with the timestamp in the second fusion module in the second estimator. In most applications, the movement speed of the object is relatively low compared to the movement of the robot. Therefore, the regression of the target position and orientation can provide more reliable predictions than the regression of sensor data.
[0084] In general, estimators 121 and 122 can be considered as the estimation layer of the system. The estimation layer estimates the target pose and velocity of the robot based on different sensor data and feedback data of the robot. The estimation operation is specific in different tasks, which are stored in the task pool of this layer.
[0085] The command generator 130 can be considered as the servo layer of the system. The pose predictions of the target and robot actuators are sent to the online planner. The speed, acceleration and accuracy of the planner can be configured by the user, and the planner will calculate the motion instructions according to the configuration and send them to the robot. The cycle time of the planner is independent of the feedback rate of the sensor, thereby improving the servo performance of slow speed sensors.
[0086] It should be understood that Figure 2 and Figure 3 The system 100 shown may be implemented in any hardware and software. For example, the system 100 may be implemented as follows: Figure 1 The computing device 140 shown. The system 100 may also be implemented as an integrated chip. The components of the system 100 may be considered as entities capable of performing certain functions, such as data collectors, estimators, instruction generators, etc. The components in the system 100 may also be considered as virtual modules capable of implementing certain functions.
[0087] In view of the above situation, Figure 2 and Figure 3 In the illustrated system 100, data from both the robot side and the sensor side will be collected by the first stage of the system and transformed into corresponding position information, which is necessary for the subsequent estimation of the target position of the robot. In particular, a set of feedback / sensing data captured within a certain period of time can be transmitted to the data collector, rather than only transmitting sensing data related to a single operation of the robot, which is beneficial to the prediction of the target position because the predicted result will be more accurate and the prediction process will be faster.
[0088] By means of the system proposed in the present disclosure, robot operations can be performed more accurately and efficiently. For the user-side operator, sensor-based servos can provide a more flexible solution. The system proposed in the first aspect can be used with any conventional robot that the user has already deployed. In this way, the user does not have to bear the costs caused by the modification of a large number of components. For manufacturers, the system can simplify the process of implementing multi-sensor based solutions and can reduce development time. At the same time, the capabilities of the automation field can be expanded and the performance of sensor-based applications can be improved.
[0089] Figure 6 A flow chart illustrating a method for operating a robot according to an embodiment of the present disclosure is shown. For the purpose of discussion, the method 600 will be described from the perspective of the system 100. Figure 2-Figure 3 Give a description.
[0090] At 610, the system 100 determines first position information of the robot based on feedback data received from the robot. At 620, the system 100 determines second position information of an object to be manipulated by the robot based on sensing data received from the sensor; generating based on the first position information and the second position information.
[0091] In some embodiments, determining the first position information includes receiving feedback data from the robot within a first time period; obtaining a set of coordinate parameters of the joint end of the robot in the coordinate system of the robot from the feedback data of the robot, and determining the first position information based on the set of coordinate parameters.
[0092] In some embodiments, determining the second position information includes receiving sensing data from the sensor within a second time period, the second time period overlapping with a first time period for receiving feedback data from the robot; acquiring a set of position relationships between the robot and the object in another coordinate system of the sensor from the sensing data of the sensor; and determining the second position information based on the set of position relationships.
[0093] At 630 , the system 100 generates a first prediction of a target position at which the object is to be manipulated by the robot based on the first position information and the second position information.
[0094] In some embodiments, the system 100 may further calibrate the first set of timestamps for the first location information and the second set of timestamps for the second information based on a predetermined time delay.
[0095] In some embodiments, generating a first prediction includes obtaining a first set of sampling parameters representing a reference position of the robot at a predetermined point in time from the first position information; obtaining a second set of sampling parameters representing a reference position relationship between the robot and the object at the predetermined point in time from the second position information; and generating a first prediction of the target position by fusing the first set of sampling parameters and the second set of sampling parameters based on a predetermined fusion pattern associated with an expected operation to be performed by the robot.
[0096] In some embodiments, the fusion mode includes at least one of a prediction mode, a filtering mode, a summation mode, and a subtraction mode.
[0097] At 640 , the system 100 generates commands for operating the robot based at least in part on the first prediction.
[0098] In some embodiments, the system 100 may further determine third position information of the object based on additional sensory data received from additional sensors, and generate a second prediction of the target position based on the first position information and the third position information.
[0099] In some embodiments, the system 100 may also generate a third prediction of the target location by fusing the first prediction of the target location with the second prediction of the target location.
[0100] In some embodiments, the system 100 may also generate commands for operating the robot based on the third prediction.
[0101] Figure 7 700 is a simplified block diagram of a device 700 suitable for implementing embodiments of the present disclosure. The device 700 may be provided to implement Figure 2 and Figure 3 The system 100 shown. Figure 7 As shown, the device 700 may include a computer processor 710 coupled to a computer readable memory unit 720, and the memory unit 720 includes instructions 722. When executed by the computer processor 710, the instructions 722 may implement the method for operating a robot as already described in the previous paragraphs, the details of which will be omitted hereinafter.
[0102] In some embodiments of the present disclosure, a computer readable medium for simulating at least one object in a production line is provided. The computer readable medium stores instructions, which when executed on at least one processor can cause the at least one processor to perform the method for operating a robot as described in the previous paragraphs, and the details will be omitted below.
[0103] Generally, various embodiments of the present disclosure may be implemented using hardware or dedicated circuits, software, logic, or any combination thereof. Some aspects may be implemented using hardware, while other aspects may be implemented using firmware or software that may be executed by a controller, microprocessor, or other computing device. Although various aspects of the embodiments of the present disclosure are illustrated and described as block diagrams, flow charts, or using some other graphical representations, it should be understood that, as non-limiting examples, the blocks, devices, systems, techniques, or methods described herein may be implemented using hardware, software, firmware, dedicated circuits or logic, general-purpose hardware or controllers or other computing devices, or some combination thereof.
[0104] The present disclosure also provides at least one computer program product tangibly stored on a non-transitory computer-readable storage medium. The computer program product includes computer executable instructions, such as instructions included in a program module, which are executed in a device on a target real or virtual processor to perform the above referenced Figure 6 A program module is a process or method described in a program. Typically, a program module includes routines, programs, libraries, objects, classes, components, data structures, etc. that perform specific tasks or implement specific abstract data types. In various embodiments, the functions of program modules can be combined or split between program modules as needed. The machine executable instructions of program modules can be executed in local or distributed devices. In distributed devices, program modules can be located in both local and remote storage media.
[0105] The program code for executing the disclosed method can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer or other programmable data processing device so that the program code causes the function / operation specified in the flow chart and / or block diagram to be realized when executed by the processor or controller. The program code can be executed completely on the machine, partially on the machine, as an independent software package, partially on the machine and partially on a remote machine, or completely on a remote machine or server.
[0106] The above program code can be embodied in a machine-readable medium, which can be any tangible medium that can contain or store a program for use by an instruction execution system, device or equipment or used in combination with it. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. The machine-readable medium can include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium will include an electrical connection with one or more wires, a portable computer floppy disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
[0107] In addition, although operations are described in a particular order, this should not be understood as requiring such operations to be performed in the particular order shown or in sequence or performing all of the operations shown to obtain the desired result. In some cases, multitasking and parallel processing may be advantageous. Similarly, although several specific implementation details are included in the above discussion, these should not be interpreted as limitations on the scope of the present disclosure, but rather as descriptions of features that may be specific to a particular embodiment. Certain features described in the context of a separate embodiment may also be implemented in combination in a single embodiment. On the other hand, the various features described in the context of a single embodiment may also be implemented in multiple embodiments individually or in any suitable sub-combination.
[0108] Although the subject matter has been described in language specific to structural features and / or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are disclosed as example forms of implementing the claims.
Claims
1. A system (100) for operating a robot (210), comprising: A first data collector (111) coupled to the robot (210) and configured to determine first position information of the robot (210) based on feedback data received from the robot (210); a second data collector (112) coupled to the sensor (221) and configured to determine second position information of an object (230) to be operated by the robot (210) based on the sensing data received from the sensor (221); a first estimator (121) coupled to the first data collector (111) and the second data collector (112) and configured to generate a first prediction of a target position at which the object (230) is to be manipulated by the robot (210) based on the first position information and the second position information; as well as a command generator (130) coupled to the first estimator (121) and configured to generate commands for operating the robot (210) based at least in part on the first prediction; The first estimator (121) is further configured to: Acquiring a first set of sampling parameters representing a reference position of the robot (210) at a predetermined time point from the first position information; Acquiring a second set of sampling parameters representing a reference position relationship between the robot (210) and the object (230) at the predetermined time point from the second position information; as well as The first prediction of the target position is generated by fusing the first set of sampled parameters and the second set of sampled parameters based on a fusion pattern associated with an expected operation to be performed by the robot (210).
2. The system (100) according to claim 1, wherein the first data collector (111) is further configured to: receiving the feedback data of the robot (210) within a first time period; acquiring a set of coordinate parameters of the joint end of the robot (210) in the coordinate system of the robot (210) from the feedback data of the robot (210); and The first position information is determined based on the set of coordinate parameters.
3. The system (100) of claim 1, wherein the second data collector (112) is further configured to: receiving the sensing data from the sensor (221) within a second time period, the second time period overlapping with a first time period for receiving the feedback data from the robot (210); Acquire a set of positional relationships between the robot (210) and the object (230) in another coordinate system of the sensor (221) from the sensing data of the sensor (221); as well as The second position information is determined based on the set of position relationships.
4. The system (100) according to claim 1, wherein the first estimator (121) is further configured to: A first set of time stamps for the first position information and a second set of time stamps for the second position information are calibrated based on a predetermined time delay.
5. The system (100) of claim 1, wherein the fusion mode comprises at least one of the following: Prediction model, Filter mode, summation mode, and Subtraction mode.
6. The system (100) of claim 1, wherein the second data collector (112) is further coupled to an additional sensor (222) and is configured to determine third position information of the object (230) based on additional sensing data received from the additional sensor (222); and The first estimator (121) is further configured to generate a second prediction of the target position based on the first position information and the third position information.
7. The system (100) according to claim 6, further comprising: A second estimator (122) is coupled to the first estimator (121) and is configured to generate a third prediction of the target position by fusing the first prediction of the target position and the second prediction of the target position.
8. The system (100) of claim 7, wherein the command generator (130) is further configured to generate the command for operating the robot (210) based on the third prediction.
9. A method for operating a robot (210), comprising: Determining first position information of the robot (210) based on feedback data received from the robot (210); determining second position information of an object (230) to be operated by the robot (210) based on sensing data received from the sensor (221); generating a first prediction of a target position at which the object (230) is to be manipulated by the robot (210) based on the first position information and the second position information; as well as generating commands for operating the robot (210) based at least in part on the first prediction; Wherein generating the first prediction comprises: Acquiring a first set of sampling parameters representing a reference position of the robot (210) at a predetermined time point from the first position information; acquiring a second set of sampling parameters representing a reference position relationship between the robot (210) and the object (230) at the predetermined time point from the second position information; and The first prediction of the target position is generated by fusing the first set of sampled parameters and the second set of sampled parameters based on a predetermined fusion pattern associated with an expected operation to be performed by the robot (210).
10. The method of claim 9, wherein determining the first location information comprises: receiving the feedback data of the robot (210) within a first time period; acquiring a set of coordinate parameters of the joint end of the robot (210) in the coordinate system of the robot (210) from the feedback data of the robot (210); and The first position information is determined based on the set of coordinate parameters.
11. The method of claim 9, wherein determining the second location information comprises: receiving the sensing data from the sensor (221) within a second time period, the second time period overlapping with a first time period for receiving the feedback data from the robot (210); Acquire a set of positional relationships between the robot (210) and the object (230) in another coordinate system of the sensor (221) from the sensing data of the sensor (221); as well as The second position information is determined based on the set of position relationships.
12. The method according to claim 9, further comprising: A first set of time stamps for the first position information and a second set of time stamps for the second position information are calibrated based on a predetermined time delay.
13. The method according to claim 9, wherein the predetermined fusion mode comprises at least one of the following: Prediction model, Filter mode, summation mode, and Subtraction mode.
14. The method according to claim 9, further comprising: determining third position information of the object (230) based on additional sensing data received from an additional sensor (222); as well as A second prediction of the target position is generated based on the first position information and the third position information.
15. The method according to claim 14, further comprising: A third prediction of the target position is generated by fusing the first prediction of the target position and the second prediction of the target position.
16. The method according to claim 15, further comprising: The command for operating the robot (210) is generated based on the third prediction.
17. An electronic device comprising: processor; as well as A memory coupled to the processor and storing instructions for execution, which, when executed by the processor, cause the device to perform the method according to any one of claims 9 to 16.
18. A computer-readable medium comprising program instructions for causing an electronic device to at least execute the method according to any one of claims 9 to 16.
