Robot control system, robot control method and program

Abstract

To provide an innovative cooperative robotic (or robot) system to perform, and a method to provide, highly efficient cooperative work between the cooperative robotic system and a cooperative operator.SOLUTION: There are provided systems and methods for controlling a cooperative robotic device including generating, for the cooperative robotic device, a plurality of candidate task end states associated with a first task. The method may further include: selecting, for the cooperative robotic device, a first task end state from the plurality of candidate task end states; receiving load data regarding a load experienced by the cooperative robotic device; and selecting, based on the load data, a second task end state from the plurality of candidate task end states.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present disclosure is generally directed to collaborative robot systems. [Background technology]

[0002] In the fields of manufacturing and construction, in order to meet diversifying customer needs, attention is being paid to high-mix low-volume production and mass customization, which are different from traditional mass production methods. These methods require frequent adjustments and changes in production volume and processes to meet diverse needs, so companies are looking for personnel (multi-skilled workers) who can handle a large number of complex processes to deal with such situations. On the other hand, labor shortages have become more serious in recent years, making it more difficult to secure human resources. Therefore, there is a need for more advanced automation and labor-saving using robots, AI, and other digital tools to replace labor.

[0003] When introducing robots into actual work sites, it is difficult for robots to completely take over human work at the current technology level, but there are cases where robots take over object transportation and other relatively simple tasks, allowing humans to focus on more complex tasks. In these cases, humans and robots may work more closely with each other. Traditionally, robots have been used within a range, i.e., an area, surrounded by a safety fence to prevent humans from approaching the robot when the robot is in operation. However, in recent years, collaborative robots have become more common, and there are many use cases where robots operate without a safety fence and in positions where they may come into contact with humans. However, although current collaborative robots have enhanced safety mechanisms to prevent injuries when they come into contact with humans, there are few actual examples of collaborative robots working simultaneously with humans. In fact, there are robot systems that work in the same space as humans, but most of them work in sequence with humans, and there are few examples of work sharing the same time and space, for example, robots moving and working together at the same time as the associated human worker moves.

[0004] When a robot collaborates with a human worker, especially when the human and the robot work simultaneously, the human may take unexpected actions. In this case, existing robots may be configured to execute pre-planned actions, and the robot does not modify its own actions in response to unexpected actions taken by the collaborative operator during its movement. Thus, the robot does not respond to the actions of the collaborative operator, and the robot's actions may be inappropriate for the collaborative task, which may interrupt the task and reduce the efficiency of the collaborative task. For example, when the collaborative operator reaches out to pick up a part or tool, the operator's hand or arm may be located in the path of the robot, causing the robot to come into contact with the human operator, which may interrupt the task. If the position of the collaborative operator's body is measured to detect whether the robot is in the path of the robot, and if so, the robot's motion is re-planned and the task is interrupted during the re-planning process, thereby reducing the efficiency of the collaborative task. Some robotic systems configure the robot to continue (i.e., begin a new motion / task) after being stopped by contact with a human worker based on a specific force pattern applied by the human worker, but such systems may not be easy to use or may reduce the efficiency of collaborative tasks because they require the human worker to perform certain motions (e.g., apply a specific force pattern) that may require additional time after stopping.

[0005] For example, since a conventional robot only moves repeatedly to a stored position, a robotic device may be made to perform a task using a motion pattern that specifies a position according to the work content. However, in tasks such as such collaborative work with humans (especially for high-mix low-volume production and / or mass customization), the work content may change irregularly, the work object may not be fixed, and the position of the work object may change frequently, i.e., constantly, as it is manipulated by the collaborative human worker. Specifically, in the case of collaborative work with a human worker, it may be impossible to achieve a specific objective (e.g., associated with a predefined goal or task) by repeatedly moving to a set of specified positions. Therefore, a collaborative robot system must be able to generate an updated goal or task and an associated set of instructions for the robot based on the situation around the robot and collaborative operator and / or the force due to the contact between the human and the robot, using an environmental measurement device such as a camera or a distance measurement device (e.g., RADAR or LIDAR) that recognizes the situation around the robot and the collaborative operator, and / or a sensor that detects the external force due to the contact between the human and the robot. However, some previous robotic systems must perform a full reevaluation of the goal after each detected change to the situation or detected force due to human contact and / or in response to additional human input before resuming or continuing the operation. Thus, a collaborative robotic system that can respond to each detected change to the situation or detected force due to human contact without a full reevaluation can enable more efficient collaboration between a human worker and the collaborative robotic system. Summary of the Invention

[0006] Exemplary implementations described herein include an innovative collaborative robotic (i.e., robot) system that performs highly efficient collaboration between the collaborative robot system and a collaborative operator by appropriately responding to actions of the collaborative operator (e.g., a human worker) and continuing the collaboration without interruption, as well as a method for providing highly efficient collaboration. The collaborative robot system may, in some embodiments, include a goal planning module that generates a plurality of robot task targets (e.g., candidate task goals or candidate task end states associated with one or more work tasks or objectives) for a robot of the collaborative robot system before the robot performs a particular task of the plurality of robot task targets. The collaborative robot system may also, in some embodiments, include a load estimation module that detects a load situation of the robot (e.g., a magnitude and direction of a current and / or historical load applied to the robot). The collaborative robot system may further, in some embodiments, include a motion planning module that selects one of the robot task targets based on the load situation of the robot and plans a motion of the robot associated with the selected robot task target. In some embodiments, the goal planning module determines multiple candidate goals (goal states) according to the task content during the preliminary motion planning, and scores each candidate goal. The load estimation module may detect the load state of the robot while the robot executes the motion associated with the selected robot task target. Based on the detected load state, the motion planning module may update the robot task target selection to change the robot task target or select one of multiple robot task targets (associated with one or more work tasks or objectives) based on the scores calculated for or associated with the multiple robot task targets (e.g., candidate task end states associated with one or more work tasks or objectives) and the detected load state (i.e., situation).The motion planning module, in some aspects, may generate an instruction set for executing the motion corresponding to the updated (newly selected) robot task target selection and provide the instruction set to a robot of the collaborative robot system.

[0007] Aspects of the present disclosure include a method of controlling a collaborative robotic device including generating, for the collaborative robotic device, a plurality of candidate task end states associated with a first task. The method may further include, for the collaborative robotic device, selecting a first task end state from the plurality of candidate task end states, receiving load data related to loads experienced by the collaborative robotic device, and selecting a second task end state from the plurality of candidate task end states based on the load data.

[0008] Aspects of the present disclosure include a non-transitory computer-readable medium storing instructions for execution by a processor, which may include instructions for controlling a collaborative robotic device, including instructions for generating, for the collaborative robotic device, a plurality of candidate task end states associated with a first task. The instructions may further include instructions for the collaborative robotic device to select a first task end state from the plurality of candidate task end states, receive load data regarding loads experienced by the collaborative robotic device, and select a second task end state from the plurality of candidate task end states based on the load data.

[0009] Aspects of the present disclosure include a system that may include a means for controlling a collaborative robotic device, including a means for generating, for the collaborative robotic device, a plurality of candidate task end states associated with a first task. The system may further include a means for selecting, for the collaborative robotic device, a first task end state from the plurality of candidate task end states, a means for receiving load data regarding loads experienced by the collaborative robotic device, and a means for selecting, based on the load data, a second task end state from the plurality of candidate task end states.

[0010] Aspects of the present disclosure include an apparatus that can include a memory and at least one processor coupled to the at least one memory, where the at least one processor may be configured, based at least in part on information stored in the at least one memory, individually or in any combination, to generate, for the collaborative robotic device, a plurality of candidate task end states associated with a first task. The at least one processor may be further configured to select, for the collaborative robotic device, a first task end state from the plurality of candidate task end states, receive load data regarding loads experienced by the collaborative robotic device, and select, based on the load data, a second task end state from the plurality of candidate task end states. [Brief description of the drawings]

[0011] [Figure 1] FIG. 1 illustrates components associated with a collaborative robot system, according to some aspects of the present disclosure. [Diagram 2] FIG. 1 illustrates a diagram of component elements of a collaborative robot system, according to some aspects of the present disclosure. [Diagram 3] FIG. 1 illustrates a robotic device and a worker assembling a work object placed on a workbench or transported by an autonomous vehicle. [Figure 4] FIG. 2 is a diagram illustrating a software configuration of a robot task controller. [Diagram 5] 1 is a flow chart illustrating the overall process the robot control system uses to set robot task targets and execute robot movements. [Figure 6] FIG. 2 illustrates an example of an object recognition result from a detection module. [Figure 7] FIG. 2 is a schematic block diagram showing a detailed software configuration of a goal planning module. [Figure 8] 6 is a flowchart illustrating a detailed process of one step in FIG. 5. [Figure 9] FIG. 13 is a diagram illustrating an example that illustrates a goal planning module that calculates candidate goal states. [Figure 10] FIG. 2 is a schematic block diagram showing a detailed software configuration of a motion planning module and a load estimation module. [Figure 11] 6 is a flowchart illustrating a detailed process of the steps in FIG. 5. [Figure 12] FIG. 13 is a diagram illustrating an example illustrating how the motion planning module and the load estimation module estimate the load state and modify the task goal according to the load state. [Figure 13] FIG. 2 is a diagram illustrating a software configuration of a robot task controller. [Figure 14] FIG. 13 illustrates how the control system can set robot goals according to operator characteristics. [Figure 15] FIG. 13 illustrates a list of example load patterns that may be used in some aspects of the disclosure to modify and / or update a task goal. [Figure 16] FIG. 1 is a flow diagram illustrating a method according to some aspects of the present disclosure. [Figure 17] FIG. 1 is a flow diagram illustrating a method according to some aspects of the present disclosure. [Figure 18] FIG. 1 illustrates an example computing environment having example computing devices suitable for use in some example implementations. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] The following detailed description provides details of the drawings and exemplary implementations of the present application. Reference numbers and descriptions of elements that overlap between drawings are omitted for clarity. The terms used throughout the description are provided by way of example and are not intended to be limiting. For example, the use of the term "automatic" may include a fully automatic implementation or a semi-automatic implementation with user or administrator control over certain aspects of the implementation, depending on the desired implementation of the skilled artisan practicing the implementations of the present application. Selection may be performed by a user through a user interface or other input means, or may be realized through a desired algorithm. The exemplary implementations as described herein may be utilized alone or in combination, and the functionality of the exemplary implementations may be realized through any means according to the desired implementation.

[0013] 1 illustrates components associated with a collaborative robot system according to some embodiments of the present disclosure. FIG. 2 illustrates components of a collaborative robot system according to some embodiments of the present disclosure.

[0014] In some embodiments, the robot device 101 may have a moving cart 9, a robot arm 110 attached to the moving cart 9, and an end effector 111 attached to the tip of the robot arm 110, and may perform a task on a work target 102 on a work table 103. Each device of the robot device 101 may be connected to a robot device controller 107 and operates based on a control command (such as a motor current of the robot arm 110, a motor current of the end effector 111, etc.) received from the robot device controller 107. The robot device 101 (i.e., a component device) may also transmit a state of the robot device 101 (e.g., a voltage of an angle sensor attached to a joint of the robot arm 110, or other measurement data) to the robot device controller 107. The robotic device controller 107 may be connected to the robotic task controller 108 via a network 106 (e.g., a wired or wireless network) and may convert (e.g., interpret and / or format information related to) the state of the robotic device 101 obtained from the robotic device 101 (e.g., joint angles of the robotic arm 110, hand positions of the end effector 111, etc.) and provide it to the robotic task controller 108. In some embodiments, the robotic device controller 107 may also calculate control commands for the robotic device 101 based on motion commands (e.g., target positions of the end effector 111, etc.) output from the robotic task controller 108 and the state of the robotic device 101 input from the robotic device 101. The camera 112 (e.g., as an example of an environmental measurement device that recognizes the situation around the robot and the collaborative operator) may be configured to measure the distance (depth) from the camera along with a color image and may be connected to the robotic task controller 108 via the network 106. The camera 112 may provide one or more of the captured data and / or depth information, along with captured images related to the work object 102 and the robotic device 101, to the robotic task controller 108 via the network 106.

[0015] FIG. 2 shows that the robotic device controller 107 may be a computer with a processing unit 171, a network interface 172, a device interface 173, and a data storage 174 in some embodiments, and these components are electrically connected (e.g., by a bus or other communication mechanism, as described below in connection with FIG. 18). The processing unit 171 may include a CPU (Central Processing Unit), a RAM (Random Access Memory), a ROM (Read Only Memory), etc., and may be configured to execute processing based on programs and various data. The data storage 174 may be an auxiliary memory device, such as a hard disk drive, and may store a control program 751 executed by the processing unit 171. The device interface 173 may be an interface that connects to the robotic device 101 and transmits control commands to the robotic device 101 and receives data related to the state of the robotic device 101, and may be appropriately configured for an apparatus that includes the robotic device 101. The network interface 172 may be an interface that connects to the robotic task controller 108 and receives motion commands for the robotic device 101 (e.g., via communication with the network interface 182 of the robotic task controller 108 over the network 106) and transmits (provides) data regarding the state of the robotic device 101. When starting up the robotic device controller 107, such as by turning on the power, the control program 751 stored in the data storage 174 may be provided to (or accessed by) the processing unit 171 and executed. The control program 751 generates control commands for the robotic device 101 based on the motion commands received from the robotic task controller 108 via the network interface 172 and the state of the robotic device 101 received from the device interface 173, and outputs the generated control commands from the device interface 173 to the robotic device 101.In some embodiments, the control program 751 may also provide the status of the robotic device 101 received via the device interface 173 to the robotic task controller 108 via the network interface 172.

[0016] The robotic task controller 108 may be a computer with a processing unit 181, a network interface 182, a user input interface 183, and a data storage 184, electrically connected (e.g., by a bus or other communication mechanism, as described below in connection with FIG. 18). The processing unit 181 may include a CPU, RAM, ROM, etc., in some embodiments, and may be configured to perform information processing based on programs and various data. The network interface 182 may be connected to the robotic device controller 107 in some embodiments, and may be an interface that transmits motion commands for the robotic device 101 to the robotic device controller 107 and receives information about the status of the robotic device 101 (e.g., via communication with the network interface 172 of the robotic device controller 107 via the network 106). The network interface 182 may also communicate with the camera 112 via the network 106 in some embodiments. The user input interface 183 may be any device that receives input from a user, such as a mouse or keyboard, and controls the execution of programs and other operations of the robotic task controller 108. The data storage 184 may be a secondary memory device, such as a hard disk drive, which in some embodiments may store the recognition program 851, the robot task data 852, the goal planning program 853, the load estimation program 861, the motion planning program 862, the control parameter data 863, and / or the robot command generation program 864. When the robot task controller 108 is activated, such as by turning on the power, it may provide (i.e., provide access to) the programs and data stored in the data storage 184 to (or for) the processing unit 181. The recognition program 851 may in some embodiments use images from the camera 112 to detect the work object 102 and other objects.The goal planning program 853 may, in some embodiments, use the recognition results of the recognition program 851 and the control parameter data 863 to plan the robot's task end state (e.g., the destination of the robot's end effector 111 and the end state such as holding / grasping, releasing, or rotating). The load estimation program 861 uses the data of the robotic device controller 107 to estimate the load state of the robot during operation. The motion planning program 862 may, in some embodiments, determine a destination and target position (e.g., associated with the task end state) based on the task end state of the motion planning program 862 and the load state estimation result of the load estimation program 861, and plan the robot's motion to the destination or target position. The robot control command generation program 864 may, in some embodiments, generate motion commands for the robotic device controller 107 based on the robot motion plan (e.g., including a set of positions or other information that defines a path or motion for the robotic device 101) and transmit the motion commands to the robotic device controller 107 via the network interface 182.

[0017] In a place where a wide variety of small-lot production is carried out, work consisting of a plurality of processes (such as assembly work) may be carried out in a single work area (cell), as symbolized by a cell production system. For example, as shown in FIG. 3, a robotic device 101 and a worker 104 may assemble a work object 102 placed on a work table 103 or transported by an autonomous vehicle 105. In such a work site, a change in the work object 102 to be handled, or replacement and / or change in the work content may occur according to fluctuations in demand. For example, after the same worker 104 assembles product A, the work object 102 transported by the autonomous vehicle 105 may be changed according to a change in the production plan, and the work may shift to assembling a different product B. As a method for improving the work efficiency in such a place, there is a use example in which the robotic device 101 supplies the work object 102 and tools to work at the right time and in the right position, so that the worker 104 can concentrate on the assembly work.

[0018] FIG. 4 shows the software configuration of the robot task controller 108. FIG. 5 is a flow chart showing the overall process used by the robot control system to set the robot task goal end state (i.e., task target) and execute the robot motion. The detection module 115 captures the image captured by the camera 112 in step 500, and performs object recognition in step 510. The object recognition can use any detection method, for example, image processing based on the feature of Histogram of Oriented Gradients (HOG), or deep learning method using CNN (Convolutional Neural Network). FIG. 6 shows an example of an object recognition result from the detection module 115. As shown in FIG. 6, the position of the robot hand (e.g., end effector 111), the assembly work object 200 being assembled, the tool 211, the part 221, and the other part 222 in the image are recognized and surrounded by a bounding box. At this time, the 3D position of the detected object is obtained by combining the center position of the bounding box in the image with the depth information. Furthermore, as shown in FIG. 6, the skeletal position of the worker 104 may be estimated and detected from the image to obtain position information of the shoulders, fingertips, etc.

[0019] In some embodiments, the goal planning module 120 may obtain the robot task data 100 that defines the task to be performed by the robot (e.g., providing the tool 211 or part 221 to the worker 104) and the object recognition result of the detection module 115 in step 520, and may output the candidate target positions (associated with the candidate task end states) to which the robot may move. FIG. 7 is a schematic block diagram showing the detailed software configuration of the goal planning module 120. FIG. 8 is a flow chart illustrating the detailed process of step 520 of FIG. 5. FIG. 9 is an illustration illustrating the goal planning module 120 that calculates the candidate task end states (e.g., alternatively referred to as candidate goal states, or candidate task goals or candidate task goal states). In step 521, the reference point detection module 121 detects the reference points according to the task (i.e., the work) to be performed by the robot using the object detection result 109 and the task data (e.g., the robot task data 100). As shown in Fig. 9(a), when the robot-handled object 311 handled by the robot is the tool 211, the right shoulder position (e.g., reference point 310) of the operator (e.g., the worker 104) may be set as the reference point to minimize the distance between the operator's hand and the selected destination of the tool 211 for the convenience of the operator (i.e., the worker 104). On the other hand, as shown in Fig. 9(b), when the robot-handled object 311 is a part of the assembly work object 200, the next attachment position (e.g., the reference point 320) may be set as the reference point to minimize the distance between the object and the attachment position (e.g., the reference point 320) and reduce the operator's reach when assembling the assembly work object 200 by placing the robot-handled object 311 close to the next attachment position (e.g., the reference point 320). In step 522, the target candidate detection module 122 uses the object detection result to output a plurality of candidate task end states (i.e., task goal candidates).In the detection result shown in FIG. 6, a range having a predetermined size is extracted in the area where the object is not detected on the worktable 103, and the center position of the target area associated with the candidate task end states 300, 301, 302, and 303 shown in FIG. 9, the size of the range, and the state of the robot at the center position (the posture of the end effector 111, the opening and closing of the end effector 111, the movement time, etc.) may be output as the candidate task end states (task goal candidates). The extraction of the predetermined range may be configured, for example, to localize the top surface of the worktable 103 with a predetermined grid size and extract a predetermined number of grids and / or grid points. In some embodiments, the distance score calculation module 123 may calculate the straight-line distance between the reference point 310 or the reference point 320 and the candidate task end states 300, 301, 302, and 303 in step 523, and may output the straight-line distance as a score associated with each task goal candidate. In FIG. 9(a), the candidate task end state 300 closest to the reference point 310 has the smallest score, and in FIG. 9(b), the candidate task end state 303 closest to the reference point 320 has the smallest score. In step 524, the motion score calculation module 124 outputs the scores associated with the motion of the robotic device 101. As shown in FIG. 9, the score of the candidate task end state 301 closest to the robot's end effector 111 has the lowest score. In step 525, the score integration module 125 combines the scores calculated by the distance score calculation module 123 and the scores calculated by the motion score calculation module 124, multiplies them by a product of predefined weights, and outputs the scores of the candidate task end states 300, 301, 302, and 303. Finally, the goal planning module 120 may output the positions and scores of the candidate task end states 300, 301, 302, and 303 in step 526.

[0020] The motion planning module 130 may extract, i.e., select, in step 530, the task goal with the highest score among the candidate task end states (task goal candidates) output by the goal planning module 120, and may plan and output, in step 540, a motion to reach the task goal. In some aspects, the motion to reach the task goal may be a series of positions and poses of the end effector 111 separated by a predetermined time interval, or may be a line segment describing the positions through which the end effector 111 must pass. The robot command generation module 150 may generate, in step 550, a robot motion command (e.g., joint angle command) based on the motion information output by the motion planning module 130 and the sensor information (e.g., robot position information) output by the robotic device controller 107, and may output the robot motion command to the robotic device controller 107. In some aspects, the load state estimation module 160 may estimate the load on the robotic device 101 based on the sensor data of the robotic device 101 obtained by the robotic device controller 107 in step 560, and may output the modified score value for each candidate task end state and the modified state of the candidate task end states (candidate task goal) to the motion planning module 130 as the goal modification value detected and / or associated with the load state estimation. In step 570, the motion planning module 130 may re-evaluate the current, i.e., selected, task end state (current task goal) based on the load state of the load state estimation module 160. If there is a change in the score order of the candidate task end states (candidate task goal), the motion planning module 130 returns to before step 530 to modify the task end states (i.e., task goal), but if there is no change, it continues the process. If processing continues, the motion planning module 130 determines in step 580 whether the motion has been completed to the specified state, and if the motion has not been completed, processing returns to step 550 to output additional motion commands to continue the motion as described above, and if the motion has been completed, all processing ends.

[0021] FIG. 10 is a schematic block diagram showing the detailed software configuration of the motion planning module 130 and the load estimation module 160. FIG. 11 is a flow chart illustrating the detailed process of step 560. FIG. 12 is an example illustrating how the motion planning module 130 and the load estimation module 160 estimate the load state and change the task end state (i.e., the task goal) according to the load state. The load vector calculation module 161 may calculate the load vector (size and direction of load) to be applied to the robot-handled object 311 being handled by the robot based on the robot position data (e.g., joint angles, etc.) and robot force data (e.g., joint torques, etc.) output by the robot device controller 107. In step 561, the load score calculation module 162 first checks whether a load exists by determining whether the magnitude (length) of the load vector output by the load vector calculation module 161 is greater than a predetermined value; if the load magnitude is less than the predetermined value, the process may end without further processing; if the load magnitude is greater than the predetermined value, the process may proceed to step 562. In step 562, it may be determined whether the load is oriented towards the candidate task end state with the highest score, i.e., towards the destination the robot is moving to, and if the orientation is consistent, the process proceeds to step 563, otherwise the process proceeds to step 565. In step 563, it may be determined whether the load magnitude is greater than a predetermined level, and if the load magnitude is greater than a predetermined level, the process may proceed to step 564, otherwise the process may end. In step 564, the state of the task end state (i.e., task goal) with the highest score is changed.For example, if the robot-handled object 311 being handled by the robotic device 101 is a tool (e.g., tool 211) and the worker 104 is about to receive it from the robotic device 101 and use the tool, a load is applied in the same direction as that of the candidate task end state 300, as shown by the solid arrow in Fig. 12, and the state of the end effector 111 in the current task end state (i.e., current task goal) may be changed to an open state, thereby handing over the tool to the on-site operator (e.g., worker 104). The state of the task end state (i.e., task goal) that is changed may not necessarily be limited to the open or closed state of the end effector 111, but may also be a reduction in travel time or a change in height at the center position of the candidate task end state 300.

[0022] If it is determined in step 562 that the direction of the load is not the same as the candidate task end state with the highest score, then in step 565, the score of the candidate task end state with the highest score may be reduced. Then, in step 566, a loop process may be performed to process all other candidate task end states one by one, and in step 567, it may be determined whether the direction of the load is toward the other candidate task end state, and if the direction is toward the candidate task end state, in step 568, the score of the corresponding candidate task end state may be increased, otherwise, step 568 may be skipped, and after performing or skipping step 568, the next loop process is performed. If the loop process is completed, the process of step 560 ends, and the process shown in FIG. 5 proceeds to step 570. If the worker 104 extends his / her arm / hand to pick up the part 221 and contact the robotic device 101, a load is applied in the opposite direction to the candidate task end state 300, as shown by the dotted arrow in FIG. 12. In this case, only the process of step 565 is performed (e.g., the loop beginning with step 566 may be skipped based on the load being identified as being in a direction associated with another candidate task end state or task goal candidate), and the process of step 560 is repeated to result in a lower score for the candidate task end state 300 and possibly a (relatively) higher score for the next highest scoring candidate task end state 301, thereby changing the robotic device 101 from the candidate task end state 300 to the candidate task end state 301 by subsequent processing of steps 570, 530, and 540. In some aspects, the process described above with respect to the determination of step 562 that the direction of the load is not the same as the candidate task end state having the highest score (including the loop beginning with 566) may also be performed for loads applied in a direction opposite to the candidate task end state having the highest score.

[0023] In some aspects, an operator (e.g., worker 104) may intentionally attempt to redirect the robotically handled object 311 by applying a load in the same direction as the direction of a candidate task end state 303 (e.g., a candidate task end state that does not currently have the highest score), for example, as indicated by the dashed arrow in FIG. 12. In this case, in addition to the processing of step 565, the processing of step 567 may identify, for the candidate task end state 303, that the load direction is the same as the target direction, and may proceed to increase the score associated with the candidate task end state 303 in step 568 (e.g., step 568 is performed for the candidate task end state 303). The processing of step 560 (e.g., the loop beginning with step 566) may continue to completion, resulting in a higher score for the candidate task end state 303, and subsequent processing in steps 570, 530, and 540 may cause the robotic device 101 to change from the candidate task end state 300 to the candidate task end state 303.

[0024] In some aspects, the score assessment module 131 may combine the goal modification values ​​(e.g., values ​​used to modify scores associated with one or more candidate task end states or task goal candidates based on the load estimation associated with step 560) output by the load score calculation module 162 with the candidate task end states (i.e., task goal candidates) output by the goal planning module 120 in step 530, and may output a plurality of candidate task end states (task goal candidates) arranged in order of highest (or lowest) score. The path planning module 132 may generate a motion path from the current position to the goal state of the candidate task end state with the highest score (e.g., to update the motion path to be consistent with the candidate task end state with the highest score after updating the scores) in step 540. The path may be a series of positions and poses of the end effector 111 separated by a predetermined time interval, or may be a line segment describing the location through which the end effector 111 must pass. 5 may, in some embodiments, continue until it determines at step 570 that there is no change in the load situation, as described above, and then determine whether the current state of the robot is at the final state of the current or selected task end state (e.g., the current or selected task goal) at step 580. If it is determined that the movement is not complete, the process may return to step 550 and movement commands may be output to continue the movement, whereas if it is determined that the movement is complete, the process may be terminated.

[0025] As shown in FIG. 4, FIG. 5, FIG. 7, and FIG. 10, the goal planning module 120 (e.g., a goal planning algorithm or program) may generate a single set of candidate task end states (task goal candidates) that may be used for the duration of the task. For example, the goal planning module 120 may generate a set of candidate task end states (task goal candidates) including a set of target area candidates (e.g., the size / area, center, or other specific location therein, and shape of the target area candidates) and a set of candidate states of the robot. The generated set of tasks may then be used by the motion planning module 130 to select and / or update task goals throughout the duration of the task (e.g., based on feedback from the actuator sensing module 175 and / or the load estimation module 160). Thus, processing resources (e.g., processing cycles, power, etc.) or processing time associated with processing feedback and updating the task end states (i.e., task goals) may be reduced by updating weights associated with the generated set of candidate task end states (task goal candidates) without the need to generate a new set of candidate task end states (task goal candidates).

[0026] In this way, by changing the task end state (e.g., task goal or task target), according to the direction and magnitude of the load applied by the moving robot, the robot can instantly (i.e., quickly) adjust its movement in an infinite (i.e., changing or mutable) environment to suit the operator's work and behave in a way that does not hinder the collaborative operator, thereby improving the efficiency of collaborative work.

[0027] The present disclosure further provides a method for storing operator characteristics and modifying (ie, generating) motion targets (eg, candidate task end-states or candidate task goals) according to the operator characteristics.

[0028] FIG. 13 illustrates the software configuration of the robot task controller 108, and FIG. 14 illustrates how the control system can set robot goals according to the characteristics of the worker. The robot task controller 108 may further include a co-worker database 170 in some embodiments. The detection module 115 may identify individual workers 104 in some embodiments, and the co-worker database 170 may output personal data associated with the workers 104 to the goal planning module 120. The goal planning module 120 may use the personal data of the workers 104 when planning task goals (and / or generate candidate task end states or candidate task goals), and may reflect (i.e., take into account) the characteristics of the workers 104 when planning task goals. The motion planning module 130 may store information in the co-worker database 170 as personal data when a task goal is changed (e.g., as described in connection with steps 530-1700 of FIG. 5).

[0029] As shown in FIG. 14, the selection of the candidate task end state (task goal) without considering the characteristics of the worker with respect to the reference point 320 may select the candidate task end state 303 at the end of the dashed arrow. If the worker 104 has previously changed the current task end state (task goal) and the amount of change scaled by a predetermined magnitude of movement is the dashed arrow, the double arrow (indicating the difference between the reference point 320 and the updated reference point 321) combining the dashed and dashed arrows may be stored as personal data representing the characteristics of the worker. As opposed to a task end state (task goal) determined without considering the personal data stored about the worker 104 (e.g., the candidate task end state 303), the task end state (task goal) may be determined and / or set based on the personal data stored about the worker 104 in some aspects. For example, instead of performing the task goal determination using the reference point 320, the personal data stored about the worker 104 may lead to the determination of the task end state (task goal) being based on the updated reference point 321, which is different from the reference point 320. 14, the candidate task end states 302 may be set as task end states (task goals). In this way, the task end states (task goals) (e.g., selected and / or determined task end states or task goals) may be modified according to the characteristics of individual workers, allowing the robot to perform tasks tailored to individual workers, thereby further improving the efficiency of collaborative work.

[0030] In some embodiments of the present disclosure, the task end state (task goal) may be modified based on the pattern of the load. FIG. 15 is a list of example load patterns that may be used in some embodiments of the present disclosure to modify and / or update the task goal. For example, when the load score calculation module 162 detects the load pattern shown in FIG. 15, it may change the state of the task end state (task goal) (e.g., the selected task end state or task goal) to perform the corresponding action shown in FIG. 15. In some embodiments, when the load score calculation module 162 detects two quick pull down movements shown in FIG. 15, it may change the hand open / close state of the end effector 11 to open the hand at the site. In this way, the operator can receive the tool and module at the site. If a quick pull force is applied after a push against the direction of movement shown in FIG. 15 and / or if a force is applied in the direction of a different candidate task end state (task goal candidate), the score of the current task goal may be decreased and the score of the candidate task end state in the direction in which the force was applied may be increased. In this way, the worker can dynamically change the task end state (task goal) and achieve a work arrangement that is efficient for the worker. By changing the state of the task end state (task goal) according to a specific load pattern in this way, the worker can explicitly change the task end state (task goal) and specify a work arrangement that is more efficient for the worker, thereby improving the efficiency of the collaborative work.

[0031] Additional embodiments may include various combinations, modifications, or extensions of the above-described elements. For example, the robot arm 110 used in this illustrative example is shown as a vertical articulated robot, but can be a Cartesian coordinate robot, a horizontal articulated robot, a parallel link robot, etc. The robot device controller 107 and the robot task controller 108 are different controllers in this illustrative example, but can be configured to execute multiple programs within a single controller.

[0032] In addition, while FIG. 6 illustrates object detection using bounding boxes, other detection methods may be used, for example, the system, i.e., detection module 115, may be configured to use classification results from image segmentation. The task end states (task goals) or associated target areas shown in FIG. 9 may be illustrated using rectangles, but may be any shape determined by the work task (i.e., objective) associated with the task end states (task goals). The task end states or target areas may be of different sizes, and in some embodiments, different task end states or target areas may have overlapping regions. Although the task end states or target areas are illustrated in FIG. 9 as a set of discontinuous regions, the task end states or target areas may be associated with location information expressed as a reference to a (continuous) coordinate system (e.g., may be specified by a point in the coordinate system along one or more axes defined relative to the coordinate system, or by a range of values).

[0033] Although the scoring method described in connection with FIG. 9 is based on distance, other metrics may be used, alone or in combination, to calculate a score for each of the multiple candidate task end states (candidate task goal). For example, the system may be configured to simulate in advance the curvature of the movement path and the static loads due to the pose of the worker 104 and the robotic device 101 at the goal point, and the simulated and / or predicted loads may be used to calculate the score for the candidate task goal. Additionally or alternatively, the method of modifying the score associated with one or more candidate task end states (candidate task goal), the selected candidate task goal, and / or the state of the system may not be limited to the method shown in FIG. 5 and FIG. 11, and may further include consideration of different modification conditions, and the state may be modified in one or more of steps 530, 540, and / or 550 by modifying the path of movement, modifying the velocity command, or modifying the stiffness and elasticity settings in the force control.

[0034] FIG. 16 is a flow diagram 1600 illustrating a method according to some embodiments of the disclosure. In some embodiments, the method may be implemented by an apparatus (e.g., one of a collaborative robot system, a robot task controller 108, or a computing device 1805) for controlling a robotic device in an industrial environment. The apparatus may receive at least one input data set including one or more of task data for a first task for the collaborative robotic device or environmental data for at least a work area and a human operator associated with the first task. The at least one input data set may, in some embodiments, be related to the purpose of the first task. In some embodiments, the first task may be a subtask of a larger manufacturing or assembly task. For example, with respect to a larger goal of assembling a product, the first task may be defined as providing tools or parts to a human worker who will assemble the product at a particular point in the assembly process. Thus, the at least one input data set may include data regarding the objective of the first task, a reference point associated with the first task (e.g., reference point 310 or reference point 320), a tool or part associated with the first task, image data associated with the work area and a human operator (i.e., worker), personal (historical) data associated with a particular human operator, or other task data relevant to defining and / or generating candidate task end states (e.g., candidate task goals or candidate robot task targets). For example, with reference to Figures 5 and 7, an apparatus (e.g., robot task controller 108, detection module 115, and / or goal planning module 120) may capture images in step 500 and perform object recognition in step 510 or may receive task data as shown in Figure 7.

[0035] At 1620, the apparatus may generate a number of candidate task end states associated with the first task for the collaborative robotic device. In some aspects, the candidate task end states may be associated with target area candidates and states of the collaborative robotic device (e.g., states of the end effectors of the collaborative robotic device). For example, each candidate task end state for the first task may be associated with a target location for placing an object held by the collaborative robotic device (e.g., a target location and a release / release state of a component of the collaborative robotic device that holds the object). For example, with reference to FIGS. 5, 7, and 9, the apparatus (e.g., the robot task controller 108, the goal planning module 120, and / or the target candidate detection module 122) may output candidate task end states (task goal candidates) such as candidate task end states 300, 301, 302, and 303 to which the robot may move at step 520.

[0036] After generating the plurality of candidate task end states at 1620, in some aspects the apparatus may generate (e.g., calculate or compute) a score for each of the plurality of candidate task end states associated with the first task. In some aspects, the score for each of the candidate task end states associated with the first task may be generated based on a set of reference points and distances, or other characteristics. For example, with reference to FIGS. 5 and 7-9, the apparatus (e.g., the robot task controller 108, the goal planning module 120, and / or the distance score calculation module 123) may calculate a score associated with each of the generated candidate task end states (task goal candidates), such as the candidate task end states 300, 301, 302, and 303 to which the robot may travel, in connection with step 520 (e.g., steps 523-1205).

[0037] At 1640, the apparatus may select a first task end state from a plurality of candidate task end states for the collaborative robotic device. The first task end state may, in some aspects, be associated with a first target location for placing an object held by the collaborative robotic device. In some aspects, selecting the first task end state at 1640 may include selecting the first task end state associated with a best score, which may be the lowest score or the highest score depending on the method used to generate (i.e., calculate) the score for each candidate task end state. For example, with reference to Figures 5 and 7-9, in step 530, the apparatus (e.g., the robot task controller 108, the motion planning module 130, the score evaluation module 131, and / or the path planning module 132) may select (i.e., extract) the task end state (i.e., task goal) having the highest score from among the candidate task end states (e.g., the candidate task end states 300, 301, 302, and 303, or the task goal candidates) output by the goal planning module 120 in step 520 (e.g., step 526).

[0038] At 1650, the apparatus may receive load data regarding the load experienced by the collaborative robotic device. The load data may, in some embodiments, include magnitude and direction data associated with the load experienced by the collaborative robotic device. In some embodiments, the magnitude and direction data may be associated with one of a plurality of patterns (e.g., the magnitude and direction patterns discussed in connection with FIG. 15). Each of the plurality of patterns, in some embodiments, may be associated with at least one value associated with a corresponding at least one of a plurality of candidate task end states. The at least one value, in some embodiments, may be a value that is added or subtracted from a score or value with which to multiply or divide the score associated with the corresponding at least one of the plurality of candidate task end states. For example, with reference to Figures 5 and 10-12, an apparatus (e.g., the robot task controller 108, the load estimation module 160, and / or the load vector calculation module 161) may collect and / or receive sensor data regarding the robotic device 101 obtained by the robotic device controller 107 (such as sensor data reflecting one of the force vectors shown in Figure 12) in step 560.

[0039] The apparatus may additionally or alternatively receive 1650 at least one additional input data set including updated environmental data regarding at least one of the work area and / or human operator associated with the first task (and multiple candidate task end states). The additional input data set may include data regarding changes to the position and / or orientation of the human operator or changes to one or more objects in the work area (e.g., a product to be assembled, a tool, a part, etc.). The additional input data set may be used to update one or more locations (e.g., reference points) associated with the generation and / or calculation of the score. For example, if the score is based at least in part on a distance between a target area associated with a candidate task end state and one of the location of a right shoulder of a human worker or the location of a component of a product to be assembled (e.g., a point on the product where a current part is attached or a current tool is used), the locations used for subsequent score calculations may be based on the updated locations indicated and / or included in the additional input data set.

[0040] The apparatus may generate an updated score for each of a plurality of candidate task end states associated with the first task based on the load data. In some aspects, the generation of the updated score may be based on at least one value associated with the load data (e.g., magnitude and direction data associated with one of the plurality of patterns). In some aspects, the apparatus may generate the updated score based on an additional input data set (e.g., based on an updated position associated with the score calculation and indicated and / or included in the additional input data set). For example, with reference to FIGS. 5 and 10-12, the apparatus (e.g., the robot task controller 108, the load estimation module 160, and / or the load score calculation module 162) may generate (i.e., output) at step 560 (e.g., via steps 565-568) a revised score value for each candidate task end state and a revised state of the candidate task end states (task goal candidates) to the motion planning module 130 as a goal revision value before returning to step 530.

[0041] At 1670, the device may select a second task end state from a plurality of candidate task end states based on the load data. In some aspects, the second task end state may be associated with a second target location for placing the object. The second task end state, in some aspects, may be associated with releasing the object at the current location. To select the second task end state at 1670, in some aspects, the device may select the second task end state associated with the best (e.g., highest or lowest) updated score (after receiving the load data at 1650 and generating the updated scores at 1660). For example, referring to Figures 5 and 7-9, the apparatus (e.g., the robot task controller 108, the motion planning module 130, the score evaluation module 131, and / or the path planning module 132) may return to step 530 and select (i.e., extract) the task end state (task goal) having the updated (i.e., current) highest score from among the candidate task end states 300, 301, 302, and 303 (i.e., task goal candidates) output by the goal planning module 120 in step 520 (e.g., step 526).

[0042] FIG. 17 is a flow diagram 1700 illustrating a method according to some embodiments of the disclosure. In some embodiments, the method may be implemented by an apparatus (e.g., one of a collaborative robot system, a robot task controller 108, or a computing device 1805) for controlling a robotic device in an industrial environment. At 1710, the apparatus may receive at least one input data set including one or more of task data for a first task for the collaborative robotic device or environmental data for at least a work area and a human operator associated with the first task. The at least one input data set may, in some embodiments, be related to the purpose of the first task. In some embodiments, the first task may be a subtask of a larger manufacturing task. For example, with respect to a larger goal of assembling a product, the first task may be defined as providing tools or parts to a human worker who will assemble the product at a particular point in the assembly process. Thus, the at least one input data set may include data regarding the objective of the first task, a reference point associated with the first task (e.g., reference point 310 or reference point 320), a tool or part associated with the first task, image data associated with the work area and a human operator (i.e., worker), personal (historical) data associated with a particular human operator, or other task data relevant to defining and / or generating candidate task end states (e.g., candidate task goals or candidate robot task targets). For example, with reference to Figures 5 and 7, an apparatus (e.g., robot task controller 108, detection module 115, and / or goal planning module 120) may capture images in step 500 and perform object recognition in step 510 or may receive task data as shown in Figure 7.

[0043] At 1720, the apparatus may generate a number of candidate task end states associated with the first task for the collaborative robotic device. In some aspects, the candidate task end states may be associated with target area candidates and states of the collaborative robotic device (e.g., states of the end effectors of the collaborative robotic device). For example, each candidate task end state for the first task may be associated with a target location for placing an object held by the collaborative robotic device (e.g., a target location of a component of the collaborative robotic device that holds the object, and a release / release state). For example, with reference to FIGS. 5, 7, and 9, the apparatus (e.g., the robot task controller 108, the goal planning module 120, and / or the target candidate detection module 122) may output candidate task end states (candidate task goals) such as candidate task end states 300, 301, 302, and 303 to which the robot may move at step 520.

[0044] After generating the plurality of candidate task end states at 1720, in some aspects the apparatus may generate (e.g., calculate or compute) a score for each of the plurality of candidate task end states associated with the first task at 1730. In some aspects, the score for each of the candidate task end states associated with the first task may be generated based on a set of reference points and distances, or other characteristics. For example, with reference to FIGS. 5 and 7-9, the apparatus (e.g., the robot task controller 108, the goal planning module 120, and / or the distance score calculation module 123) may calculate a score associated with each of the generated candidate task end states (task goal candidates), such as the candidate task end states 300, 301, 302, and 303 to which the robot may travel, in connection with step 520 (e.g., steps 523-1205).

[0045] At 1740, the apparatus may select a first task end state from a plurality of candidate task end states for the collaborative robotic device. The first task end state may, in some aspects, be associated with a first target location for placing an object held by the collaborative robotic device. In some aspects, selecting the first task end state at 1740 may include selecting the first task end state associated with a best score, which may be the lowest score or the highest score depending on the method used to generate (i.e., calculate) the score at 1730. For example, with reference to Figures 5 and 7-9, in step 530, the apparatus (e.g., the robot task controller 108, the motion planning module 130, the score evaluation module 131, and / or the path planning module 132) may select (i.e., extract) the task end state (i.e., task goal) having the highest score from among the candidate task end states (e.g., the candidate task end states 300, 301, 302, and 303, or the task goal candidates) output by the goal planning module 120 in step 520 (e.g., step 526).

[0046] At 1750, the apparatus may receive load data regarding the load experienced by the collaborative robotic device. The load data may, in some embodiments, include magnitude and direction data associated with the load experienced by the collaborative robotic device. In some embodiments, the magnitude and direction data may be associated with one of a plurality of patterns (e.g., the magnitude and direction patterns discussed in connection with FIG. 15). Each of the plurality of patterns, in some embodiments, may be associated with at least one value associated with a corresponding at least one of a plurality of candidate task end states. The at least one value, in some embodiments, may be a value that is added or subtracted from a score or value with which to multiply or divide the score associated with the corresponding at least one of the plurality of candidate task end states. For example, with reference to Figures 5 and 10-12, an apparatus (e.g., the robot task controller 108, the load estimation module 160, and / or the load vector calculation module 161) may collect and / or receive sensor data regarding the robotic device 101 obtained by the robotic device controller 107 (such as sensor data reflecting one of the force vectors shown in Figure 12) in step 560.

[0047] The apparatus may additionally or alternatively receive 1750 at least one additional input data set including updated environmental data regarding at least one of the work area and / or human operator associated with the first task (and multiple candidate task end states). The additional input data set may include data regarding changes to the position and / or orientation of the human operator or changes to one or more objects in the work area (e.g., a product to be assembled, a tool, a part, etc.). The additional input data set may be used to update one or more locations (e.g., reference points) associated with the generation and / or calculation of the score. For example, if the score is based at least in part on a distance between a target area associated with a candidate task end state and one of the location of a right shoulder of a human worker or the location of a component of a product to be assembled (e.g., a point on the product where a current part is attached or a current tool is used), the locations used for subsequent score calculations may be based on the updated locations indicated and / or included in the additional input data set.

[0048] At 1760, the device may generate an updated score for each of a plurality of candidate task end states associated with the first task based on the load data. In some aspects, the generation of the updated score at 1760 may be based on at least one value associated with the load data (e.g., magnitude and direction data associated with one of the plurality of patterns). In some aspects, the device may generate the updated score at 1760 based on an additional input data set (e.g., based on an updated position associated with the score calculation and indicated and / or included in the additional input data set). For example, with reference to FIGS. 5 and 10-12, the device (e.g., the robot task controller 108, the load estimation module 160, and / or the load score calculation module 162) may generate (i.e., output) a revised score value for each candidate task end state and a revised state of the candidate task end states (task goal candidates) to the motion planning module 130 as a goal revision value before returning to step 530 (e.g., via steps 565-568).

[0049] At 1770, the device may select a second task end state from a plurality of candidate task end states based on the load data. In some aspects, the second task end state may be associated with a second target location for placing the object. The second task end state, in some aspects, may be associated with releasing the object at the current location. To select the second task end state at 1770, in some aspects, the device may select the second task end state associated with the best updated score (after receiving the load data at 1750 and generating the updated score at 1760). For example, referring to Figures 5 and 7-9, the apparatus (e.g., the robot task controller 108, the motion planning module 130, the score evaluation module 131, and / or the path planning module 132) may return to step 530 and select (i.e., extract) the task end state (task goal) having the updated (i.e., current) highest score from among the candidate task end states 300, 301, 302, and 303 (i.e., task goal candidates) output by the goal planning module 120 in step 520 (e.g., step 526).

[0050] As mentioned above, the exemplary implementations described herein include an innovative collaborative robotic (i.e., robot) system that performs a highly efficient collaborative task between the collaborative robot system and the collaborative operator by appropriately responding to the actions of the collaborative operator (e.g., a human worker) and continuing the collaborative task without interruption, as well as a method of providing the same. In some embodiments, the proposed system improves the efficiency of the collaborative task by changing the task goal according to the changing conditions (e.g., the direction of the load applied to the robot) when the worker and the robot are moving simultaneously to perform the collaborative task, so that the worker can behave without interrupting the task and the collaborative robot system can behave without being an obstacle to the collaborative operator and / or the worker. The exemplary implementations discussed herein may improve the efficiency of the collaborative task by setting multiple task targets for the robot in advance and changing the task target without interrupting the operation. In addition, in some embodiments, the setting of the robot's task goal based on the reference point and the sensing result according to the task content enable the exemplary implementations to optimize each task of a process in which multiple tasks are performed. The system, in some aspects of the present disclosure, may achieve efficient operations for each task and streamline the work of processes across multiple processes.

[0051] 18 illustrates an example computing environment having an example computing device suitable for use in some example implementations. The computing device 1805 of the computing environment 1800 can include one or more processing units, cores, or processors 1810, memory 1815 (e.g., RAM, ROM, and / or other), internal storage 1820 (e.g., magnetic, optical, solid-state storage, and / or organic), and / or IO interface 1825, any of which can be coupled by a communication mechanism or bus 1830 for communicating information or incorporated into the computing device 1805. The IO interface 1825 is also configured to receive images from a camera or provide images to a projector or display, depending on the desired implementation.

[0052] The computing device 1805 can be communicatively coupled to an input / user interface 1835 and an output device / interface 1840. Either or both of the input / user interface 1835 and the output device / interface 1840 can be wired or wireless interfaces and can be removable. The input / user interface 1835 can include any device, component, sensor, or interface, physical or virtual, that can be used to provide input (e.g., buttons, touch screen interfaces, keyboards, pointing / cursor control, microphones, cameras, Braille, motion sensors, accelerometers, optical readers, and / or others). The output device / interface 1840 can include displays, televisions, monitors, printers, speakers, Braille, and the like. In some example implementations, the input / user interface 1835 and the output device / interface 1840 can be embedded in or physically coupled to the computing device 1805. In other example implementations, other computing devices may function as or provide the functionality of the input / user interface 1835 and the output device / interface 1840 of the computing device 1805 .

[0053] Examples of computing devices 1805 may include, but are not limited to, highly mobile devices (e.g., smart phones, automobiles or other mechanical devices, devices carried by people and animals, etc.), mobile devices (e.g., tablets, notebooks, laptops, personal computers, portable televisions, radios, etc.), and devices not designed for mobility (e.g., desktop computers, other computers, information kiosks, televisions with one or more processors embedded and / or combined televisions, radios, etc.).

[0054] Computing device 1805 can be communicatively coupled (e.g., via IO interface 1825) to external storage 1845 and a network 1850 for communicating with any number of networked components, devices, and systems, including one or more computing devices of the same or different configurations. Computing device 1805, or any connected computing device, may function, provide services, or be referred to as a server, a client, a thin server, a general purpose machine, a special purpose machine, or another label.

[0055] The IO interface 1825 can include, but is not limited to, wired and / or wireless interfaces using any communication or IO protocol or standard (e.g., Ethernet, 1802.11x, Universal System Bus, WiMax, modem, cellular network protocols, etc.) to communicate information to and from at least all connected components, devices, and networks of the computing environment 1800. The network 1850 can be any network or combination of networks (e.g., the Internet, a local area network, a wide area network, a telephone network, a cellular network, a satellite network, etc.).

[0056] The computing device 1805 can use and / or communicate using computer usable or computer readable media, including transitory and non-transitory media. Transitory media include transmission media (e.g., metallic cables, optical fibers), signals, carrier waves, and the like. Non-transitory media include magnetic media (e.g., disks and tapes), optical media (e.g., CD ROM, digital video disks, Blu-ray disks), solid media (e.g., RAM, ROM, flash memory, solid-state storage), and other non-volatile storage or memory.

[0057] The computing device 1805 can be used to implement techniques, methods, applications, processes, or computer-executable instructions in some example computing environments. The computer-executable instructions can be retrieved from a transitory medium and can be stored and retrieved from a non-transitory medium. The executable instructions can be in one or more of any programming, scripting, and machine language (e.g., C, C++, C#, Java, Visual Basic, Python, Perl, JavaScript, etc.).

[0058] The processors 1810 can run individually or in any combination, in a native or virtual environment, under any operating system (OS) (not shown). One or more applications can be deployed, including a logic unit 1860, an application programming interface (API) unit 1865, an input unit 1870, an output unit 1875, and an inter-unit communication mechanism 1895 for different units to communicate with each other, with the OS, and with other applications (not shown). The described units and elements can vary in design, function, configuration, or implementation, and are not limited to the description provided. The processor 1810 can be in the form of a hardware processor, such as a central processing unit (CPU), or a combination of hardware and software units.

[0059] In some example implementations, when information or instructions to execute are received by the API unit 1865, they may be communicated to one or more other units (e.g., logic unit 1860, input unit 1870, output unit 1875). In some examples, logic unit 1860 may be configured to control information flow between units and direct services provided by API unit 1865, input unit 1870, output unit 1875 in some example implementations described above. For example, the flow of one or more processes or implementations may be controlled by logic unit 1860 alone or in combination with API unit 1865. Input unit 1870 may be configured to obtain inputs for calculations described in example implementations, and output unit 1875 may be configured to provide outputs based on calculations described in example implementations.

[0060] The processor 1810 can be configured to generate, individually or in any combination, for the collaborative robotic device, a plurality of candidate task end states associated with the first task. The processor 1810 can be configured to select, individually or in any combination, for the collaborative robotic device, a first task end state from the plurality of candidate task end states. The processor 1810 can be configured to receive, individually or in any combination, load data regarding loads experienced by the collaborative robotic device. The processor 1810 can be configured to select, individually or in any combination, a second task end state from the plurality of candidate task end states based on the load data.

[0061] The processor 1810 may also be configured to generate a score for each of a plurality of candidate task end states associated with the first task, individually or in any combination. The processor 1810 may also be configured to select the first task end state associated with the best score, individually or in any combination. The processor 1810 may also be configured to generate an updated score for each of a plurality of candidate task end states associated with the first task based on the load data, individually or in any combination. The processor 1810 may also be configured to select the second task end state associated with the best updated score, individually or in any combination. The processor 1810 may also be configured to generate an updated score based on at least one value associated with a pattern, individually or in any combination. The processor 1810 may also be configured to receive at least one input data set including one or more of task data associated with the task associated with the plurality of candidate task end states, individually or in any combination, or environmental data associated with at least the work area and the human operator. The processor 1810 may also be configured to generate a plurality of candidate task end states based on the first input data set of the at least one input data set, individually or in any combination.

[0062] Some portions of the detailed description are presented in terms of algorithms and symbolic representations of operations within a computer. These algorithmic descriptions and symbolic representations are the means used by those skilled in the data processing arts to convey the substance of their innovations to others skilled in the art. An algorithm is a sequence of predefined steps leading to a desired end state or result. In an exemplary implementation, the steps performed require physical manipulations of tangible quantities to achieve a tangible result.

[0063] Unless otherwise specifically indicated, and as will be apparent from the discussion, it will be recognized that throughout the description, discussion utilizing terms such as "processing," "computing," "calculating," "determining," "displaying," and the like can include operations and processes of a computer system or other information processing device that manipulate and convert data represented as physical (electronic) quantities in the computer system's registers and memory into other data similarly represented as physical quantities in the computer system's memory or registers or other information storage, transmission, or display device.

[0064] An exemplary implementation may also relate to an apparatus for performing the operations of the present specification. The apparatus may be specially constructed for the required purposes or may include one or more general-purpose computers selectively activated or reconfigured by one or more computer programs. Such computer programs may be stored on a computer-readable medium, such as a computer-readable storage medium or a computer-readable signal medium. A computer-readable storage medium may include tangible media, such as optical disks, magnetic disks, read-only memory, random access memory, solid-state devices and drives, or any other type of tangible or non-transitory medium suitable for storing electronic information. A computer-readable signal medium may include media such as carrier waves. The algorithms and displays presented herein are not inherently related to any particular computer or other apparatus. A computer program may include a pure software implementation, including instructions for performing the operations of a desired implementation.

[0065] Various general-purpose systems may be used with the programs and modules according to the examples herein, or it may prove useful to construct more specialized apparatus to perform the desired method steps. Additionally, the example implementations are not described with reference to any particular programming language. It will be appreciated that various programming languages ​​may be used to implement the teachings of the example implementations as described herein. Instructions in the programming language may be executed by one or more processing devices, such as a central processing unit (CPU), processor, or controller.

[0066] As known in the art, the operations described above may be implemented by hardware, software, or some combination of software and hardware. Various aspects of the implementations may be implemented using circuits and logic devices (hardware), while other aspects may be implemented using instructions stored on a machine-readable medium (software) that, when executed by a processor, will cause the processor to perform a method for implementing an implementation of the present application. Furthermore, some exemplary implementations of the present application may be implemented exclusively by hardware, while other exemplary implementations may be implemented exclusively by software. Furthermore, the various functions described may be implemented in a single unit or may be spread across multiple components in various manners. When implemented by software, the method may be executed by a processor, such as a general-purpose computer, based on instructions stored on a machine-readable medium. If desired, the instructions may be stored on the medium in compressed and / or encrypted form.

[0067] Moreover, other implementations of the present application will be apparent to those skilled in the art from consideration of the specification and practice of the teachings of the present application. Various aspects and / or components of the described example implementations may be used alone or in any combination. It is intended that the specification and example implementations be considered as examples only, with a true scope and spirit of the present application being indicated by the following claims.

Claims

1. At least one memory; at least one processor coupled to said at least one memory; and wherein based at least in part on information stored in the at least one memory, the at least one processor, individually or in any combination: generating a plurality of candidate task end states associated with the first task for the collaborative robotic device; selecting a first task end state from the plurality of candidate task end states for the collaborative robotic device; receiving load data relating to a load experienced by the collaborative robotic device; Selecting a second task end state from the plurality of candidate task end states based on the load data. A control system configured as follows.

2. The at least one processor further comprises: configured to generate a score for each of the plurality of candidate task end states associated with the first task, and for selecting the first task end state, the at least one processor configured to select the first task end state associated with a best score. The control system of claim 1 .

3. The at least one processor further comprises: configured to generate an updated score for each of the plurality of candidate task end states associated with the first task based on the load data, and for selecting the second task end state, the at least one processor configured to select the second task end state associated with a lowest updated score. The control system of claim 2.

4. The control system of claim 3 , wherein the load data includes magnitude and direction data associated with the load experienced by the collaborative robotic device.

5. 5. The control system of claim 4, wherein the magnitude data and the direction data are associated with one of a plurality of patterns, each of the plurality of patterns being associated with at least one value associated with a corresponding at least one of the plurality of candidate task end states, and wherein for generating the updated score, the at least one processor is further configured to generate the updated score based on the at least one value associated with the one of the patterns.

6. The control system of claim 2 , wherein the first task end state is associated with a first target location for placing an object held by the collaborative robotic device.

7. The control system of claim 6 , wherein the second task end state is associated with a second target location for placing the object.

8. The control system of claim 6 , wherein the second task end state is associated with releasing the object at a current position.

9. The at least one processor further comprises: and configured to receive at least one input data set including one or more of task data related to a task associated with the plurality of candidate task end states, or environmental data related to at least a work area and a human operator. The control system of claim 1 .

10. The at least one processor further comprises: configured to generate the plurality of candidate task end states based on a first input data set of the at least one input data set.

10. The control system of claim 9.

11. 11. The control system of claim 10, wherein the environmental data includes image data associated with the work area and the human operator, and wherein for selecting the second task end state from the plurality of candidate task end states, the at least one processor is further configured to select the second task end state based on a second input data set of the at least one input data set that includes the environmental data.

12. generating a plurality of candidate task end states associated with the first task for the collaborative robotic device; selecting a first task end state from the plurality of candidate task end states for the collaborative robotic device; receiving load data relating to a load experienced by the collaborative robotic device; selecting a second task end state from the plurality of candidate task end states based on the load data; 16. A method for controlling a collaborative robotic device, comprising:

13. generating a score for each of the plurality of candidate task end states associated with the first task, wherein selecting the first task end state includes selecting the first task end state associated with a best score. The method of claim 12.

14. generating an updated score for each of the plurality of candidate task end states associated with the first task based on the load data, and wherein selecting the second task end state includes selecting the second task end state associated with a best updated score. The method of claim 13.

15. 15. The method of claim 14, wherein the load data includes magnitude data and directional data associated with the load experienced by the collaborative load device, the magnitude data and the directional data are associated with one of a plurality of patterns, each of the plurality of patterns being associated with at least one value associated with a corresponding at least one of the plurality of candidate task end states, and generating the updated score includes generating the updated score based on the at least one value associated with the one of the patterns.

16. 14. The method of claim 13, wherein the first task end state is associated with a first target location for placing an object held by the collaborative robotic device, and the second task end state is associated with a second target location for placing the object.

17. The method of claim 16 , wherein the second task end state is associated with releasing the object at a current position.

18. receiving at least one input data set including one or more of task data related to tasks associated with the plurality of candidate task end states, or environmental data related to at least a work area and a human operator; The method of claim 12 further comprising:

19. generating the plurality of candidate task end states based on a first input data set of the at least one input data set; 20. The method of claim 18, further comprising:

20. 20. The method of claim 19, wherein the environmental data includes image data associated with the work area and the human operator, and selecting the second task end state from the plurality of candidate task end states is further based on a second input data set of the at least one input data set that includes the environmental data.

Images