Robot control system, robot control method, and program

Abstract

To provide an innovative cooperative robotic (or robot) system to perform, and a method to provide, highly efficient cooperative work between the cooperative robotic system and a cooperative operator.SOLUTION: There are provided systems and methods for controlling a cooperative robotic device including generating, for the cooperative robotic device, a plurality of candidate task end states associated with a first task. The method may further include: selecting, for the cooperative robotic device, a first task end state from the plurality of candidate task end states; receiving load data regarding a load experienced by the cooperative robotic device; and selecting, based on the load data, a second task end state from the plurality of candidate task end states.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure is generally directed to collaborative robotic systems. [Background technology]

[0002] In the manufacturing and construction sectors, attention is being focused on high-mix, low-volume production and mass customization, which differ from traditional mass production methods in order to meet diversifying customer needs. These methods require frequent adjustments and changes to production volume and processes to meet diverse needs, and companies are therefore seeking personnel (multi-skilled workers) who can handle the complex array of processes required to deal with such situations. Meanwhile, labor shortages have become more severe in recent years, making it more difficult to secure human resources. Therefore, there is a need for more advanced automation and labor-saving measures using robots, AI, and other digital tools to replace labor.

[0003] When introducing robots into actual workplaces, it is difficult for robots to completely take over human tasks at the current level of technology. However, there are cases where robots can take over object transportation and other relatively simple tasks, allowing humans to focus on more complex tasks. In these cases, humans and robots may work more closely together. Traditionally, robots have been used within ranges or areas surrounded by safety fences to prevent humans from approaching the robot while it is operating. However, in recent years, collaborative robots have become more common, and there are many use cases in which robots operate without safety fences and in positions where they may come into contact with humans. However, although current collaborative robots have enhanced safety mechanisms to prevent injury when they come into contact with humans, there are few actual examples of collaborative robots working simultaneously with humans. In fact, there are robot systems that work in the same space as humans, but most of them work sequentially with humans, and there are few examples of working together at the same time and space, such as when a robot moves and works simultaneously with an associated human worker.

[0004] When a robot collaborates with a human worker, especially when the human and the robot operate simultaneously, the human may take unexpected actions. In this case, existing robots may be configured to execute pre-planned actions and not modify their own actions in response to unexpected actions taken by the collaborative operator while moving. Therefore, the robot does not respond to the collaborative operator's actions, and the robot's actions may be inappropriate for the collaborative task, which may interrupt the task and reduce the efficiency of the collaborative task. For example, when the collaborative operator reaches out to pick up a part or tool, the operator's hand or arm may be located in the robot's path, causing the robot to come into contact with the human operator, which may interrupt the task. If the collaborative operator's body position is measured to detect whether the robot is in the robot's path, and if so, the robot's motion is re-planned, and the task is interrupted during the re-planning process, thereby reducing the efficiency of the collaborative task. Some robotic systems configure the robot to stop upon contact with a human worker and then continue (i.e., begin a new action / task) based on a specific force pattern applied by the human worker, but such systems may be less user-friendly or may reduce the efficiency of collaborative tasks because they require the human worker to perform a specific action (e.g., apply a specific force pattern) which may require additional time after stopping.

[0005] For example, because conventional robots repeatedly move only to memorized positions, robotic devices may be configured to perform tasks using motion patterns that specify positions according to the work content. However, in tasks such as collaborative work with humans (especially for high-mix, low-volume production and / or mass customization), the work content may change irregularly, the work object may not be fixed, and the position of the work object may change frequently, i.e., constantly, as it is manipulated by the collaborative human worker. Specifically, in collaborative work with human workers, it may be impossible to achieve a specific objective (e.g., associated with a predefined goal or task) by repeatedly moving to a specified set of positions. Therefore, a collaborative robot system must be able to use environmental measurement devices, such as cameras or distance measurement devices (e.g., RADAR or LIDAR), that recognize the situation around the robot and collaborative operator, and / or sensors that detect external forces due to contact between the human and the robot, and generate an updated goal or task and an associated set of instructions for the robot based on the situation around the robot and collaborative operator and / or the forces due to contact between the human and the robot. However, some previous robotic systems must perform a full reevaluation of the goal each time a change to the situation or a force due to human contact is detected and / or in response to additional human input before resuming or continuing operation. Thus, a collaborative robotic system that can respond to each detected change to the situation or a force due to human contact without a full reevaluation can enable more efficient collaboration between a human worker and the collaborative robotic system. Summary of the Invention

[0006] Example implementations described herein include an innovative collaborative robotic (i.e., robot) system that performs highly efficient collaborative work between the collaborative robot system and a collaborative operator by appropriately responding to the actions of the collaborative operator (e.g., a human worker) and continuing the collaborative work without interruption, as well as a method for providing such efficient collaborative work. In some embodiments, the collaborative robot system may include a goal planning module that generates multiple robot task targets (e.g., candidate task goals or candidate task end states associated with one or more work tasks or objectives) for a robot of the collaborative robot system before the robot performs a specific task among the multiple robot task targets. In some embodiments, the collaborative robot system may also include a load estimation module that detects the robot's load situation (e.g., the magnitude and direction of current and / or historical loads applied to the robot). In some embodiments, the collaborative robot system may further include a motion planning module that selects one of the robot task targets based on the robot's load situation and plans a robot motion associated with the selected robot task target. In some embodiments, the goal planning module determines multiple candidate goals (goal states) according to the task content during preliminary motion planning and scores each candidate goal. The load estimation module may detect the load state of the robot while the robot executes a motion associated with a selected robot task target. Based on the detected load state, the motion planning module may update the robot task target selection to change the robot task target or select one of multiple robot task targets (associated with one or more work tasks or objectives) based on scores calculated for or associated with the multiple robot task targets (e.g., candidate task end states associated with one or more work tasks or objectives) and the detected load state (i.e., situation).The motion planning module, in some aspects, may generate an instruction set for executing a motion corresponding to the updated (newly selected) robot task target selection and provide the instruction set to a robot of the collaborative robot system.

[0007] Aspects of the present disclosure include a method of controlling a collaborative robotic device that includes generating, for the collaborative robotic device, a plurality of candidate task end states associated with a first task. The method may further include, for the collaborative robotic device, selecting a first task end state from the plurality of candidate task end states, receiving load data related to loads experienced by the collaborative robotic device, and selecting a second task end state from the plurality of candidate task end states based on the load data.

[0008] Aspects of the present disclosure include a non-transitory computer-readable medium storing instructions for execution by a processor, which may include instructions for controlling a collaborative robotic device, including instructions for generating, for the collaborative robotic device, a plurality of candidate task end states associated with a first task. The instructions may further include instructions for the collaborative robotic device to select a first task end state from the plurality of candidate task end states, receive load data regarding loads experienced by the collaborative robotic device, and select a second task end state from the plurality of candidate task end states based on the load data.

[0009] Aspects of the present disclosure include a system that may include means for controlling a collaborative robotic device, including means for generating, for the collaborative robotic device, a plurality of candidate task end states associated with a first task. The system may further include means for the collaborative robotic device to select a first task end state from the plurality of candidate task end states, means for receiving load data related to loads experienced by the collaborative robotic device, and means for selecting a second task end state from the plurality of candidate task end states based on the load data.

[0010] Aspects of the present disclosure include an apparatus that may include a memory and at least one processor coupled to the at least one memory, where the at least one processor may be configured to generate, individually or in any combination, a plurality of candidate task end states associated with a first task for a collaborative robotic device based at least in part on information stored in the at least one memory. The at least one processor may be further configured to select, for the collaborative robotic device, a first task end state from the plurality of candidate task end states, receive load data regarding loads experienced by the collaborative robotic device, and select, based on the load data, a second task end state from the plurality of candidate task end states. [Brief explanation of the drawings]

[0011] [Figure 1] FIG. 1 illustrates components associated with a collaborative robot system, according to some aspects of the present disclosure. [Figure 2] FIG. 1 illustrates component elements of a collaborative robot system according to some aspects of the present disclosure. [Figure 3] FIG. 1 illustrates a robotic device and a worker assembling a work object placed on a workbench or transported by an autonomous vehicle. [Figure 4] FIG. 2 is a diagram illustrating a software configuration of a robot task controller. [Figure 5] 1 is a flowchart illustrating the overall process the robot control system uses to set robot task targets and execute robot movements. [Figure 6] FIG. 10 is a diagram illustrating an example of an object recognition result from a detection module. [Figure 7] FIG. 2 is a schematic block diagram showing a detailed software configuration of a goal planning module. [Figure 8] 6 is a flowchart illustrating a detailed process of one step in FIG. 5. [Figure 9] FIG. 10 is a diagram illustrating an example of a goal planning module that calculates candidate goal states. [Figure 10] FIG. 2 is a schematic block diagram showing the detailed software configuration of a motion planning module and a load estimation module. [Figure 11] 6 is a flowchart illustrating a detailed process of the steps in FIG. 5. [Figure 12] FIG. 10 shows an example illustrating how the motion planning module and the load estimation module estimate the load state and modify the task goal according to the load state. [Figure 13] FIG. 2 is a diagram illustrating a software configuration of a robot task controller. [Figure 14] FIG. 10 illustrates how the control system can set robot goals according to worker characteristics. [Figure 15] FIG. 10 illustrates a list of example load patterns that may be used in some aspects of the present disclosure to modify and / or update task goals. [Figure 16] FIG. 1 is a flow diagram illustrating a method according to some aspects of the present disclosure. [Figure 17] FIG. 1 is a flow diagram illustrating a method according to some aspects of the present disclosure. [Figure 18] FIG. 1 illustrates an example computing environment having example computing devices suitable for use in some example implementations. DETAILED DESCRIPTION OF THE INVENTION

[0012] The following detailed description provides details of the drawings and exemplary implementations of the present application. Reference numbers and descriptions of elements that are duplicated between drawings are omitted for clarity. Terms used throughout the description are provided by way of example and are not intended to be limiting. For example, the use of the term "automatic" can include a fully automatic implementation or a semi-automatic implementation with user or administrator control over certain aspects of the implementation, depending on the desired implementation of those skilled in the art practicing the implementations of the present application. Selection can be performed by a user through a user interface or other input means, or can be achieved through a desired algorithm. The exemplary implementations as described herein can be utilized alone or in combination, and the functionality of the exemplary implementations can be achieved through any means according to the desired implementation.

[0013] 1 illustrates components associated with a collaborative robot system according to some embodiments of the present disclosure. FIG. 2 illustrates components of a collaborative robot system according to some embodiments of the present disclosure.

[0014] In some embodiments, the robot device 101 may have a mobile cart 9, a robot arm 110 attached to the mobile cart 9, and an end effector 111 attached to the tip of the robot arm 110, and may perform a task on a work object 102 on a work table 103. Each device of the robot device 101 may be connected to a robot device controller 107 and operates based on control commands (such as a motor current of the robot arm 110 and a motor current of the end effector 111) received from the robot device controller 107. The robot device 101 (i.e., a component device) may also transmit a state of the robot device 101 (e.g., a voltage of an angle sensor attached to a joint of the robot arm 110, or other measurement data) to the robot device controller 107. The robotic device controller 107 may be connected to the robotic task controller 108 via a network 106 (e.g., a wired or wireless network) and may convert (e.g., interpret and / or format information about) the state of the robotic device 101 obtained from the robotic device 101 (e.g., joint angles of the robotic arm 110, hand positions of the end effector 111, etc.) and provide the state information to the robotic task controller 108. In some embodiments, the robotic device controller 107 may also calculate control commands for the robotic device 101 based on motion commands (e.g., target positions of the end effector 111) output from the robotic task controller 108 and the state of the robotic device 101 input from the robotic device 101. The camera 112 (e.g., as an example of an environmental measurement device that recognizes the situation around the robot and the collaborative operator) may be configured to measure the distance (depth) from the camera along with a color image and may be connected to the robotic task controller 108 via the network 106. The camera 112 may provide one or more of the captured data and / or depth information, along with the captured images related to the work object 102 and the robotic device 101, to the robotic task controller 108 via the network 106.

[0015] FIG. 2 illustrates that the robotic device controller 107 may, in some embodiments, be a computer including a processing unit 171, a network interface 172, a device interface 173, and data storage 174, which are electrically connected (e.g., by a bus or other communication mechanism, as described below in connection with FIG. 18 ). The processing unit 171 may include a central processing unit (CPU), random access memory (RAM), read-only memory (ROM), etc., and may be configured to perform processing based on programs and various data. The data storage 174 may be an auxiliary memory device such as a hard disk drive, and may store a control program 751 executed by the processing unit 171. The device interface 173 may be an interface that connects to the robotic device 101, transmits control commands to the robotic device 101, and receives data related to the state of the robotic device 101, and may be configured appropriately for the equipment that includes the robotic device 101. The network interface 172 may be an interface that connects to the robotic task controller 108 and receives motion commands for the robotic device 101 (e.g., via communication with the network interface 182 of the robotic task controller 108 over the network 106) and transmits (provides) data regarding the state of the robotic device 101. When the robotic device controller 107 is started, such as by being powered on, a control program 751 stored in the data storage 174 may be provided to (or accessed by) and executed by the processing unit 171. The control program 751 generates control commands for the robotic device 101 based on the motion commands received from the robotic task controller 108 via the network interface 172 and the state of the robotic device 101 received from the device interface 173, and outputs the generated control commands to the robotic device 101 from the device interface 173.In some embodiments, the control program 751 may also provide the status of the robotic device 101 received via the device interface 173 to the robotic task controller 108 via the network interface 172.

[0016] In some embodiments, the robotic task controller 108 may be a computer including a processing unit 181, a network interface 182, a user input interface 183, and data storage 184, all electrically connected (e.g., by a bus or other communication mechanism, as described below in connection with FIG. 18 ). In some embodiments, the processing unit 181 may include a CPU, RAM, ROM, etc., and may be configured to perform information processing based on programs and various data. In some embodiments, the network interface 182 may be connected to the robotic device controller 107 and may be an interface that sends movement commands for the robotic device 101 to the robotic device controller 107 and receives information about the status of the robotic device 101 (e.g., via communication with the network interface 172 of the robotic device controller 107 via the network 106). In some embodiments, the network interface 182 may also communicate with the camera 112 via the network 106. The user input interface 183 may be any device that receives input from a user, such as a mouse or keyboard, and controls the execution of programs and other operations of the robotic task controller 108. The data storage 184 may be a secondary memory device, such as a hard disk drive, which may, in some embodiments, store the recognition program 851, the robot task data 852, the goal planning program 853, the load estimation program 861, the motion planning program 862, the control parameter data 863, and / or the robot command generation program 864. When the robot task controller 108 is activated, such as by turning on power, it may provide (i.e., provide access to) the programs and data stored in the data storage 184 to (or for) the processing unit 181. The recognition program 851 may, in some embodiments, use images from the camera 112 to detect the work workpiece 102 and other objects.In some embodiments, the goal planning program 853 may use the recognition results of the recognition program 851 and the control parameter data 863 to plan the robot's task end state (e.g., the destination of the robot's end effector 111 and an end state such as holding / grasping, releasing, or rotating). The load estimation program 861 uses data from the robotic device controller 107 to estimate the robot's load state during operation. In some embodiments, the motion planning program 862 may determine a destination and target position (e.g., associated with the task end state) based on the task end state of the motion planning program 862 and the load state estimation results of the load estimation program 861, and plan the robot's movement to the destination or target position. In some embodiments, the robot control command generation program 864 may generate motion commands for the robotic device controller 107 based on the robot motion plan (e.g., including a set of positions or other information defining a path or motion for the robotic device 101) and may transmit the motion commands to the robotic device controller 107 via the network interface 182.

[0017] In a location where high-mix, low-volume production is carried out, work consisting of multiple processes (e.g., assembly work) may be performed in a single work area (cell), as symbolized by a cellular production system. For example, as shown in FIG. 3 , a robotic device 101 and a worker 104 may assemble a work object 102 placed on a work table 103 or transported by an autonomous vehicle 105. In such a workplace, the work object 102 being handled may be changed, or the work content may be swapped and / or changed in response to fluctuations in demand. For example, after the same worker 104 assembles product A, the work object 102 transported by the autonomous vehicle 105 may be changed in accordance with a change in the production plan, and the work may shift to assembling a different product B. One use case for improving work efficiency in such a location is for the robotic device 101 to supply the work object 102 and tools for work at the right time and in the right position, allowing the worker 104 to focus on the assembly work.

[0018] FIG. 4 shows the software configuration of the robot task controller 108. FIG. 5 is a flowchart illustrating the overall process used by this robot control system to set the robot's task goal end state (i.e., task target) and execute robot motion. The detection module 115 captures images captured by the camera 112 in step 500 and performs object recognition in step 510. Object recognition can use any detection method, such as image processing based on Histogram of Oriented Gradients (HOG) features or a deep learning method using a Convolutional Neural Network (CNN). FIG. 6 shows an example of an object recognition result from the detection module 115. As shown in FIG. 6, the positions of the robot hand (e.g., end effector 111), the assembly workpiece 200 being assembled, the tool 211, the part 221, and the other part 222 in the image are recognized and surrounded by a bounding box. The 3D position of the detected object is then obtained by combining the center position of the bounding box in the image with depth information. Furthermore, as shown in FIG. 6, the skeletal position of the worker 104 may be estimated and detected from the image to obtain position information of the shoulders, fingertips, etc.

[0019] In some embodiments, the goal planning module 120 may obtain the robot task data 100 defining a task to be performed by the robot (e.g., providing a tool 211 or a part 221 to the worker 104) and the object recognition results of the detection module 115 in step 520, and may output candidate target locations (associated with candidate task end states) to which the robot may move. FIG. 7 is a schematic block diagram showing a detailed software configuration of the goal planning module 120. FIG. 8 is a flowchart illustrating the detailed process of step 520 in FIG. 5. FIG. 9 is an illustration illustrating the goal planning module 120 calculating candidate task end states (e.g., alternatively referred to as candidate goal states, or candidate task goals or candidate task goal states). In step 521, the reference point detection module 121 uses the object detection results 109 and task data (e.g., the robot task data 100) to detect reference points according to the task (i.e., task) to be performed by the robot. 9(a), when the robot-handled object 311 to be handled by the robot is the tool 211, the right shoulder position (e.g., reference point 310) of the operator (e.g., worker 104) may be set as the reference point to minimize the distance between the operator's hand and the selected destination of the tool 211 for ease of use by the operator (i.e., worker 104). On the other hand, when the robot-handled object 311 is a part of the assembly work object 200, the next attachment position (e.g., reference point 320) may be set as the reference point to minimize the distance between the object and the attachment position (e.g., reference point 320) and place the robot-handled object 311 near the next attachment position (e.g., reference point 320), thereby reducing the operator's reach when assembling the assembly work object 200. In step 522, the target candidate detection module 122 uses the object detection results to output a plurality of candidate task end states (i.e., task goal candidates).In the detection result shown in FIG. 6 , a range having a predetermined size is extracted in an area where no object is detected on the work table 103, and the center position of the target area associated with the candidate task end states 300, 301, 302, and 303 shown in FIG. 9 , the size of the range, and the state of the robot at the center position (such as the posture of the end effector 111, the opening and closing of the end effector 111, and the movement time) may be output as the candidate task end states (task goal candidates). Extraction of the predetermined range may be configured, for example, to localize the top surface of the work table 103 with a predetermined grid size and extract a predetermined number of grids and / or grid points. In some embodiments, the distance score calculation module 123 may calculate the straight-line distance between the reference point 310 or the reference point 320 and the candidate task end states 300, 301, 302, and 303 in step 523 and output the straight-line distance as a score associated with each task goal candidate. 9(a), the candidate task end state 300 closest to the reference point 310 has the lowest score, and in FIG. 9(b), the candidate task end state 303 closest to the reference point 320 has the lowest score. In step 524, the movement score calculation module 124 outputs scores associated with the movement of the robotic device 101. As shown in FIG. 9, the candidate task end state 301 closest to the robot's end effector 111 has the lowest score. In step 525, the score integration module 125 combines the scores calculated by the distance score calculation module 123 and the scores calculated by the movement score calculation module 124, multiplies them by a product of predetermined weights, and outputs the scores of the candidate task end states 300, 301, 302, and 303. Finally, the goal planning module 120 may output the positions and scores of the candidate task end states 300, 301, 302, and 303 in step 526.

[0020] The motion planning module 130 may extract, or select, the task goal with the highest score among the candidate task end states (candidate task goals) output by the goal planning module 120 in step 530, and may plan and output a motion to reach the task goal in step 540. In some aspects, the motion to reach the task goal may be a series of positions and poses of the end effector 111 separated by a predetermined time interval, or may be a line segment describing the positions through which the end effector 111 must pass. The robot command generation module 150 may generate robot motion commands (e.g., joint angle commands) in step 550 based on the motion information output by the motion planning module 130 and sensor information (e.g., robot position information) output by the robotic device controller 107, and may output the robot motion commands to the robotic device controller 107. In some aspects, the load state estimation module 160 may estimate the load on the robotic device 101 based on sensor data of the robotic device 101 obtained by the robotic device controller 107 in step 560, and may output a modified score value for each candidate task end state and the modified states of the candidate task end states (candidate task goals) to the motion planning module 130 as goal modifications detected and / or associated with the load state estimation. In step 570, the motion planning module 130 may re-evaluate the current, i.e., selected, task end state (current task goal) based on the load state of the load state estimation module 160. If there is a change in the score order of the candidate task end states (candidate task goals), the motion planning module 130 returns to step 530 and modifies the task end states (i.e., task goals); if there is no change, the motion planning module 130 continues the process. If processing continues, the exercise planning module 130 determines in step 580 whether the movement has been completed to the specified state; if the movement has not been completed, the processing returns to step 550 to output additional movement commands and continue the movement as described above; if the movement has been completed, all processing ends.

[0021] FIG. 10 is a schematic block diagram showing the detailed software configuration of the motion planning module 130 and the load estimation module 160. FIG. 11 is a flowchart illustrating the detailed process of step 560. FIG. 12 is an example illustrating how the motion planning module 130 and the load estimation module 160 estimate the load state and change the task end state (i.e., task goal) according to the load state. The load vector calculation module 161 may calculate a load vector (size and direction of load) to be applied to the robot-handled object 311 being handled by the robot based on the robot position data (e.g., joint angles, etc.) and robot force data (e.g., joint torques, etc.) output by the robot device controller 107. In step 561, the load score calculation module 162 first checks whether a load exists by determining whether the magnitude (length) of the load vector output by the load vector calculation module 161 is greater than a predetermined value. If the magnitude of the load is less than the predetermined value, the process may end without further processing. If the magnitude of the load is greater than the predetermined value, the process may proceed to step 562. In step 562, it may be determined whether the load is oriented towards the candidate task end state with the highest score, i.e., towards the destination the robot is moving to; if the orientation is consistent, the process proceeds to step 563; otherwise, the process proceeds to step 565. In step 563, it may be determined whether the magnitude of the load is greater than a predetermined level; if the magnitude of the load is greater than the predetermined level, the process may proceed to step 564; otherwise, the process may terminate. In step 564, the state of the task end state (i.e., task goal) with the highest score is changed.For example, if the robot-handled object 311 being handled by the robotic device 101 is a tool (e.g., tool 211) and the worker 104 receives it from the robotic device 101 and intends to use the tool, a load may be applied in the same direction as the direction of the candidate task end state 300, as shown by the solid arrow in Figure 12, and the state of the end effector 111 in the current task end state (i.e., current task goal) may be changed to an open state, thereby handing the tool over to the on-site operator (e.g., worker 104). The state of the changed task end state (i.e., task goal) may not necessarily be limited to the open or closed state of the end effector 111, but may also be a reduction in travel time or a change in height at the center position of the candidate task end state 300.

[0022] If it is determined in step 562 that the direction of the load is not the same as the candidate task end state with the highest score, then in step 565, the score of the candidate task end state with the highest score may be reduced. Next, in step 566, a loop process may be performed to process all other candidate task end states one by one. In step 567, it may be determined whether the direction of the load is toward the other candidate task end state. If the direction is toward the candidate task end state, then in step 568, the score of the corresponding candidate task end state may be increased. If not, step 568 may be skipped, and after performing, i.e., skipping, step 568, the next loop process is performed. If the loop process is completed, the process of step 560 ends, and the process shown in FIG. 5 proceeds to step 570. If the worker 104 reaches out his / her arm / hand to pick up the part 221 and make contact with the robotic device 101, a load is applied in the direction opposite to the candidate task end state 300, as shown by the dotted arrow in FIG. 12. In this case, only the process of step 565 is performed (e.g., the loop beginning with step 566 may be skipped based on the load being identified as being in a direction associated with another candidate task end state or candidate task goal), and the process of step 560 is repeated to result in a lower score for the candidate task end state 300 and possibly a (relatively) higher score for the next highest-scoring candidate task end state 301, thereby changing the robotic device 101 from the candidate task end state 300 to the candidate task end state 301 by subsequent processing of steps 570, 530, and 540. In some aspects, the process described above with respect to the determination in step 562 that the direction of the load is not the same as the candidate task end state with the highest score (including the loop beginning with 566) may also be performed for loads applied in a direction opposite to the candidate task end state with the highest score.

[0023] In some aspects, an operator (e.g., worker 104) may intentionally attempt to change the destination of the robot-handled object 311 by applying a load in the same direction as the direction of a candidate task end state 303 (e.g., a candidate task end state that does not currently have the highest score), for example, as indicated by the dashed arrow in FIG. 12 . In this case, in addition to the processing of step 565, the processing of step 567 may identify the load direction as the same as the target direction for the candidate task end state 303 and proceed to increase the score associated with the candidate task end state 303 in step 568 (e.g., step 568 is performed for the candidate task end state 303). The processing of step 560 (e.g., the loop beginning in step 566) may continue to completion, resulting in a higher score for the candidate task end state 303, and subsequent processing in steps 570, 530, and 540 may change the robotic device 101 from the candidate task end state 300 to the candidate task end state 303.

[0024] In some aspects, the score evaluation module 131 may combine the goal modification values ​​output by the load score calculation module 162 (e.g., values ​​used to modify scores associated with one or more candidate task end states or task goal candidates based on the load estimation associated with step 560) with the candidate task end states (i.e., task goal candidates) output by the goal planning module 120 in step 530, and may output multiple candidate task end states (task goal candidates) arranged in order of highest (or lowest) score. The path planning module 132 may generate a motion path from the current position to the goal state of the candidate task end state with the highest score (e.g., to update the motion path to be consistent with the candidate task end state with the highest score after updating the scores) in step 540. The path can be a series of positions and poses of the end effector 111 separated by a predetermined time interval, or it can be a line segment describing the location the end effector 111 must traverse. 5 may, in some embodiments, continue until it determines in step 570 that there is no change in the load situation, as described above, and then determine whether the current state of the robot is at the final state of the current or selected task end state (e.g., the current or selected task goal) in step 580. If it is determined that the movement is not complete, the process may return to step 550 and movement commands may be output to continue the movement, whereas if it is determined that the movement is complete, the process may be terminated.

[0025] As shown in FIGS. 4, 5, 7, and 10, the goal planning module 120 (e.g., a goal planning algorithm or program) may generate a single set of candidate task end states (candidate task goals) that may be used for the duration of the task. For example, the goal planning module 120 may generate a set of candidate target areas (e.g., the size / area, center, or other specific location within, and shape of the candidate target areas) and a set of candidate states for the robot. The generated set of candidate task states may then be used by the motion planning module 130 (e.g., based on feedback from the actuator sensing module 175 and / or the load estimation module 160) to select and / or update task goals throughout the duration of the task. Thus, processing resources (e.g., processing cycles, power, etc.) or processing time associated with processing feedback and updating task end states (i.e., task goals) may be reduced by updating weights associated with the generated set of candidate task end states (candidate task goals) without having to generate a new set of candidate task end states (candidate task goals).

[0026] In this way, by changing the task end state (e.g., task goal or task target), according to the direction and magnitude of the load applied by the moving robot, the robot can instantly (i.e., quickly) adjust its movement in an infinite (i.e., changing or mutable) environment to suit the operator's work, and behave in a way that does not hinder the collaborative operator, thereby improving the efficiency of collaborative work.

[0027] The present disclosure further provides a method for storing operator characteristics and modifying (ie, generating) motion targets (eg, candidate task end-states or candidate task goals) according to the operator characteristics.

[0028] FIG. 13 illustrates the software configuration of the robot task controller 108, and FIG. 14 illustrates how this control system can set robot goals according to worker characteristics. The robot task controller 108 may further include a collaborator database 170 in some embodiments. In some embodiments, the detection module 115 may identify individual workers 104, and the collaborator database 170 may output personal data associated with the workers 104 to the goal planning module 120. In some embodiments, the goal planning module 120 may use the personal data of the workers 104 when planning task goals (and / or generating candidate task end states or candidate task goals) and may reflect (i.e., take into account) the characteristics of the workers 104 when planning task goals. In some embodiments, the motion planning module 130 may store information in the collaborator database 170 as personal data when a task goal is changed (e.g., as described in connection with steps 530-1700 of FIG. 5).

[0029] As shown in FIG. 14 , selecting a candidate task end state (task goal) without considering worker characteristics relative to the reference point 320 may select the candidate task end state 303 at the end of the dashed arrow. If the worker 104 has previously changed the current task end state (task goal), and the amount of change scaled by a predetermined magnitude of movement is represented by the dotted arrow, the double arrow (representing the difference between the reference point 320 and the updated reference point 321) combining the dashed and dotted arrows may be stored as personal data representing the worker's characteristics. As opposed to a task end state (task goal) (e.g., the candidate task end state 303) determined without considering personal data stored about the worker 104, the task end state (task goal) may, in some embodiments, be determined and / or set based on personal data stored about the worker 104. For example, instead of performing task goal determination using the reference point 320, personal data stored about the worker 104 may lead to the task end state (task goal) being determined based on an updated reference point 321 that differs from the reference point 320. 14, the candidate task end state 302 may be set as the task end state (task goal). In this way, the task end state (task goal) (e.g., the selected and / or determined task end state or task goal) may be modified according to the characteristics of an individual worker, allowing the robot to perform a task tailored to the individual worker, thereby further improving the efficiency of collaborative work.

[0030] In some embodiments of the present disclosure, a task end state (task goal) may be modified based on a load pattern. FIG. 15 is a list of example load patterns that may be used in some embodiments of the present disclosure to modify and / or update a task goal. For example, when the load score calculation module 162 detects the load pattern shown in FIG. 15 , it may change the state of the task end state (task goal) (e.g., a selected task end state or task goal) to perform the corresponding action shown in FIG. 15 . In some embodiments, when the load score calculation module 162 detects two quick pull-down movements shown in FIG. 15 , it may change the hand open / close state of the end effector 11 to release the hand at the site. In this way, the operator can receive the tool and module at the site. If a quick pull force is applied opposite the direction of movement shown in FIG. 15 and / or if a force is applied in the direction of a different candidate task end state (candidate task goal), the score of the current task goal may be decreased, and the score of the candidate task end state in the direction of the applied force may be increased. In this way, the worker can dynamically change the task end state (task goal) and achieve a work arrangement that is efficient for the worker. By changing the task end state (task goal) in this way according to a specific load pattern, the worker can explicitly change the task end state (task goal) and specify a work arrangement that is more efficient for the worker, thereby improving the efficiency of collaborative work.

[0031] Additional embodiments may include various combinations, modifications, or extensions of the above-described elements. For example, the robot arm 110 used in this illustrative example is shown as a vertical articulated robot, but can be a Cartesian coordinate robot, a horizontal articulated robot, a parallel link robot, etc. The robot device controller 107 and the robot task controller 108 are different controllers in this illustrative example, but can be configured to execute multiple programs within a single controller.

[0032] Additionally, while FIG. 6 illustrates object detection using bounding boxes, other detection methods may be used; for example, the system, i.e., detection module 115, may be configured to use classification results from image segmentation. While the task end states (task goals) or associated target areas shown in FIG. 9 may be depicted using rectangles, they may be any shape determined by the work task (i.e., objective) associated with the task end states (task goals). Task end states or target areas may be of different sizes, and in some embodiments, different task end states or target areas may have overlapping regions. While the task end states or target areas are depicted in FIG. 9 as a set of discontinuous regions, the task end states or target areas may be associated with location information expressed as reference to a (continuous) coordinate system (e.g., specified by a point within the coordinate system along one or more axes defined relative to the coordinate system, or by a range of values).

[0033] 9 is based on distance, other metrics may be used, alone or in combination, to calculate a score for each of multiple candidate task end states (candidate task goals). For example, the system may be configured to simulate in advance the curvature of the travel path and static loads due to the pose of the worker 104 and robotic device 101 at the goal point, and the simulated and / or predicted loads may be used to calculate a score for the candidate task goals. Additionally or alternatively, the methods for modifying scores associated with one or more candidate task end states (candidate task goals), selected candidate task goals, and / or system states may not be limited to those shown in FIGS. 5 and 11 and may further include consideration of different modification conditions, and states may be modified in one or more of steps 530, 540, and / or 550 by modifying the path of movement, modifying velocity commands, or modifying stiffness and elasticity settings in force control.

[0034] FIG. 16 is a flow diagram 1600 illustrating a method according to some embodiments of the present disclosure. In some embodiments, the method may be implemented by an apparatus (e.g., one of a collaborative robot system, a robot task controller 108, or a computing device 1805) for controlling a robotic device in an industrial environment. The apparatus may receive at least one input data set including one or more of task data related to a first task for the collaborative robotic device or environmental data related to at least a work area and a human operator associated with the first task. The at least one input data set may, in some embodiments, be related to the objective of the first task. In some embodiments, the first task may be a subtask of a larger manufacturing or assembly task. For example, with respect to the larger goal of assembling a product, the first task may be defined as providing tools or parts to a human worker who will assemble the product at a particular point in the assembly process. Thus, the at least one input data set may include data regarding the objective of the first task, a reference point associated with the first task (e.g., reference point 310 or reference point 320), a tool or part associated with the first task, image data associated with the work area and a human operator (i.e., worker), personal (historical) data associated with a particular human operator, or other task data relevant to defining and / or generating candidate task end states (e.g., candidate task goals or candidate robot task targets). For example, with reference to Figures 5 and 7, an apparatus (e.g., robotic task controller 108, detection module 115, and / or goal planning module 120) may capture images in step 500 and perform object recognition in step 510, or may receive task data as shown in Figure 7.

[0035] At 1620, the apparatus may generate multiple candidate task end states associated with the first task for the collaborative robotic device. In some aspects, the candidate task end states may be associated with candidate target areas and states of the collaborative robotic device (e.g., states of the collaborative robotic device's end effector). For example, each candidate task end state for the first task may be associated with a target location for placing an object held by the collaborative robotic device (e.g., a target location and release / release state of a component of the collaborative robotic device that holds the object). For example, with reference to FIGS. 5, 7, and 9, the apparatus (e.g., robot task controller 108, goal planning module 120, and / or target candidate detection module 122) may output candidate task end states (candidate task goals) such as candidate task end states 300, 301, 302, and 303 to which the robot may move at step 520.

[0036] After generating the plurality of candidate task end states in 1620, in some embodiments, the apparatus may generate (e.g., calculate or compute) a score for each of the plurality of candidate task end states associated with the first task. In some embodiments, the score for each of the candidate task end states associated with the first task may be generated based on a set of reference points and distances, or other characteristics. For example, with reference to FIGS. 5 and 7-9, the apparatus (e.g., robot task controller 108, goal planning module 120, and / or distance score calculation module 123) may calculate a score associated with each of the generated candidate task end states (candidate task goals), such as candidate task end states 300, 301, 302, and 303, to which the robot may travel, in connection with step 520 (e.g., steps 523-1205).

[0037] At 1640, the apparatus may select a first task end state from a plurality of candidate task end states for the collaborative robotic device. The first task end state, in some aspects, may be associated with a first target location for placing an object held by the collaborative robotic device. In some aspects, selecting the first task end state at 1640 may include selecting the first task end state associated with the best score, which may be the lowest score or the highest score depending on the method used to generate (i.e., calculate) the score for each candidate task end state. For example, referring to Figures 5 and 7-9, in step 530, the apparatus (e.g., robot task controller 108, motion planning module 130, score evaluation module 131, and / or path planning module 132) may select (i.e., extract) the task end state (i.e., task goal) with the highest score from among the candidate task end states (e.g., candidate task end states 300, 301, 302, and 303, or candidate task goals) output by the goal planning module 120 in step 520 (e.g., step 526).

[0038] At 1650, the apparatus may receive load data regarding the load experienced by the collaborative robotic device. The load data, in some embodiments, may include magnitude data and directional data associated with the load experienced by the collaborative robotic device. In some embodiments, the magnitude data and directional data may be associated with one of a plurality of patterns (e.g., the magnitude and directional patterns discussed in connection with FIG. 15 ). Each of the plurality of patterns, in some embodiments, may be associated with at least one value associated with a corresponding at least one of a plurality of candidate task end states. The at least one value, in some embodiments, may be a value that is added to or subtracted from a score or value by which to multiply or divide the score associated with the corresponding at least one of the plurality of candidate task end states. For example, with reference to FIGS. 5 and 10-12, an apparatus (e.g., the robot task controller 108, the load estimation module 160, and / or the load vector calculation module 161) may collect and / or receive sensor data (e.g., sensor data reflecting one of the force vectors shown in FIG. 12) about the robot device 101 obtained by the robot device controller 107 in step 560.

[0039] The apparatus may additionally or alternatively receive 1650 at least one additional input data set including updated environmental data regarding at least one of the work area and / or human operator associated with the first task (and multiple candidate task end states). The additional input data set may include data regarding changes to the position and / or orientation of the human operator or changes to one or more objects in the work area (e.g., a product being assembled, a tool, a part, etc.). The additional input data set may be used to update one or more locations (e.g., reference points) associated with the generation and / or calculation of the score. For example, if the score is based at least in part on the distance between the target area associated with the candidate task end state and one of the location of the human worker's right shoulder or the location of a component of the product being assembled (e.g., a point on the product where the current part is attached or the current tool is used), the locations used in subsequent score calculations may be based on the updated locations indicated and / or included in the additional input data set.

[0040] The device may generate an updated score for each of a plurality of candidate task end states associated with the first task based on the load data. In some embodiments, generating the updated score may be based on at least one value associated with the load data (e.g., magnitude and direction data associated with one of the plurality of patterns). In some embodiments, the device may generate the updated score based on an additional input data set (e.g., based on an updated position associated with the score calculation and indicated and / or included in the additional input data set). For example, with reference to FIGS. 5 and 10-12, the device (e.g., robot task controller 108, load estimation module 160, and / or load score calculation module 162) may generate (i.e., output) a revised score value for each candidate task end state and the revised states of the candidate task end states (candidate task goals) to motion planning module 130 as goal revision values ​​at step 560 (e.g., via steps 565-568) before returning to step 530.

[0041] At 1670, the device may select a second task end state from a plurality of candidate task end states based on the load data. In some aspects, the second task end state may be associated with a second target location for placing the object. In some aspects, the second task end state may be associated with releasing the object at the current location. To select the second task end state at 1670, in some aspects, the device may select the second task end state associated with the best (e.g., highest or lowest) updated score (after receiving the load data at 1650 and generating the updated score at 1660). For example, referring to Figures 5 and 7-9, the device (e.g., the robot task controller 108, the motion planning module 130, the score evaluation module 131, and / or the path planning module 132) may return to step 530 and select (i.e., extract) the task end state (task goal) with the updated (i.e., current) highest score from among the candidate task end states 300, 301, 302, and 303 (i.e., task goal candidates) output by the goal planning module 120 in step 520 (e.g., step 526).

[0042] FIG. 17 is a flow diagram 1700 illustrating a method according to some embodiments of the present disclosure. In some embodiments, the method may be implemented by an apparatus (e.g., one of a collaborative robot system, a robot task controller 108, or a computing device 1805) for controlling a robotic device in an industrial environment. At 1710, the apparatus may receive at least one input data set including one or more of task data related to a first task for the collaborative robotic device or environmental data related to at least a work area and a human operator associated with the first task. The at least one input data set may, in some embodiments, be related to the objective of the first task. In some embodiments, the first task may be a subtask of a larger manufacturing task. For example, with respect to a larger goal of assembling a product, the first task may be defined as providing tools or parts to a human worker who will assemble the product at a particular point in the assembly process. Thus, the at least one input data set may include data regarding the objective of the first task, a reference point associated with the first task (e.g., reference point 310 or reference point 320), a tool or part associated with the first task, image data associated with the work area and a human operator (i.e., worker), personal (historical) data associated with a particular human operator, or other task data relevant to defining and / or generating candidate task end states (e.g., candidate task goals or candidate robot task targets). For example, with reference to Figures 5 and 7, an apparatus (e.g., robotic task controller 108, detection module 115, and / or goal planning module 120) may capture images in step 500 and perform object recognition in step 510, or may receive task data as shown in Figure 7.

[0043] At 1720, the apparatus may generate multiple candidate task end states associated with the first task for the collaborative robotic device. In some aspects, the candidate task end states may be associated with candidate target areas and states of the collaborative robotic device (e.g., states of the collaborative robotic device's end effector). For example, each candidate task end state for the first task may be associated with a target location for placing an object held by the collaborative robotic device (e.g., a target location and release / release state of a component of the collaborative robotic device holding the object). For example, with reference to FIGS. 5, 7, and 9, the apparatus (e.g., robot task controller 108, goal planning module 120, and / or target candidate detection module 122) may output candidate task end states (candidate task goals) such as candidate task end states 300, 301, 302, and 303 to which the robot may move at step 520.

[0044] After generating the plurality of candidate task end states at 1720, in some aspects, the apparatus may generate (e.g., calculate or compute) a score for each of the plurality of candidate task end states associated with the first task at 1730. In some aspects, the score for each of the candidate task end states associated with the first task may be generated based on a set of reference points and distances, or other characteristics. For example, with reference to FIGS. 5 and 7-9, the apparatus (e.g., robot task controller 108, goal planning module 120, and / or distance score calculation module 123) may calculate a score associated with each of the generated candidate task end states (candidate task goals), such as candidate task end states 300, 301, 302, and 303, to which the robot may travel, in connection with step 520 (e.g., steps 523-1205).

[0045] At 1740, the apparatus may select a first task end state from a plurality of candidate task end states for the collaborative robotic device. The first task end state, in some aspects, may be associated with a first target location for placing an object held by the collaborative robotic device. In some aspects, selecting the first task end state at 1740 may include selecting the first task end state associated with the best score, which may be the lowest score or the highest score depending on the method used to generate (i.e., calculate) the score at 1730. For example, referring to Figures 5 and 7-9, in step 530, the apparatus (e.g., robot task controller 108, motion planning module 130, score evaluation module 131, and / or path planning module 132) may select (i.e., extract) the task end state (i.e., task goal) with the highest score from among the candidate task end states (e.g., candidate task end states 300, 301, 302, and 303, or candidate task goals) output by the goal planning module 120 in step 520 (e.g., step 526).

[0046] At 1750, the apparatus may receive load data regarding the load experienced by the collaborative robotic device. The load data, in some embodiments, may include magnitude data and direction data associated with the load experienced by the collaborative robotic device. In some embodiments, the magnitude data and direction data may be associated with one of a plurality of patterns (e.g., the magnitude and direction patterns discussed in connection with FIG. 15 ). Each of the plurality of patterns, in some embodiments, may be associated with at least one value associated with a corresponding at least one of a plurality of candidate task end states. The at least one value, in some embodiments, may be a value that is added to or subtracted from a score or value by which to multiply or divide the score associated with the corresponding at least one of the plurality of candidate task end states. For example, with reference to FIGS. 5 and 10-12, an apparatus (e.g., the robot task controller 108, the load estimation module 160, and / or the load vector calculation module 161) may collect and / or receive sensor data (e.g., sensor data reflecting one of the force vectors shown in FIG. 12) about the robot device 101 obtained by the robot device controller 107 in step 560.

[0047] The apparatus may additionally or alternatively receive 1750 at least one additional input data set including updated environmental data regarding at least one of the work area and / or human operator associated with the first task (and multiple candidate task end states). The additional input data set may include data regarding changes to the position and / or orientation of the human operator or changes to one or more objects in the work area (e.g., a product being assembled, a tool, a part, etc.). The additional input data set may be used to update one or more locations (e.g., reference points) associated with the generation and / or calculation of the score. For example, if the score is based at least in part on the distance between the target area associated with the candidate task end state and one of the location of the human worker's right shoulder or the location of a component of the product being assembled (e.g., a point on the product where the current part is attached or the current tool is used), the locations used in subsequent score calculations may be based on the updated locations indicated and / or included in the additional input data set.

[0048] At 1760, the device may generate an updated score for each of a plurality of candidate task end states associated with the first task based on the load data. In some embodiments, generating the updated score at 1760 may be based on at least one value associated with the load data (e.g., magnitude and direction data associated with one of the plurality of patterns). In some embodiments, the device may generate the updated score at 1760 based on an additional input data set (e.g., based on an updated position associated with the score calculation and indicated and / or included in the additional input data set). For example, with reference to FIGS. 5 and 10-12 , the device (e.g., robot task controller 108, load estimation module 160, and / or load score calculation module 162) may generate (i.e., output) a revised score value for each candidate task end state and the revised states of the candidate task end states (candidate task goals) to motion planning module 130 as goal revision values ​​at step 560 (e.g., via steps 565-568) before returning to step 530.

[0049] At 1770, the device may select a second task end state from a plurality of candidate task end states based on the load data. In some aspects, the second task end state may be associated with a second target location for placing the object. In some aspects, the second task end state may be associated with releasing the object at the current location. To select the second task end state at 1770, in some aspects, the device may select the second task end state associated with the best updated score (after receiving the load data at 1750 and generating the updated score at 1760). For example, referring to Figures 5 and 7-9, the device (e.g., the robot task controller 108, the motion planning module 130, the score evaluation module 131, and / or the path planning module 132) may return to step 530 and select (i.e., extract) the task end state (task goal) with the updated (i.e., current) highest score from among the candidate task end states 300, 301, 302, and 303 (i.e., task goal candidates) output by the goal planning module 120 in step 520 (e.g., step 526).

[0050] As described above, example implementations described herein include an innovative collaborative robotic (i.e., robot) system and method for providing it that performs highly efficient collaborative work between a collaborative robot system and a collaborative operator by appropriately responding to the actions of the collaborative operator (e.g., a human worker) and continuing the collaborative work without interruption. In some embodiments, the proposed system improves the efficiency of the collaborative work by changing task goals according to changing conditions (e.g., the direction of a load applied to the robot) when the worker and robot simultaneously move and perform the collaborative work, thereby allowing the worker to perform the work without interrupting the work and the collaborative robot system without becoming an obstacle to the collaborative operator and / or the worker. Example implementations discussed herein may improve the efficiency of the collaborative work by setting multiple work targets for the robot in advance and changing the work targets without interrupting the operation. Additionally, in some embodiments, setting the robot's task goals based on reference points and sensing results according to the work content enables example implementations to optimize each task in a process in which multiple tasks are performed. The system, in some aspects of the present disclosure, may achieve efficient operation for each task and streamline the work of processes across multiple processes.

[0051] 18 illustrates an example computing environment having example computing devices suitable for use in some illustrative implementations. The computing device 1805 of the computing environment 1800 can include one or more processing units, cores, or processors 1810, memory 1815 (e.g., RAM, ROM, and / or other), internal storage 1820 (e.g., magnetic, optical, solid-state storage, and / or organic), and / or IO interface 1825, any of which can be coupled by a communication mechanism or bus 1830 for communicating information or incorporated into the computing device 1805. The IO interface 1825 is also configured to receive images from a camera or provide images to a projector or display, depending on the desired implementation.

[0052] The computing device 1805 can be communicatively coupled to an input / user interface 1835 and an output device / interface 1840. Either or both of the input / user interface 1835 and the output device / interface 1840 can be wired or wireless interfaces and can be detachable. The input / user interface 1835 can include any device, component, sensor, or interface, physical or virtual, that can be used to provide input (e.g., buttons, touchscreen interface, keyboard, pointing / cursor control, microphone, camera, Braille, motion sensor, accelerometer, optical reader, and / or the like). The output device / interface 1840 can include a display, television, monitor, printer, speaker, Braille, etc. In some example implementations, the input / user interface 1835 and the output device / interface 1840 can be embedded in or physically coupled to the computing device 1805. In other example implementations, other computing devices may function as or provide the functionality of input / user interface 1835 and output device / interface 1840 of computing device 1805 .

[0053] Examples of computing devices 1805 may include, but are not limited to, highly mobile devices (e.g., smartphones, devices in automobiles or other machines, devices carried by people and animals, etc.), mobile devices (e.g., tablets, notebooks, laptops, personal computers, portable televisions, radios, etc.), and devices not designed for mobility (e.g., desktop computers, other computers, information kiosks, televisions with one or more processors embedded and / or combined televisions, radios, etc.).

[0054] Computing device 1805 can be communicatively coupled (e.g., via IO interface 1825) to external storage 1845 and a network 1850 for communicating with any number of networked components, devices, and systems, including one or more computing devices of the same or different configurations. Computing device 1805, or any connected computing device, may function as, provide services for, or be referred to as a server, client, thin server, general-purpose machine, special-purpose machine, or another label.

[0055] IO interface 1825 can include, but is not limited to, wired and / or wireless interfaces using any communication or IO protocol or standard (e.g., Ethernet, 1802.11x, Universal System Bus, WiMax, modem, cellular network protocols, etc.) for communicating information to and from at least all connected components, devices, and networks of computing environment 1800. Network 1850 can be any network or combination of networks (e.g., the Internet, a local area network, a wide area network, a telephone network, a cellular network, a satellite network, etc.).

[0056] The computing device 1805 can use and / or communicate using computer-usable or computer-readable media, including transitory media and non-transitory media. Transitory media include transmission media (e.g., metallic cables, fiber optics), signals, carrier waves, etc. Non-transitory media include magnetic media (e.g., disks and tape), optical media (e.g., CD ROM, digital video disks, Blu-ray disks), solid media (e.g., RAM, ROM, flash memory, solid-state storage), and other non-volatile storage or memory.

[0057] The computing device 1805 can be used to implement techniques, methods, applications, processes, or computer-executable instructions in some example computing environments. The computer-executable instructions can be retrieved from a transitory medium or stored on and retrieved from a non-transitory medium. The executable instructions can be in one or more of any programming, scripting, and machine language (e.g., C, C++, C#, Java, Visual Basic, Python, Perl, JavaScript, etc.).

[0058] The processors 1810, individually or in any combination, can run under any operating system (OS) (not shown), in a native or virtual environment. One or more applications can be deployed, including a logic unit 1860, an application programming interface (API) unit 1865, an input unit 1870, an output unit 1875, and an inter-unit communication mechanism 1895 through which different units communicate with each other, the OS, and other applications (not shown). The described units and elements may vary in design, function, configuration, or implementation and are not limited to the provided description. The processor 1810 can be in the form of a hardware processor, such as a central processing unit (CPU), or a combination of hardware and software units.

[0059] In some example implementations, once information or instructions to execute are received by API unit 1865, they may be communicated to one or more other units (e.g., logic unit 1860, input unit 1870, output unit 1875). In some examples, logic unit 1860 may be configured to control the flow of information between units and, in some example implementations described above, direct the services provided by API unit 1865, input unit 1870, and output unit 1875. For example, the flow of one or more processes or implementations may be controlled solely by logic unit 1860 or in combination with API unit 1865. Input unit 1870 may be configured to obtain inputs for calculations described in example implementations, and output unit 1875 may be configured to provide outputs based on the calculations described in example implementations.

[0060] The processor 1810 can be configured, individually or in any combination, to generate, for the collaborative robotic device, a plurality of candidate task end states associated with a first task. The processor 1810 can be configured, individually or in any combination, to select, for the collaborative robotic device, a first task end state from the plurality of candidate task end states. The processor 1810 can be configured, individually or in any combination, to receive load data regarding loads experienced by the collaborative robotic device. The processor 1810 can be configured, individually or in any combination, to select, based on the load data, a second task end state from the plurality of candidate task end states.

[0061] The processor 1810 may also be configured to generate a score for each of a plurality of candidate task end states associated with the first task, individually or in any combination. The processor 1810 may also be configured to select the first task end state associated with the best score, individually or in any combination. The processor 1810 may also be configured to generate an updated score for each of a plurality of candidate task end states associated with the first task based on the load data, individually or in any combination. The processor 1810 may also be configured to select the second task end state associated with the best updated score, individually or in any combination. The processor 1810 may also be configured to generate the updated score based on at least one value associated with a pattern, individually or in any combination. The processor 1810 may also be configured to receive at least one input data set including one or more of task data related to the task associated with the plurality of candidate task end states, or environmental data related to at least the work area and the human operator, individually or in any combination. The processor 1810 may also be configured to generate a plurality of candidate task end states based on the first input data set of the at least one input data set, individually or in any combination.

[0062] Some portions of the detailed descriptions are presented in terms of algorithms and symbolic representations of operations within a computer. These algorithmic descriptions and symbolic representations are the means used by those skilled in the data processing arts to convey the essence of their innovations to others skilled in the art. An algorithm is a prescribed sequence of steps leading to a desired end state or result. In exemplary implementations, the performed steps require physical manipulations of tangible quantities to achieve a tangible result.

[0063] Unless otherwise specifically indicated, as will be apparent from the discussion, it will be recognized that throughout the description, discussion utilizing terms such as "processing," "computing," "calculating," "determining," "displaying," and the like can include operations and processes of a computer system or other information processing device that manipulate and convert data represented as physical (electronic) quantities in the computer system's registers and memory into other data similarly represented as physical quantities in the computer system's memory or registers or other information storage, transmission, or display device.

[0064] Example implementations may also relate to an apparatus for performing the operations herein. This apparatus may be specially constructed for the required purposes, or may include one or more general-purpose computers selectively activated or reconfigured by one or more computer programs. Such computer programs may be stored on a computer-readable medium, such as a computer-readable storage medium or a computer-readable signal medium. Computer-readable storage media may include tangible media, such as optical disks, magnetic disks, read-only memory, random-access memory, solid-state devices and drives, or any other type of tangible or non-transitory medium suitable for storing electronic information. Computer-readable signal media may include media such as carrier waves. The algorithms and displays presented herein are not inherently related to any particular computer or other apparatus. A computer program may include a pure software implementation containing instructions that perform the operations of a desired implementation.

[0065] Various general-purpose systems may be used with the programs and modules according to the examples herein, or it may prove convenient to construct more specialized apparatus to perform the desired method steps. Additionally, the example implementations are not described with reference to any particular programming language. It will be appreciated that a variety of programming languages ​​may be used to implement the teachings of the example implementations as described herein. Instructions in the programming language may be executed by one or more processing devices, such as a central processing unit (CPU), processor, or controller.

[0066] As is known in the art, the operations described above can be implemented by hardware, software, or some combination of software and hardware. Various aspects of the implementations may be implemented using circuits and logic devices (hardware), while other aspects may be implemented using instructions stored on a machine-readable medium (software) that, when executed by a processor, cause the processor to perform methods that implement the implementations of the present application. Furthermore, some example implementations of the present application may be implemented solely by hardware, while other example implementations may be implemented solely by software. Furthermore, the various functions described may be implemented in a single unit or may be spread across multiple components in various manners. When implemented in software, the methods may be executed by a processor, such as a general-purpose computer, based on instructions stored on a machine-readable medium. If desired, the instructions may be stored on the medium in compressed and / or encrypted format.

[0067] Additionally, other implementations of the present application will be apparent to those skilled in the art from consideration of the specification and practice of the teachings of the present application. Various aspects and / or components of the described exemplary implementations may be used alone or in any combination. It is intended that the specification and exemplary implementations be considered as examples only, with the true scope and spirit of the present application being indicated by the following claims.

Claims

1. at least one memory; at least one processor coupled to said at least one memory; and wherein, based at least in part on information stored in the at least one memory, the at least one processor, individually or in any combination: generating a plurality of candidate task end states associated with the first task for a collaborative robotic device having an end effector; selecting a first task end state from the plurality of candidate task end states for the collaborative robot device to move to a first target position for placing an object held by the end effector; moving the robot to reach the first task end state; receiving load data regarding loads experienced by the collaborative robotic device during the movement; When it is determined that the magnitude of the load is greater than a predetermined value and the load is in the same direction as the direction of the first task end state, At the determined position, the end effector selects a second task end state in which the object is released, and moves the robot to reach the second task end state. A control system configured as follows.

2. The at least one processor further comprises: configured to generate a score for each of the plurality of candidate task end states associated with the first task, and for selecting the first task end state, the at least one processor configured to select the first task end state associated with a best score. The control system of claim 1 .

3. The at least one processor further comprises: configured to generate an updated score for each of the plurality of candidate task end states associated with the first task based on the weight data, and for selecting the second task end state, the at least one processor configured to select the second task end state associated with the best updated score. The control system of claim 2 .

4. The control system of claim 3 , wherein the load data includes magnitude and direction data associated with the load experienced by the collaborative robotic device.

5. 5. The control system of claim 4, wherein the magnitude data and the direction data are associated with one pattern of a plurality of patterns, each of the plurality of patterns being associated with at least one value associated with a corresponding at least one of the plurality of candidate task end states, and wherein to generate the updated score, the at least one processor is further configured to generate the updated score based on the at least one value associated with the one pattern.

6. The at least one processor further comprises: configured to receive at least one input data set including one or more of task data related to a task associated with the plurality of candidate task end states, or environmental data related to at least a work area and a human operator; The control system of claim 1 .

7. The at least one processor further comprises: configured to generate the plurality of candidate task end states based on a first input data set of the at least one input data set. The control system of claim 6.

8. 8. The control system of claim 7, wherein the environmental data includes image data associated with the work area and the human operator, and wherein for selecting the second task end state from the plurality of candidate task end states, the at least one processor is further configured to select the second task end state based on a second input data set of the at least one input data set that includes the environmental data.

9. A method for generating a plurality of candidate task end states associated with a first task for a collaborative robotic device having an end effector; selecting a first task end state from the plurality of candidate task end states for the collaborative robotic device to move to a first target position for placing an object held by the end effector; moving the robot to reach the first task end state; receiving load data regarding loads experienced by the collaborative robotic device during the movement; When it is determined that the magnitude of the load is greater than a predetermined value and the load is in the same direction as the direction of the first task end state, At the determined position, the end effector selects a second task end state in which the held object is released.

1. A method for controlling a collaborative robotic device, comprising:

10. generating a score for each of the plurality of candidate task end states associated with the first task, wherein selecting the first task end state includes selecting the first task end state associated with a best score.

10. The method of claim 9.

11. generating an updated score for each of the plurality of candidate task end states associated with the first task based on the load data, and wherein selecting the second task end state includes selecting the second task end state associated with a best updated score. The method of claim 10.

12. 12. The method of claim 11 , wherein the load data includes magnitude data and directional data associated with the load experienced by the collaborative robotic device, the magnitude data and the directional data being associated with one of a plurality of patterns, each of the plurality of patterns being associated with at least one value associated with a corresponding at least one of the plurality of candidate task end states, and generating the updated score includes generating the updated score based on the at least one value associated with the one of the patterns.

13. receiving at least one input data set including one or more of task data related to a task associated with the plurality of candidate task end states or environmental data related to at least a work area and a human operator; 10. The method of claim 9, further comprising:

14. generating the plurality of candidate task end states based on a first input data set of the at least one input data set; 14. The method of claim 13, further comprising:

15. 15. The method of claim 14, wherein the environmental data includes image data associated with the work area and the human operator, and wherein selecting the second task end state from the plurality of candidate task end states is further based on a second input data set of the at least one input data set that includes the environmental data.

Images