Method, robot and robot system for safety monitoring

Abstract

Embodiments of the present disclosure relate to method security monitoring. The method includes obtaining sensor data collected by at least one sensor on a transport device associated with and in communication with a robot. The method further includes determining a presence of a human based on the sensor data. The method further includes determining a relative position of the human relative to the robot based on the sensor data. The method further includes determining a control operation based on the relative position. In this manner, by utilizing sensor data collected with sensors on the transport means, additional sensors for the safety protection function are not required, thereby reducing the cost of the robot.

Description

TECHNICAL FIELD

[0001] Embodiments of the present disclosure generally relate to the field of robotics, and more particularly, to a method for safety monitoring, a robot, and a robot system. BACKGROUND

[0002] Nowadays, the application demand for using free-accessible robots is increasing. In some examples, in an industrial facility in which a plurality of workstations are arranged in a decentralized manner, a transport device such as an automated guided vehicle (AGV) or an autonomous mobile robot (AMR) can be provided to transfer materials to be processed from one workstation to another. In such an industrial facility, the workstations at which the robots operate are usually not provided with fences to isolate the site personnel. In this case, a safety protection function is required. That is, when some personnel are too close to the robot work area of the operating robot, the operating robot can need to reduce its operating speed, or even stop immediately.

[0003] In order to take measures automatically in response to detecting that the personnel are too close to the operating robot, the robot needs to monitor its vicinity and determine whether personnel enter a certain area around the robot. SUMMARY

[0004] In view of the foregoing description, exemplary embodiments of the present disclosure propose a solution for safety monitoring to provide a safety protection function.

[0005] In a first aspect of the present disclosure, exemplary embodiments of the present disclosure provide a method for safety monitoring. The method for safety monitoring includes obtaining sensor data collected by at least one sensor on a transport device associated with and in communication with a robot. The method further includes determining a presence of a human based on the sensor data. The method further includes determining a relative position of the human with respect to the robot based on the sensor data. The method further includes determining a control operation based on the relative position. For these embodiments, by utilizing the sensor data collected with the sensor on the transport device, no additional sensor is needed for the protection function, thereby reducing the cost of the robot system.

[0006] In some exemplary embodiments, the method further includes receiving, by the robot, the sensor data from the transport device, and performing, by the robot, the control operation after determining the control operation. For these embodiments, the method for safety monitoring can be implemented at the robot. The robot can constantly receive the sensor data from the transport device, and perform the determined control operation when determining the control operation.

[0007] In some example embodiments, the method further includes transmitting, by the transport device, instructions to the robot to control the robot to perform the control operation. For these embodiments, the method for safety monitoring is implemented at the transport device.

[0008] In some example embodiments, in determining the relative position, the robot or the transport device can determine, based on the sensor data, at least one human position of the human relative to the at least one sensor. The robot or the transport device can determine at least one sensor position of the at least one sensor. The robot or the transport device can determine the relative position based on the at least one sensor position and the at least one human position. For these embodiments, by calculating the relative position of both the human and the robot relative to the sensor, the relative position between the human and the robot can be obtained.

[0009] In some example embodiments, in determining the relative position based on the at least one sensor position and the at least one human position, the robot or the transport device can determine, based on the at least one human position and the at least one sensor position, at least one human vector between the at least one sensor and the human. The robot or the transport device can determine, based on the at least one sensor position and a robot position of the robot, at least one sensor vector between the at least one sensor and the robot. The robot or the transport device can determine, based on the at least one human vector and the at least one sensor vector, at least one detection vector. If it is determined that the at least one detection vector indicates a single distance between the human and the robot, the robot or the transport device can determine one of the at least one detection vector that represents the relative position between the human and the robot.

[0010] In some alternative embodiments, if it is determined that the at least one relative position indicates multiple distances between the human and the robot, the robot or the transport device can determine a relative position that indicates a smallest distance of the multiple distances to represent the relative position between the human and the robot. For these embodiments, by selecting the relative position with the smaller distance, the safety level is improved.

[0011] In some example embodiments, in determining the control operation, the robot or the transport device can determine, based on the relative position, a distance between the robot and the human. The robot or the transport device can obtain at least one distance threshold associated with a working range of the robot. The robot or the transport device can determine the control operation based on the distance and the at least one distance threshold. For these embodiments, by comparing the obtained distance with the at least one distance threshold, a suitable control operation can be determined from a plurality of control operations corresponding to different scenarios.

[0012] In some example embodiments, in determining the control operation based on the distance and the at least one distance threshold, if it is determined that the at least one distance threshold includes a first distance threshold associated with a maximum working range of the robot, the robot or the transport device can compare the first distance threshold with the distance. If it is determined that the distance is less than the first distance threshold, the robot or the transport device can determine the control operation as pausing operation of the robot. For these embodiments, the first distance threshold is greater than the maximum working range of the robot.

[0013] In some alternative embodiments, in determining the control operation based on the distance and the at least one distance threshold, if it is determined that the distance is greater than the first distance threshold, the robot or the transport device can compare the distance with a second distance threshold that is greater than the first distance. If it is determined that the distance is less than the second distance threshold, the robot or the transport device can determine the control operation as reducing an operating speed of the robot.

[0014] In some alternative embodiments, in determining the control operation based on the distance and the at least one distance threshold, if it is determined that the distance is greater than the second distance threshold, the robot or the transport device can determine the control operation as maintaining operation of the robot. For these embodiments, by sequentially comparing the determined distance with all distance thresholds, a risk of potential collision can be determined.

[0015] In some example embodiments, the transport device includes an automated guided vehicle (AGV) or an autonomous mobile robot (AMR).

[0016] In some example embodiments, the robot is integrated on the transport device.

[0017] In some example embodiments, in some cases where the robot is separate from the transport device, the transport device can communicate with the robot wirelessly. In some cases where the robot is integrated on the transport device, the transport device can communicate with the robot wirelessly or through a cable with a dedicated interface. For these embodiments, the robot and the transport device can be more flexibly arranged in an industrial facility.

[0018] In a second aspect, example embodiments of the present disclosure provide a robot. The robot includes at least one controller; and at least one memory storing instructions that, when executed by the at least one controller, cause the robot to perform the method for safety monitor according to the first aspect of the present disclosure.

[0019] In a third aspect, example embodiments of the present disclosure provide a robotic system. The robotic system comprises at least one robot configured to process a material and configured to perform the method according to the first aspect of the present disclosure. The robotic system further comprises at least one transport device. The at least one transport device comprises at least one sensor configured to detect a human, and the at least one transport device is configured to transport the material to the at least one robot.

[0020] In a fourth aspect, example embodiments of the present disclosure provide a computer readable medium having stored thereon instructions which, when executed on at least one processor, cause the at least one processor to perform the method of safety monitoring according to the first or second aspect of the present disclosure. BRIEF DESCRIPTION OF DRAWINGS

[0021] The above and other objects, features and advantages of the example embodiments disclosed herein will become more apparent from the following detailed description thereof when taken in conjunction with the accompanying drawings. In the drawings, various example embodiments disclosed herein will be shown in example and non-limiting manners, in which:

[0022] Figure 1 a block diagram schematically illustrating an example robotic system in which example embodiments of the present disclosure can be implemented;

[0023] Figure 2 a signaling diagram schematically illustrating an example process for safety monitoring according to embodiments of the present disclosure;

[0024] Figures 3A to 3C a flow diagram schematically illustrating a method for determining a control operation according to embodiments of the present disclosure;

[0025] Figure 4 a schematic diagram schematically illustrating an example robotic system according to some embodiments of the present disclosure;

[0026] Figure 5 a schematic diagram schematically illustrating an example robotic system according to some further embodiments of the present disclosure;

[0027] Figure 6 a schematic diagram schematically illustrating an example robotic system according to yet some further embodiments of the present disclosure; and

[0028] Figure 7 a schematic diagram schematically illustrating a controller for implementing a method according to embodiments of the present disclosure.

[0029] Throughout the drawings, the same or similar reference numerals and letters indicate the same or similar elements. DETAILED DESCRIPTION

[0030] The principles of the present disclosure will now be described with reference to some example embodiments. It should be understood that the embodiments are described for illustrative purposes only and to assist with the understanding of and implementation of the present disclosure by those skilled in the art, and do not imply any limitation on the scope of the present disclosure. The disclosure described herein can be implemented in various ways other than those described below.

[0031] In the following description and claims, unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs.

[0032] Reference is made in this disclosure to "one embodiment", "an embodiment", "example embodiments", and the like, meaning that a described embodiment can include a particular feature, structure, or characteristic, but each embodiment does not necessarily include the particular feature, structure, or characteristic. Furthermore, such phrases are not necessarily referring to the same embodiment. Further, when a particular feature, structure, or characteristic is described in connection with an embodiment, it is submitted that it is within the purview of one of ordinary skill in the art to effect such feature, structure, or characteristic in connection with other

[0033] It should be understood that although the terms "first" and "second" and the like can be used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another. For example, a first element could be termed a second element, and, similarly, a second element could be termed a first element, without departing from the scope of example embodiments. As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items.

[0034] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of example embodiments. As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises", "comprising", "has", "having", "includes" and / or "including" when used herein, specify the presence of stated features, elements and / or components etc. but do not preclude the presence or addition of one or more other features, elements, components and / or combinations thereof.

[0035] As described above, conventionally, in order to monitor moving objects in the vicinity of a freely accessible robot, a large majority of all humans, a dedicated laser sensor with a wide sensing range needs to be provided for object detection. However, such laser sensors are expensive, thereby increasing the overall cost of the robot system.

[0036] In view of the above, a safety monitoring mechanism is provided. In the safety monitoring mechanism, the robot acquires and utilizes sensor data collected by a transport device associated with and in communication with the robot. During the course of the transport device moving according to its predefined route and stopping at a workstation to perform a collaborative operation with the robot operating in the workstation, the transport device utilizes its sensors to detect objects on its route to avoid collisions. The detection range of the sensors on the transport device can cover a very wide range, especially the working area of the robot. Thus, the robot can utilize the sensor data to determine whether a person is in the working area of the robot. In this way, a safety guarding function can be achieved without additional laser sensors, thereby reducing the overall cost of the robot system.

[0037] Reference will be made to Figure 1 A framework according to embodiments of the present disclosure will be described. Figure 1 A block diagram of an example robot system 100 in which example embodiments of the present disclosure can be implemented is shown schematically. As Figure 1 As shown, the robot system 100 includes a robot 110 deployed in a workstation 120, e.g., in an industrial facility. In the industrial facility, there can be multiple workstations arranged on one side of an aisle. The aisle is provided for the transport device to travel to the associated robot. The workstation 120 is fixed in the industrial facility at a location known to the controller 110 of the robot 120. The controller 110 is configured to control the operation of the robot 120.

[0038] As shown, the robot system 100 further includes a transport device 130. The transport device 130 is associated with the robot 110, and the transport device and the robot are wirelessly connected and communicate with each other over a wireless network. For example, the robot 110 and the transport device 130 can all be connected to a wireless local area network (WLAN) of the industrial facility. The transport device 130 includes a controller 131 configured to control the transport device 130 to travel along a preprogrammed route. When the transport device 130 reaches the workstation 120, the arm 112 of the robot 110 can pick up the material carried by the transport device 130 for further processing. After the collaborative operation is completed, the transport device 130 will continue to travel along the route and can carry the remaining material to the next workstation.

[0039] The robot 110 comprises a base 111 and an arm 112. The base 111 can be fixed at the workstation 120. The arm 112 can be provided, for example, with an end effector at one end. The arm 112 can comprise a plurality of segments. Each segment can be rotatable about another segment to which it is connected. In the illustrated embodiment, the arm 112 can reach the edge of its maximum working area 140 of radius R1 when all segments of the arm 112 are aligned along a straight line. If a human enters the maximum working range 140, the human can collide with the arm 112.

[0040] The transport device 130 is configured to detect obstacles in the predefined route with laser scanners during movement to avoid collisions. In the illustrated embodiment, two laser scanners are provided on the transport device 130, including a first laser scanner 132 located in front and a second laser scanner 133 located in the back. The first laser scanner 132 can be configured to detect objects in a front range 151 and the second laser scanner 133 can be configured to detect objects in a back range 152 such that they can cover a wide range around the transport device 130. When the transport device 130 detects a human, for example the human 160 as illustrated, in its predefined route, the transport device 130 can determine the relative distance to the human 160 to check whether the human 160 poses a risk of collision with the transport device 130. If the transport device 130 finds that there is indeed a risk, the transport device 130 can slow down or stop.

[0041] Further, since the transport device 130 can communicate with the robot 110, sensor data about the presence of the human 160 can be transmitted from the transport device 130 to the robot 110. In this way, the robot 110 can also be aware of the presence of the human 160 and take corresponding measures to avoid collisions between the arm 112 and the human 160.

[0042] In the following, reference will be made to Figures 2 to 6 The safety monitoring mechanism is described in detail. The safety monitoring mechanism is implemented by a cooperative operation between the robot and the transport device. In this mechanism, the control operation that is performed in response to the detection of a human in the range can be determined by Figure 2 the robot as illustrated in

[0043] Figure 2 A signaling diagram of an example process 200 for safety monitoring according to embodiments of the present disclosure is schematically illustrated. For the purpose of discussion, reference will be made to Figure 1 The process 200 will be described. The process 200 can be implemented between the robot 110 and the transport device 130 in Figure 1

[0044] As Figure 2 ​As shown, at 202, the transport device 130 collects sensor data, e.g., with the first laser scanner 132 and the second laser scanner 132. At 204, after acquiring the sensor data, the transport device 130 transmits the sensor data 205 to the robot 206. At 206, the robot 110 receives the sensor data 205. At 208, the robot 110 determines a control operation based on the received sensor data. For example, a human in the vicinity of the robot can be detected by utilizing at least one laser scanner already equipped on the transport device. In some example embodiments, the human can be identified by a pattern derived from the sensor data. For example, if the width of an object detected by the sensor falls within a predefined width range of a human, the controller can determine that a human is detected. Alternatively, a speed of the detected object can be derived from the sensor data collected at time intervals. In this case, when the derived speed falls within a predefined range of human speed, the controller can determine that a human is detected. It should be appreciated that a human can also be detected by other suitable techniques, e.g., with the aid of machine vision.

[0045] In the case where two laser scanners are provided at both the front side and the rear side of the transport means, the distance between the human and the robot can be calculated by: (1) wherein denotes the position of the first laser scanner at the front side, denotes the position of the human, denotes the position of the robot, denotes the position of the second laser scanner at the rear side. Thus, denotes the vector from the robot to the first laser scanner, denotes the vector from the first laser scanner to the human, and denotes the vector from the robot to the second laser scanner, denotes the vector from the second laser scanner to the human. In this case, the distance is the minimum of a first distance indicated by the vector derived from the sensor data collected by the first laser scanner and a second distance indicated by the vector derived from the sensor data collected by the second laser scanner.

[0046] After determining the distance between the human and the robot, the robot 110 uses the determined distance to determine a control operation. For example, there are two predefined distance thresholds, both of which are greater than the maximum working range R1 of the robot 110, including a first distance threshold R2 and a second distance threshold R3. The control operation can be determined according to the following criteria. For example, if D > R 3 the robot can maintain a full maximum operating speed. If R3 ≥ D > R 2 then the robot needs to reduce the operational speed. If R 2 ≥ D then the robot should immediately stop.

[0047] In this case, the control operation is an operation performed by the robot 110 under the control of the controller 111 in response to the detection of a human being in the vicinity of the robot 110. At 210, after determining the suitable control operation, the robot 210 performs the control operation.

[0048] For both embodiments, by exploiting the sensor data collected by the laser sensors of the transport device, the distance between the human being and the robot can be derived, and a suitable control operation can be determined based on the derived distance. In this way, no additional sensors are required, thereby reducing the cost of the robot. Reference will be made to Figures 3A to 3C The method for determining the control operation is described in detail.

[0049] Figure 3A A flowchart of a method 300 for determining a control operation according to an embodiment of the present disclosure is schematically illustrated. The method 300 can be implemented by the controller 111 of the robot 110 in Figure 1 .

[0050] As illustrated in Figure 3A , at 310, the controller 111 acquires sensor data. In the case of the controller 111, the robot 110 can acquire the sensor data from the transport device 130 via wireless communication. In this case, the sensor data is collected by at least one sensor disposed on the transport device 130, which is initially configured to detect obstacles along a predefined route of the transport device 130 to avoid collisions between the transport device 130 and the obstacles.

[0051] At 320, the controller 111 determines the presence of a human being based on the sensor data. If the controller 111 determines that no human being is detected, the method 300 will return to 310 and acquire the sensor data, e.g. according to a predefined interval. In contrast, if the controller 111 determines that a human being is detected, the method 300 will proceed to 330. At 330, the controller 111 determines the relative position of the human being with respect to the robot based on the sensor data. Then, at 340, the controller 111 determines the control operation based on the relative position. In the following, the method for determining the relative position and the method for determining the control operation will be described in detail with reference to Figure 3B and Figure 3C .

[0052] Figure 3BA flowchart of a method 330 for determining a relative position of a human with respect to a robot according to embodiments of the present disclosure is schematically shown. The method 330 can correspond to step 330 as shown in Figure 3A and implemented by the robot 110 in Figure 1 or the controller 131 of the transport device 130, in particular the controller 111 of the robot 110 or the controller 131 of the transport device 130. Figure 1

[0053] As shown in Figure 3B at 331, the controller 111 determines at least one human position of the human with respect to the at least one sensor based on the sensor data. In some example embodiments, the transport device can comprise a single sensor, and the controller can determine a single human position between the sensor and the human. Alternatively, the transport device can comprise a plurality of sensors, and the controller can determine a plurality of human positions between the human and respective ones of the plurality of sensors. In this case, each of the detected human positions is collected by a respective one of the plurality of sensors.

[0054] At 332, the controller 111 determines at least one sensor position of the at least one sensor. In some example embodiments, the sensor position can be a global coordinate of the sensor in a coordinate system of the industrial facility. Alternatively, the sensor position can be a coordinate of the sensor with respect to any reference point in the facility.

[0055] At 333, the controller 111 determines at least one human vector between the at least one sensor and the human based on the at least one human position and the at least one sensor position. In some example embodiments, the at least one human vector can be derived from a coordinate of the human and at least one coordinate of the at least one sensor in the same coordinate system.

[0056] At 334, the controller 111 determines at least one sensor vector between the at least one sensor and the robot based on the at least one sensor position and a robot position of the robot. In some example embodiments, the at least one sensor vector can be derived from the at least one coordinate of the at least one sensor and a coordinate of the robot in the same coordinate system.

[0057] ​At 335, the controller 111 determines at least one detection vector based on the at least one human vector and the at least one sensor vector. In the case of multiple sensors, the detection vectors derived from different sensor data collected by different sensors can differ due to tolerances and operating conditions of the sensors, which can differ from one another. At 336, after the detection vectors are derived, the controller 111 determines a number of distances indicated by the at least one detection vector. If the controller 111 determines that the at least one detection vector indicates multiple distances, i.e., more than one distance, the method 330 proceeds to 337. At 337, the controller 111 determines the detection vector that indicates the smallest distance among the multiple distances to represent the relative position between the human (e.g., the human 160) and the robot 110.

[0058] If the controller 111 determines that the at least one detection vector indicates a single distance, the method 330 proceeds to 338. For example, the at least one detection vector includes only one detection vector. Alternatively, the at least one detection vector includes multiple detection vectors, and the differences between the distances indicated by the multiple detection vectors are within a predefined tolerance range. At 338, the controller 111 determines any one of the at least one detection vector to represent the relative position between the human and the robot 110.

[0059] For these embodiments, the relative position can be acquired among the at least one sensor as appropriate, thereby avoiding the influence of interference or unfavorable operating conditions.

[0060] Figure 3C A flowchart of a method 340 for determining a control operation based on a relative position according to embodiments of the present disclosure is schematically shown. The method 340 can correspond to the step 340 as shown in Figure 3A and be implemented by the robot 110 in Figure 1 or the controller 111 of the robot 110 or the controller 131 of the transportation device 130 in Figure 1 .

[0061] As shown in Figure 3C , at 341, the controller 111 determines a distance between the robot 110 and the human based on the relative position, in this case, the distance between the robot 110 and the human 160. At 342, the controller 111 acquires at least one distance threshold associated with a working range of the robot 110. In some example embodiments, the at least one distance threshold can include two distance thresholds.

[0062] At 343, the controller 111 compares the distance to a first distance threshold of the at least one distance threshold. In this case, the first distance threshold can be associated with the maximum working range 140 of the robot 110 or the maximum length of the robot’s arm. In some example embodiments, the first distance threshold is equal to or greater than the maximum length of the robot’s arm.

[0063] If the controller 111 determines that the distance is less than the first distance threshold, the method will proceed to 344. At 344, the controller 111 determines a pause of the current operation of the robot 110 as the control operation. In other words, if the controller 111 determines that the human 160 is close enough to the robot 110, e.g. enters a first zone defined by the first distance threshold, the controller 111 can stop the robot 110 and put the current operation sequence on hold. In some example embodiments, the controller 111 can resume the put-on-hold operation when the controller 111 determines that the human 160 walks out of the defined first zone.

[0064] If the controller 111 determines that the distance is not less than the first distance threshold, the method will proceed to 345. At 345, the controller 111 compares the distance to a second distance threshold. In this case, the second distance threshold is greater than the first distance threshold. In some example embodiments, the second distance threshold is selected such that when the human 160 enters a zone between the first distance threshold and the second distance threshold, the human 160 cannot enter the working zone of the robot at the next detection time point. If the controller 111 determines that the distance is less than the second distance threshold, the method will proceed to 346. At 346, the controller 111 determines a reduction of the operation speed of the robot 110 as the control operation. If the controller 111 determines that the distance is not less than the second distance threshold, the method will proceed to 347. At 347, the controller 111 determines to maintain the current operation of the robot 110 as the control operation.

[0065] For these embodiments, a suitable control operation can be determined according to a comparison between the distance indicated by the relative position and a set of predefined distance thresholds.

[0066] Figure 4 A schematic diagram of an example robot system 400 according to some embodiments of the disclosure is schematically illustrated. As Figure 4As shown, the robotic system 400 includes a robot 410 deployed in a workstation 420. The robot 410 includes a controller 411 configured to control operation of the robot 410. The robotic system 400 further includes a transport device 430 docked to a side of the robot 410. In the illustrated embodiment, the transport device 430 is an automated guided vehicle (AGV). The AGV 430 includes a controller 431 configured to control movement of the AGV 430 in the industrial facility. The AGV 430 further includes a first laser sensor 432 and a second laser sensor 433. The detection ranges of the first laser sensor 432 and the second laser sensor 433 can both cover a wide range that encompasses the working range of the robot 110. In this case, the controller 431 can operate with the first laser sensor 432 and the second laser sensor 433 to implement a simultaneous localization and mapping (SLAM) technique to determine the location of the AGV 430.

[0067] The location of the robot 410 can be expressed in earth coordinates in the facility coordinate system and represented as The AGV 430 can automatically travel along a predefined route and navigate according to its location in the industrial facility, which is expressed in global coordinates in the facility coordinate system.

[0068] In the illustrated embodiment, the AGV 430 is configured to transport materials to the workstation 420. Thus, the controller 431 knows the location of the robot 410 and directs the AGV 430 to travel to the robot according to the locations of the AGV 430 and the robot 410. Further, the AGV 430 continually transmits sensor data collected by the first laser sensor 432 and the second laser sensor 433 to the robot 410. In this way, when the AGV 430 detects the human 461, the robot 110 can also identify the human 461 based on the received sensor data.

[0069] In some example embodiments, after the robot 410 receives the sensor data and identifies the human 461, the controller 411 of the robot 410 can perform a method for determining a control operation as shown in Figures 3A to 3C After determining the control operation, the controller 411 can cause the robot 410 to perform the determined control operation. Alternatively, when the AGV 430 detects the human 461, the controller 431 of the AGV 430 can perform a method for determining a control operation as shown in Figures 3A to 3C After determining the control operation, the AGV 430 can transmit instructions to the robot 410 to cause the robot 410 to perform the determined control operation.

[0070] Hereinafter, the process for determining the control operation will be described in detail. The controller 411 or 431 determines a first human position of the human 461 with respect to the first laser sensor 432 and a second human position of the human 461 with respect to the second laser sensor 433 from the received sensor data. In this case, the first human position can be a coordinate in a coordinate system of the first laser sensor 432 and expressed as . Similarly, the second human position can be a coordinate in a coordinate system of the second laser sensor 433 and expressed as . To determine the relative position between the human 461 and the robot 410, the controller 411 or 431 determines a first sensor position of the first laser sensor 432 and a second sensor position of the second laser sensor 433. For example, the first sensor position can be a terrestrial coordinate in a facility coordinate system and expressed as , and the coordinate of the first sensor position in the coordinate system of the first sensor can be expressed as . The second sensor position can be a terrestrial coordinate in the facility coordinate system and expressed as , and the coordinate of the second sensor position in the coordinate system of the first sensor can be expressed as .

[0071] Then, the first global coordinate of the human associated with the sensor data collected by the first sensor can be determined by the following equation assuming that the first laser sensor is located at the origin of the coordinate system of the first laser sensor 432 : (2) (3)

[0072] Similarly, the second global coordinate of the human associated with the sensor data collected by the second sensor can be determined by the following equation assuming that the second laser sensor 433 is located at the origin of the coordinate system of the second laser sensor 433 : (4) (5)

[0073] The first human vector from the human 461 to the first laser sensor 432 is determined by the controller 411 or 431 as follows : (6)

[0074] The second human vector from the human 461 to the second laser sensor 433 can be determined as follows (7)​

[0075] The controller 411 or 431 then determines a first sensor vector from the first sensor 432 to the robot 410 as follows : (8).

[0076] The controller 411 or 431 then determines a second sensor vector from the first sensor 432 to the robot 410 as follows : (9)

[0077] The controller 411 or 431 then determines a first detection vector from the human 461 to the robot 410 as follows : (10)

[0078] The controller 411 or 431 then determines a second detection vector from the human 461 to the robot 410 as follows : (11)

[0079] In this way, a first detection vector is determined that represents the relative position between the human 461 and the robot 410 and a second detection vector is also determined that represents the relative position between the human 461 and the robot 410 Although both of these resulting vectors represent the same relative position, these two detection vectors are determined based on sensor data collected by two different laser sensors. Thus, due to errors or tolerances caused by either sensor, it is possible that the resulting detection vectors are different. In this case, a suitable relative position needs to be selected.

[0080] The controller 411 or 431 determines a first distance of the first detection vector and a second distance of the second detection vector If the controller 411 or 431 compares the first distance to the second distance If the controller 411 or 431 determines that the first distance to the second distance is the same, either of them can be determined as the relative distance between the human 461 and the robot 410 and the corresponding detection vector will be determined as representing. If the first distance is less than the second distance , for safety reasons, the corresponding first detection vector to represent relative distance.

[0081] After determining the relative position between the human 461 and the robot 410, the control operation can be determined according to the method as shown in Figure 4 In this embodiment, the proximity area 440 around the robot 410 is divided into 4 areas. The first area is the area inside a circle 441 corresponding to the maximum working area of the robot 410. In this case, the radius R1 of the circle 441 corresponds to the maximum extension of the robot arm. When a human enters the first area, it can not be possible to avoid collision with the robot. Therefore, the operation of the robot must be stopped before the human gets too close to the robot. Thus, a first distance threshold R2 is selected to be greater than the radius R1, and a second area is shown as the area between the circle 442 with a radius equal to the first distance threshold R2 and the circle 441. When it is determined that the human is inside the circle 442, the controller 411 will control the robot 410 to stop immediately. When it is determined that the human has left the circle 442, the robot 410 can resume its operation. Outside the circle 442, there is another circle 443. The radius of the circle 443 corresponds to a second distance threshold R3 which is greater than the first distance threshold. When it is determined that the human is inside the third area between the circle 442 and the circle 443, the controller 411 will control the robot 410 to reduce the operation speed to a predefined range, so that if the human moves further towards the robot 410, the robot 410 can stop in time. In other cases, when it is determined that the human is in the fourth area outside the circle 443, the robot 410 can restore its speed to the normal operation speed.

[0082] As shown in Figure 5 , the controller 411 or 431 determines that the first distance of the human 461 is greater than the second distance threshold R3. In this case, the human 461 is outside the circle R3, and the controller 411 or 431 determines that the control operation associated with the human 461 is to maintain the operation of the robot 110. At the same time, the AGV 430 also detects another human 462. By performing the same method, the controller 411 or 431 determines that the distance between the human 462 and the robot 110 is greater than the first distance threshold and less than the second distance threshold. The controller 411 or 431 determines that the control operation associated with the human 462 is to reduce the operation speed of the robot 410.

[0083] In the scenario where the transport device has detected more than one human, the control operation can be determined based on the distance of the human closest to the robot.

[0084] Figure 5 A schematic diagram of an example robot system 500 according to some further embodiments of the disclosure is schematically illustrated. As Figures 3A to 3CAs shown, the robotic system 500 includes a robot 510 deployed on a transport device 530. In the illustrated embodiment, the transport device 530 is an autonomous mobile robot (AMR). The robot 510 includes a controller 511 configured to control operation of the robot 510, and the AMR 530 includes a controller 531 configured to control movement of the AMR 530 in an industrial facility. The AMR 530 can automatically travel along predefined routes and transport the robot 510 to a plurality of workstations arranged in the industrial facility. The AMR 530 navigates according to its location in the industrial facility.

[0085] The AMR 530 further includes a first laser sensor 532 and a second laser sensor 533. The detection ranges of the first laser sensor 532 and the second laser sensor 533 can both cover a wide range that encompasses a working range of the robot 510. In this case, the controller 531 can operate with the first laser sensor 532 and the second laser sensor 533 to implement a simultaneous localization and mapping (SLAM) technique to determine a location of the AMR 530 as well as a location of the robot 510.

[0086] In the illustrated embodiment, the AMR 530 continuously collects sensor data with the first laser sensor 532 and the second laser sensor 533. In some example embodiments, the AMR 530 and the robot 510 can share the same controller. That is, the controller 511 and the controller 531 are the same controller that performs control of both the AMR 530 and the robot 510. In these embodiments, such a controller can determine a presence of a human based on the collected sensor data, and can perform the method for determining a control operation as shown in Figure 5 After determining the control operation, the controller can further cause the robot 510 to perform the determined control operation.

[0087] In the following, the process for determining a control operation will be described in detail. The controller 511 or 531 determines a first human position of the human 561 relative to the first laser sensor 532 and a second human position of the human 561 relative to the second laser sensor 533 from the received sensor data. Since the AMR 530 transports the robot 510, the location of the robot 510 can be considered the same as the location of the AMR 530. Therefore, for simplicity, the relevant positions can all be represented by coordinates in a coordinate system of the AMR 530. In this case, the first human position and the second human position can be coordinates in a coordinate system of the AMR 530. In this way, a coordinate transformation from the coordinate system of the AMR 530 to a global coordinate system can be omitted. The first human vector from the human 562 to the first laser sensor 532 can be directly derived from the collected sensor data and a second human vector from the human 562 to the second laser sensor 533 .

[0088] Since the robot 510 is fixed relative to the first laser sensor 532 and the second laser sensor 533, a first sensor vector from the first sensor 432 to the robot 410 can also be obtained directly and a second sensor vector from the first sensor 432 to the robot 410 .

[0089] Then, the controller 511 or 531 determines a first detection vector from the human 562 to the robot 510 as follows : (12)

[0090] Then, the controller 411 or 431 determines a second detection vector from the human 562 to the robot 510 as follows : (13)

[0091] In this way, a first detection vector is determined that represents the relative position between the human 562 and the robot 510 and a second detection vector is also determined that represents the relative position between the human 562 and the robot 510 . Accordingly, since the first distance of the first detection vector is the same as the second distance of the second detection vector , the corresponding first detection vector is selected to represent the relative distance.

[0092] In this embodiment, the adjacent area 540 around the robot 510 is also divided into 4 regions, including a first region inside a circle 541 with a radius equal to the maximum length R1 of the arm of the robot 510, a second region between the circle 541 and a circle 542 with a radius R2, a third region between the circle 542 and a circle 543 with a radius of a second distance threshold R3, and a fourth region outside the circle 543.

[0093] As shown in Figure 6 , the controller 511 or 531 determines that the first distance of the human 562 is less than the first distance threshold R2. That is, the human 562 is inside the circle 542. The controller 511 or 531 determines that the control operation associated with the human 562 is to immediately stop the operation of the robot 110. Then, the controller 511 controls the robot 510 to stop.

[0094] Figure 6 A schematic diagram of an example robot system 600 is shown schematically illustrating yet some further embodiments according to the present disclosure. As Figures 3A to 3C shown, the robot system 600 comprises a robot 610-1 deployed at a workstation 620-1 and a robot 610-2 deployed at a workstation 620-2. The robot system 600 further comprises a transport device 630 configured to transport material to be processed to the associated robot. The transport device 630 comprises a controller 631 configured to control the transport device 630 to travel along a predefined route 660. As shown, the predefined route 660 is configured to sequentially pass the workstation 620-2 and the workstation 620-1.

[0095] In operation, the transport device 630 first transports material to the workstation 620-2 and docks at one side to allow the robot 610-2 to pick up its material. After the operation at the workstation 620-2 is completed, the transport device 630 travels along the route 660 and transports the remaining material to the workstation 620-1 to allow the robot 610-1 to pick up its material. For navigation, the transport device 630 further comprises a first laser sensor 632 and a second laser sensor 633. Both a first detection range 651 of the first laser sensor 632 and a second detection range 652 of the second laser sensor 633 can cover a wide range encompassing a proximity area 640-1 of the robot 610-1 and a proximity area 640-2 of the robot 610-2.

[0096] During operation, the transport device 630 is associated and communicates with the robot 610-1 and the robot 610-2. The transport device 630 continuously transmits sensor data collected with the first laser sensor 632 and the second laser sensor 633 to the robot 610-1 and the robot 610-2 to allow respective controllers of the robot 610-1 and the robot 610-2 to determine whether a human being is too close to the respective robot according to the method as Figure 4 shown. For example, the transport device 630 can transmit sensor data regarding the human being 660 to the robot 610-1 and the robot 610-2. The robot 610-1 can determine that the human being 660 is in the third zone as discussed with reference to Figure 5 and Figure 4 and determine to reduce the operation speed. In contrast, the robot 610-2 can determine that the human being 660 is in the fourth zone as discussed with reference to Figure 5 and Figure 7 and determine to maintain the current operation. In this way, more than one sensor can be saved such that the cost of the robot system can be further reduced.

[0097] It will be appreciated that the robotic system can include any number of robots and any number of transport vehicles capable of implementing the safety monitoring mechanisms of the present disclosure.

[0098] In some embodiments of the present disclosure, a computing device for implementing the above-described methods 300, 330, and 340 is provided. Figure 1 A schematic of a controller 700 for implementing the methods according to embodiments of the present disclosure is shown. The controller 700 can correspond to Figure 1 the controller 111 of the robot 110 in Figures 3A to 3C the controller 131 of the transport device in The controller 700 includes at least one processor 710 and at least one memory 720. The at least one processor 710 can be coupled to the at least one memory 720. The at least one memory 720 includes instructions 722 that, when executed by the at least one processor 710, implement the method 300, 330, or 340.

[0099] In some embodiments of the present disclosure, a computer readable medium for adjusting a robot path is provided. The computer readable medium has stored thereon instructions and the instructions, when executed on at least one processor, can cause the at least one processor to perform the method for managing a camera system as described in the preceding paragraphs and details will be omitted hereafter.

[0100] In general, the various embodiments of the present disclosure can be implemented in hardware or special-purpose circuits, software, logic or any combination thereof. Some aspects can be implemented in hardware, while other aspects can be implemented in

[0101] The present disclosure also provides at least one computer program product tangibly stored on a non-transitory computer readable storage medium. The computer program product includes computer executable instructions, such as those included in program modules, executed by devices on a target real or virtual processor to perform the methods as described above with reference to ​The processes or methods described herein can be implemented using the systems described herein. As will be apparent, however, embodiments can be practiced according to other system configurations, and other programmatic arrangements will be conceivable to those skilled in the art. The processes or methods depicted can be performed or implemented with one or more programs or pieces of code, which can be embodied in machine readable medium. In this regard, the programs can be implemented in a high level procedural or object oriented programming language to communicate with a computer system. The programs can also be implemented in assembly or machine language, if desired. The language can be a compiled or interpreted language and combined with hardware implementations. The program code can be supplied from an external source, electronic, optical, mechanical or other storage medium, in a machine readable medium, to a data processing system employed to implement the system and methods of the present disclosure. Alternatively, the systems and methods can be embodied in a propagation signal, which carries or communicates program code in a data signal that can be embodied in either analog or digital form in a variety of signal-bearing media.

[0102] Program code used by or in connection with the described embodiments, when implemented in software can be stored in various portions of volatile or non-volatile storage. The memory or memories can also store one or more operating systems, database management systems, and computing platforms that can also include components for accessing the memory, and programs associated therewith. The various embodiments further can include receiving, sending or storing instructions and / or data inherited from one or more remote computers or devices over a network. In this regard, the various embodiments can be implemented using any hardware, software, systems, or collections thereof, known now or later developed that can perform the functionality disclosed herein.

[0103] The program code can be embodied in machine readable medium, which can be any tangible medium that can contain, or store the program for use by or in connection with an instruction execution system, apparatus, or device. The machine readable medium can be a machine readable signal medium or a machine readable storage medium. Machine readable storage medium can include but not limited to electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the machine readable storage medium will include an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0104] Further, while operations are depicted in a particular order, this should not be understood as requiring such an order, or that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing can be advantageous. Likewise, while a number of specific embodiments have been described, these should be considered as merely illustrative of the many possible specific embodiments that fulfill the criteria set forth in the appended claims. Certain features, which are, for clarity, described under separate headings of embodiments, can also be implemented in combination. Conversely, various implementations of the features under a single heading can also be implemented independently of one another.

[0105] Although the subject matter has been described in language specific to structural features and / or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are disclosed as example forms of implementing the claims.

[0106] It should be understood that the above detailed description of the present disclosure is only for illustration or explanation of the principles of the present disclosure, and does not limit the present disclosure. Therefore, any modification, equivalent alternative and improvement, etc. shall be included in the protection scope of the present disclosure without departing from the spirit and scope of the present disclosure. Meanwhile, the appended claims of the present disclosure are intended to cover all changes and modifications falling within the scope and boundary or equivalents of the claims.

[0107] It should be understood that the summary section is not intended to identify key or essential features of embodiments of the present disclosure, nor is it intended to be used to limit the scope of the present disclosure. Other features of the present disclosure will become readily apparent from the following description.

Claims

1. A method for safety monitoring, comprising: obtaining sensor data collected by at least one sensor on a transport device associated with a robot; detecting a presence of a human based on the sensor data; determining a relative position of the human with respect to the robot based on the sensor data; and determining a control operation for the robot based on the relative position.

2. The method of claim 1, further comprising: receiving, by the robot, the sensor data from the transport device; and performing, by the robot, the control operation after determining the control operation.

3. The method of claim 1, wherein determining the relative position comprises: determining at least one human position of the human with respect to the at least one sensor based on the sensor data; determining at least one sensor position of the at least one sensor; and determining the relative position based on the at least one sensor position and the at least one human position.

4. The method of claim 3, wherein determining the relative position based on the at least one sensor position and the at least one human position comprises: determining at least one human vector between the at least one sensor and the human based on the at least one human position and the at least one sensor position; determining at least one sensor vector between the at least one sensor and the robot based on the at least one sensor position and a robot position of the robot; determining at least one detection vector based on the at least one human vector and the at least one sensor vector; and determining one of the at least one detection vector that represents the relative position between the human and the robot in response to determining that the at least one detection vector indicates a single distance between the human and the robot.

5. The method of claim 4, wherein determining the relative position based on the at least one sensor position and the at least one human position further comprises: determining a relative position that indicates a minimum distance of a plurality of distances representing the relative position between the human and the robot in response to determining that the at least one relative position indicates a plurality of distances between the human and the robot.

6. The method of claim 1, wherein determining the control operation comprises: determining a distance between the robot and the human based on the relative position; obtaining at least one distance threshold associated with a working range of the robot; and determining the control operation based on the distance and the at least one distance threshold.

7. The method of claim 6, wherein determining the control operation based on the distance and the at least one distance threshold comprises: comparing a first distance threshold associated with a maximum working range of the robot to the distance in response to determining that the at least one distance threshold includes the first distance threshold; and determining the control operation to pause operation of the robot in response to determining that the distance is less than the first distance threshold. ​ ​ ​ ​ ​ ​ 8. The method of claim 7, wherein determining the control operation based on the distance and the at least one distance threshold further comprises: in response to determining that the distance is greater than the first distance threshold, comparing the distance to a second distance threshold that is greater than the first distance; in response to determining that the distance is less than the second distance threshold, determining the control operation to be to reduce an operational speed of the robot.

9. The method of claim 8, wherein determining the control operation based on the distance and the at least one distance threshold further comprises: in response to determining that the distance is greater than the second distance threshold, determining the control operation to be to maintain the operation of the robot.

10. The method of claim 1, wherein the transport device comprises an automated guided vehicle (AGV) or an autonomous mobile robot (AMR).

11. The method of claim 10, wherein the transport device communicates with the robot wirelessly or through a cable.

12. The method of claim 1, wherein the robot is integrated on the transport device.

13. A robot, comprising: at least one controller; and at least one memory storing instructions that, when executed by the at least one controller, cause the robot to perform the method of any one of claims 1 to 12.

14. A robotic system, comprising: at least one robot of claim 13 configured for processing material; and at least one transport device comprising at least one sensor configured for detecting a human, and the at least one transport device is configured for transporting material to at least one of the robots.

15. A computer readable medium having stored thereon instructions that, when executed on at least one processor, cause the at least one processor to perform the method of any one of claims 1 to 12.

Images