Method for generating a power and force limiting zone for a robot

Abstract

The disclosure relates to a method (100) for generating a power and force limiting, PFL, zone (10) for a robot (60), the method comprising: - obtaining hazard zone data of the robot (60), wherein the hazard zone data is indicative of a potential contact area (11) between the robot (60) and a person (1), - obtaining image data, wherein the image data is indicative of an operational environment (20) of the robot (60); - generating a spatial representation (21) of the operational environment (20) of the robot (60), based on the image data, wherein the spatial representation (21) is segmented into spatial elements (22), wherein each spatial element (22) comprises information about an occupancy state (01 to 03; F1 to F3), depending on whether the spatial element (22) is occupied (01 to 03) by an object or unoccupied (F1 to F3); - determining a safety parameter for each of the spatial elements (22) based on the spatial representation (21) of the operational environment (20) of the robot (60) and the hazard zone data, wherein a higher potential risk for the person (1) is assumed for an occupied (01 to 03) area and a lower potential risk for the person (1) is assumed for an unoccupied (F1 to F3) area; and - generating a PFL zone (10) based on the determined safety parameter, wherein the safety parameter comprises at least one or more of the following:  a maximum speed within the PFL zone (10),  a maximum force limitation within the PFL zone (10),  a maximum pressure limit within the PFL zone (10),  a directional movement restriction within the PFL zone (10),  a response time for hazard detection and reaction within the PFL zone (10).

Description

[0001] - 1 - 18 October 2024

[0002] ABB Schweiz AG A19330WO

[0003] METHOD FOR GENERATING A POWER AND FORCE LIMITING ZONE FOR A ROBOT

[0004] TECHNICAL FIELD

[0005] The present invention relates to a method for generating a power and force limiting (PFL) Zone for a robot, one or more computer program products, a data processing system, and a robot.

[0006] BACKGROUND

[0007] Robots are increasingly utilized in industrial production environments where they work directly alongside human operators. Ensuring the safety of human-robot interactions is paramount, necessitating effective safety measures to mitigate potential contact hazards between humans and robots.

[0008] Current methods for configuring PFL zones predominantly rely on manual inputs provided by users through graphical user interfaces (GUIs). In these existing approaches, users must specify the coordinates of safety zones and configure relevant parameters themselves, a process that is both time-consuming and requires substantial expertise in robotics and safety engineering. Systems such as Polyscope, TMflow, YRC 1000 / FSU, ROBOGUIDE, and DCS exemplify these manual configuration methods, necessitating a deep understanding of the manipulator’s structure and underlying coordinate systems. This reliance on manual configuration not only increases the complexity and time required for setup but also introduces the risk of human error, potentially leading to inadequate safety measures or overly conservative settings that hinder operational efficiency.

[0009] Moreover, the efforts involved in risk assessment and safety configuration are still considered challenging, excessive, or even unnecessary by some users. Main complaints include the complexity of the task, the required expertise to manage it with today’s tools, and the lack of guidance in the configuration systems themselves. For instance, to configure a speed limit in a particular zone of the robot’s workspace, it is necessary to specify the coordinates of the zone, either using a CAD-style robot application simulation tool or in plain text via a text editor based on a mental model of the robot application's coordinate system, as well as the desired speed limit value, assuming that the integrator has determined this value through separate considerations. Additionally, consulting standards documents, such as ISO / TS 15066, for guidance on sufficient risk reduction further complicates the task.

[0010] SUMMARY

[0011] The above problem or need is at least partially solved or alleviated by the subject matters of the independent claims of the present disclosure, wherein further examples are incorporated in the dependent claims.

[0012] According to an aspect of the present disclosure, there is provided a method for generating a power and force limiting, PFL, Zone for a robot, the method comprising: obtaining hazard zone data of the robot, wherein the hazard zone data is indicative of a potential contact area between the robot and a person, obtaining image data, wherein the image data is indicative of an environment of the robot; generating a spatial representation of the environment of the robot, based on the image data, wherein the spatial representation is segmented into spatial elements, wherein each spatial element comprises information about an occupancy state, depending on whether the spatial element is occupied by an object or unoccupied; determining a safety parameter for each of the spatial elements based on the spatial representation of the working area and the hazard zone data, wherein a higher potential risk for the person is assumed for an occupied area and a lower potential risk for the person is assumed for an unoccupied area; and generating a PFL zone based on the determined safety parameter, wherein the safety parameter comprises at least one or more of the following: a maximum speed within the PFL zone, a maximum force limitation within the PFL zone, a maximum pressure limit within the PFL zone, a directional movement restriction within the PFL zone, a response time for hazard detection and reaction within the PFL zone.

[0013] The method of the first aspect may in particular be at least a partially or fully computer implemented method. This means that at least one, multiple or all of the steps of the method may be carried out by a data processing system, which may comprise one or more data processing apparatuses, which may be in the form of computers or computing units, which may comprise one or more processors and data storages or memories. Different steps may be carried out by the same or by different data processing apparatuses of the data processing system.

[0014] Generally, the power and force limiting, PFL, Zone may be understood as a defined area within a robot's workspace configured to regulate and restrict the robot's movements and interactions to ensure the safety of the person. Within a PFL Zone, the robot may operate under predefined safety parameters that limit its maximum speed, force, and pressure exertion, for example. Additionally, these zones may impose directional movement restrictions and define response times for hazard detection and reaction, thereby preventing excessive force or speed that could result in injury during accidental or incidental contact with a person. Hazard zone data may refer to information that defines potential contact areas between a robot and a person. This potential contact area may be within an operational environment of the robot, for example. Image data may encompass a diverse range of information acquired from various sensing devices employed to monitor and interpret the robot's operational environment. All types of sensors capable of generating depth information or 3D information are suitable for producing image data for this method. This may comprise sensors such as but not limited to 3D cameras, LiDAR sensors, ultrasonic sensors, or infrared sensors, for example. In particular, a 3D camera may be utilized to obtain the image data. Suitable types may include Stereo Cameras, Time-of-Flight (ToF) Cameras, or Structured Light Cameras, or similar. For example, a 3D model of the operational environment may be obtained with state-of-the-art techniques for 3D reconstruction using 2D cameras based on stereovision or image / video sequences, e.g., smartphone video-based monocular 3D reconstruction. In addition, the use of hand-held devices, such as but not limited to Lidar (Light Detection and Ranging), Mono-Cameras, Inertial Measurement Units (IMUs) equipped with a variety of sensors may be utilized to collect three-dimensional (3D) data of the operational environment of the robot. Image data collected using hand-held devices may be processed and integrated with other data sources to generate a comprehensive spatial representation of the operational environment, for example. The sensor may capture detailed spatial information about the robot's environment. The sensor may be provided at the robot in a position that maximizes its field of view while minimizing occlusions caused by the robot itself. Calibration may be performed to establish a spatial relationship between the sensor and the robot's coordinate system. This may ensure that the captured image data may be accurately mapped to the robot's operational environment. The sensor may continuously capture image data in the form of point clouds, which may represent the spatial positions of objects within the robot's environment. Each point in the cloud may correspond to a specific location in a three-dimensional space, for example. The sensor may capture image data on the robot's operational workspace and its surroundings. This may include both static objects (e.g., machinery, tools) and dynamic elements (e.g., human operators, moving objects). A robot-mounted camera may adapt its position and orientation to cover different areas as the robot moves, providing more flexible monitoring capabilities that static cameras may offer. The sensor may be temporarily mounted on the robot or picked up by the robot for generating image data. The sensor may also be moved or positioned manually to generate the image data. This may mean not by the robot itself, but by a human operator, for example. The image data may be pre-processed. For example, image data may contain noise and irrelevant information that may be filtered out. The preprocessing steps may comprise such as but not limited to, e.g. apply filtering algorithms such as the Random Sample Consensus (RANSAC) algorithm to remove outliers and reduce noise, simplify point clouds by eliminating unnecessary points to enhance processing speed, align the image data to a uniform coordinate system based on the sensor’s calibration. The pre-processed image data may be converted into a spatial representation divided into spatial elements, in particular voxels. Other examples for spatial elements may be Polygonal meshes, point clouds, Non-Uniform Rational B- Splines, octrees, and other suitable methods. A grid that comprises the robot’s operational environment and its immediate surroundings may be generated. Each point in the point cloud may be assigned to a corresponding spatial element based on its spatial coordinates, for example. Each spatial element within the grid may be classified based on its occupancy state. A spatial element may be marked as occupied if at least one point from the point cloud resides within it. This may indicate the presence of a physical object or obstacle in that area. A spatial element may be marked as unoccupied if no points from the point cloud are detected within it. This may indicate free space where no objects are present. Accurate classification of spatial elements may be facilitated through the application of algorithms. For example, threshold-based methods may be employed, where the number of points within a spatial element determines its state. Density-based approaches may further refine this classification by assessing the concentration of points, providing a more nuanced understanding of the environment. Additionally, machine learning techniques may be integrated to train models that recognize complex patterns within the point clouds, thereby improving the precision of occupancy determinations. Sensor fusion, which combines data from multiple sensors, may also be utilized to mitigate uncertainties and enhance the reliability of the occupancy state. The classified spatial elements may be continuously updated within an occupancy state, reflecting real-time changes and movements within the operational environment of the robot. This dynamic updating may ensure that PFL zones are consistently aligned with the current state of the operational environment, enhancing both safety and operational efficiency. For each spatial element within the spatial representation, the occupancy state in conjunction with the hazard zone data to determine the associated safety parameters may be assessed. The underlying principle may be that occupied areas, which indicate e.g., the presence of objects or a human body part, pose a higher potential risk for accidental contact or collision. Consequently, these areas may require stricter safety measures. In contrast, unoccupied areas, where no objects or human presence may be detected, may be considered to have a lower risk, allowing for more lenient safety parameters that do not unnecessarily restrict the robot’s operations. Algorithms and machine learning techniques may be employed to analyze the combined data from the spatial representation and hazard zones. These algorithms may evaluate various factors, including the proximity of a person, the nature of objects within occupied areas, and the robot’s current and planned movements, for example. By doing so, safety parameters may be dynamically adjusted, responding to changes in the environment and ensuring continuous adherence to safety standards such as ISO / TS 15066, for example. One of the primary challenges in this step may be accurately assessing the risk levels associated with each spatial element, especially in dynamic environments where a person’s movements and object placements may frequently change. To address this, real-time data processing and predictive modeling to anticipate potential hazards and adjust safety parameters proactively may be incorporated. Additionally, the integration of multiple sensors and data sources may enhance the reliability of occupancy detection, reducing the likelihood of misclassification and ensuring that safety measures are appropriately scaled to actual risk levels. Once the safety parameters for each spatial element are determined, such as maximum speed, force limitation, pressure limit, directional movement restrictions, and response time for hazard detection, these parameters may be integrated into the PFL zone. This PFL zone may be spatially mapped within the robot’s operational environment, with high-risk areas receiving stringent controls and low-risk areas allowing for more efficient robotic operations, for example. This approach may ensure that the robot performs its tasks with optimal efficiency while maintaining a high standard of safety for human collaborators, for example. Several other safety parameters may be incorporated to further enhance the safety and efficiency of PFL zones in collaborative robotic environments such as but not limited to maximum acceleration and deceleration, minimum safe distance, safe stopping distance, maximum accumulated force, torque, and rotational force limits. The method may incorporate any safety parameters relevant to collaborative robotic applications.

[0015] In an example, the hazard zone data may be determined by a swept volume of a trajectory of the robot and a safety margin. This hazard zone data may include information about the robot’s operational boundaries, movement trajectories, and areas where human-robot interactions are most likely to occur. The swept volume may refer to a three-dimensional space that the robot occupies as it moves along a predefined or programmed path. For example, the operational environment may be scanned by moving the robot along the programmed path while the sensor continuously captures image data. This may comprise all possible positions and orientations the robot’s moving parts may assume during operation. By calculating the swept volume, all areas within the robot's operational range where potential contact with a person or objects might occur may be identified. This comprehensive spatial coverage may ensure that no hazardous zones are overlooked, thereby providing a robust safety configuration. In addition to the swept volume, the safety margin may be integrated into the hazard zone determination to account for uncertainties and dynamic changes in the operational environment of the robot. The safety margin may serve as an additional buffer around the swept volume, accommodating factors such as the robot's stopping distance, potential deviations from the planned trajectory, and unforeseen movements of a person such as human operators or objects within the workspace. Further, a robot's braking distance at maximum speed may be incorporated into the determination of the hazard zone data to ensure safety in case of sudden stops, for example. The braking distance may be the distance a robot travels from the moment a braking command is issued until it comes to a complete stop. At maximum speed, this distance may be relevant as it represents the farthest point the robot can move in the event of an emergency stop. Thus, incorporating the braking distance may ensure that the hazard zones may minimize the risk of collision or injury during abrupt stops. By incorporating a safety margin, it may be ensured that even if the robot does not halt instantaneously upon detecting a hazard, the expanded hazard zone remains sufficient to prevent unsafe interactions. By defining the hazard zone through both the swept volume and the safety margin, a higher degree of safety and reliability may be ensured. The swept volume may provide a static representation of the robot’s operational path, while the safety margin introduces a dynamic element that adapts to real-time changes and uncertainties, for example. Furthermore, the integration of the swept volume and safety margin into the hazard zone data may facilitate the determination of safety parameters for each spatial element. Occupied areas within the swept volume, especially those augmented by the safety margin, may be identified as higher-risk zones requiring stricter safety measures. Conversely, unoccupied areas outside the safety margin may be recognized as lower-risk zones where more lenient safety parameters may be applied, optimizing both safety and operational efficiency.

[0016] In an example, the method may further comprise classifying the spatial elements based on their occupancy state and their spatial proximity to adjacent occupied areas into at least two contact types:

[0017] - free contacts: determining spatial elements as free contacts when the spatial element is unoccupied;

[0018] - constrained contacts: determining spatial elements as constrained contacts when the spatial element is occupied or is located within a predefined distance from an adjacent occupied spatial element; assigning the contact types based on the classified spatial elements, wherein free contacts are associated with a lower potential risk level for a person and constrained contacts are associated with a higher potential risk level for a person. For example, each spatial element within a potential contact area may contain a list of contact cases that represent various aspects of potential interactions between the robot and a person, such as a human operator. These contact cases may be divided into two main categories: Free contacts and constrained contacts. Free contacts may refer to collisions or physical interactions that may occur without obstacles or nearby objects, for example. These contacts may happen anywhere within the potential contact area since there are no immediate obstacles present that could restrict movement or cause the clamping of human body parts. Constrained contacts may refer to situations when the robot operates in close proximity to environmental objects, increasing the risk of clamping or pinching human body parts, for example. These contacts may arise when the robot moves near occupied spatial elements or within a predefined safety distance to these spatial elements. Spatial proximity to adjacent occupied areas may mean that a specific threshold distance may be established to determine the proximity of an occupied spatial element to adjacent occupied areas, for example. Algorithms may calculate the distance between occupied spatial elements and adjacent occupied areas. A predefined threshold distance may be set based on safety standards (e.g., ISO / TS 15066:2016) to determine the spatial proximity. Spatial elements classified as free contacts may be those that are unoccupied and are located outside a predefined distance from any adjacent occupied area, for example. These areas may pose minimal risk of accidental contact between the robot and a person, as there are no immediate obstacles or occupied areas that could result in entanglement or collision. Robots may operate more freely in these zones with higher permissible speeds and less restrictive force limitations. Spatial elements classified as constrained contacts may be either occupied or are located within a predefined distance from adjacent occupied areas. These areas may have a higher potential for contact-related hazards due to the proximity of obstacles or occupied zones that could lead to accidental interactions, for example. Robots may adhere to stricter safety parameters in these zones, including lower maximum speeds, force limitations, and directional movement restrictions to mitigate injury risks. Once spatial elements are classified into free or constrained contacts, each contact type may be associated with a corresponding risk level, wherein a lower risk potential may mean higher maximum speed, greater force allowances, and more lenient pressure limits may be assigned to these areas without compromising safety and a higher potential risk for a person may mean that stricter safety measures, including reduced maximum speeds, lower force limitations, tighter pressure constraints, and restricted directional movements, may be imposed to ensure safety in these high-risk zones.

[0019] In an example, the classification of the spatial elements may further comprise: Identifying a human body part within each spatial element based on the obtained image data; and considering the identified human body part in the classification of each spatial element into free contacts and constrained contacts, wherein different human body parts are associated with different risk levels and corresponding safety parameters. The identification process of a human body part may be e.g., employed by algorithms to accurately detect and classify human body parts within the captured image data. This may include automated detection such as machine learning algorithms which utilizes convolutional neural networks (CNNs) or other deep learning models trained on extensive datasets to recognize and segment specific human body parts from the image data, or image processing techniques such as edge detection, color segmentation, and shape recognition to identify and classify body parts. The identification process of human body parts may be performed manual, for example by user-based annotations or customizations. Once human body parts are identified, they may be mapped onto the spatial elements. Different body parts may possess varying degrees of vulnerability and risk exposure when interacting with robotic systems. For instance, hands and fingers may be more susceptible to injury compared to the torso or legs. This may allow for a differentiated application of safety measures that optimizes both the safety and operational efficiency of the robot. For example, a distance threshold may be determined based on normative dimensions of the potentially clamped human body parts. For example, if a body part could fit and may appear between the spatial element and the environmental object, a clamping hazard is identified.

[0020] In an example, the classification of the spatial elements may further comprise considering the robot's pose and motion. Considering the robot's pose may comprise determining a current position and orientation of the robot within the operational environment. The robot's pose may be continuously monitored using sensors such as encoders, inertial measurement units (I Mils), and robot-mounted cameras, for example. By knowing the robot's exact position and orientation, the spatial elements may be classified more precisely. For example, if the robot is oriented towards a specific area, the nearby spatial elements in that direction may be classified as constrained contacts due to the increased likelihood of interaction. Considering the motion direction may comprise determining a trajectory or planned movement path of the robot based on its current program or intended task. This may involve predicting the robot's future positions and movements within the operational environment. By knowing the robot's planned movements this may allow to proactively adjust the classification of spatial elements. For example, areas along the robot's trajectory may be classified as constrained contacts to enforce stricter safety parameters, while areas outside the path may remain as free contacts, permitting more flexible operation. By integrating the robot's pose and motion direction, the classification process may become more dynamic and responsive to real-time changes, for example. Spatial elements may be continuously re-evaluated as the robot moves, ensuring that safety parameters remain appropriate for the robot's current and planned activities.

[0021] In an example, generating the PFL zone may comprise grouping spatial elements having similar safety parameters. Grouping spatial elements with similar safety parameters may involve clustering areas that require comparable safety measures, such as maximum speed limits, force restrictions, and directional movement constraints, for example. Each spatial element's assigned safety parameters may be analyzed to identify commonalities. Parameters may include maximum speed, force limitations, pressure limits, directional restrictions, and response times, for example. Spatial elements may be grouped together if their safety parameters fall within predefined ranges or share specific characteristics, to define PFL zone geometry with corresponding speed and force limits, for example. Clustering algorithms, such as K- means, hierarchical clustering, or DBSCAN, may be employed to automate the grouping process. These algorithms may analyze the safety parameter data and determine optimal groupings based on similarity metrics. Each cluster may form a PFL zone. These PFL zones may be defined by the aggregated safety parameters of their constituent spatial elements. Grouping the spatial elements may reduce the complexity of managing numerous individual spatial elements by consolidating them into manageable zones. This simplification may facilitate monitoring and adjustment of safety parameters. In robotic safety applications, safety zones may be defined using a combination of two-dimensional (2D) spatial parameters supplemented by vertical boundaries (minimum and maximum heights). This approach may simplify the computational complexity while still providing adequate safety coverage. The 2D parameters typically include the X and Y coordinates on a plane, while the height criteria (Z-axis) define the vertical extent of the safety zone. However, it may be possible to add a height criterion which refines the classification by ensuring that safety parameters are appropriately adjusted based on the vertical position of potential human body parts or obstacles. For example, areas within the horizontal safety zone but outside the vertical safety boundaries may be treated differently in terms of safety parameters. This may reduce the processing load by limiting the primary analysis to two dimensions, which is less computationally intensive than full 3D analysis, for example.

[0022] In an example, a person may demonstrate human poses manually or through predefined movements within the robot's operational environment for spatial mapping of where a human body part is likely to interact with the robot’s potential contact area. This demonstration may serve as a step for calibrating and refining the robot's safety parameters based on actual human-robot interactions, for example. A manual demonstration may be a person physically moving and within the robot's operational environment, mimicking typical tasks and interactions that might occur during regular operations. A sensor, particularly an imaging sensor, may record these movements and poses, providing detailed spatial and motion data that reflect genuine human behaviors. In another example, a set of predefined movement patterns that simulate common human-robot interactions may be used. The demonstrated poses and movements may be analyzed to identify patterns and specific areas of frequent interaction between a person and the robot. For example, by moving the robot at a low speed while the person demonstrates various human body postures within the robot's operational environment, relevant body parts and potential hazard areas may be accurately identified.

[0023] In an example, generating the spatial representation of the robot's operational environment may comprise using point cloud data and the sensor's field of view to classify spatial elements into free spatial elements, occupied spatial elements, and unknown spatial elements. Point cloud data may provide a detailed three-dimensional map of the environment by capturing numerous data points that represent the surfaces of objects within the robot's operational environment. However, point cloud data does not provide information about spaces where no measurements are returned. This absence of data may result in an inability to distinguish between truly free space and areas that are not measurable due to factors such as occlusion or the inherent range limitations of the sensors. To address this challenge, the analysis of the sensor's field of view (FOV) in conjunction with the point cloud data may be utilized. By considering the FOV, additional context about the observable environment may be gained, allowing it to identify regions within the operational environment that are rendered as free, occupied, or remain unknown. Specifically, the FOV analysis may enable the classification of spatial elements into three distinct categories:

[0024] Free Spatial Elements: These may be regions within the operational environment that are unoccupied and lie outside a predefined distance from any adjacent occupied areas. The classification as free may indicate that these areas present a lower potential risk level for human operators, permitting the robot to operate with higher efficiency and fewer restrictions in these areas.

[0025] Occupied Spatial Elements: These spatial elements may be either currently occupied by objects or are situated within the predefined proximity of occupied areas. The classification as occupied signifies a higher potential risk level, necessitating stricter safety parameters to prevent collisions and ensure the safety of human operator. Unknown Spatial Elements: These may be regions where the point cloud data does not provide sufficient information due to occlusions or the limited range of the sensors. Unknown spatial elements may represent areas where it is unclear whether the space is free or occupied, thereby requiring additional safety measures or sensor adjustments to mitigate potential risks. By integrating the sensor's FOV analysis with point cloud data, the method may enhance the accuracy and reliability of the spatial representation. This approach may ensure that all areas within the robot's operational environment are appropriately classified, even in scenarios where direct measurements are incomplete or obstructed. Consequently, the system may dynamically adjust the PFL zones based on comprehensive and accurate spatial information, thereby optimizing both safety and operational efficiency in collaborative robotic applications. Further, the movements of the robot may be controlled such that the sensor avoids blind spots by calculating an optimal offset angle for the sensor. Blind spots may be areas within the robot's operational environment that are not adequately monitored by its sensor. Blind spots may occur due to the fixed positioning of sensors, limited field of view, or obstructions within the operational environment, for example. To minimize blind spots and ensure comprehensive coverage of the robot's environment, the sensor’s position and orientation may be dynamically adjusted. This may involve tilting, panning, or rotating the sensor based on the robot's movements and tasks. The sensor may be offset from the robot's primary movement path to capture areas that might otherwise be obscured. For instance, slight lateral or vertical offsets can enhance the camera's ability to monitor critical regions. Adaptive algorithms may control the camera's movement in real-time, responding to changes in the robot's environment or task requirements.

[0026] In an example, the occupancy state may be continuously or periodically updated. This may be important to account for changes in the environment or the robot's task. For example, the robot may continuously collect data from a sensor, such as a 3D camera, LiDAR, or proximity sensors, to monitor the operational environment in real-time. Algorithms may process incoming image data to detect changes in the environment, such as new obstacles, moving humans, or alterations in object positions. Updated occupancy states may lead to reclassification of spatial elements into free or constrained contacts based on their new status. Through continuous or periodic updates of the occupancy state, dynamically reconfiguration of the PFL zones to address changes in the environment or the robot's tasks, may be achieved.

[0027] In an example, the method may further comprise validating the generated PFL zone. The generated PFL zone may be compared against predefined safety standards to ensure that the PFL zone meets or exceeds relevant safety standards, such as ISO / TS 15066:2016, for example. This may include validating these zones through a series of simulations and physical tests. A user may provide input based on their observations and experiences, allowing for manual adjustments to the PFL zone to better align with specific safety requirements or operational preferences. In an example, generating the spatial representation of the operational environment of the robot may include using digital modelling data of the operational environment, wherein the digital modelling data comprises at least one or more of: computer-aided design (CAD) data, digital maps, and three-dimensional (3D) models. By utilizing CAD data or 3D reconstructions using any known techniques, more accurate 3D models of the workspace, including fixed structures, machinery, and layout configurations may be generated. This integration may allow for more accurate mapping and segmentation of the operational environment into spatial elements, facilitating better classification and safety zone generation.

[0028] In an example, the method may further comprise applying one or more algorithms to:

[0029] - minimize the number of PFL zones;

[0030] - maximize the operational efficiency of the robot; and

[0031] - set the safety parameters as high as possible without compromising safety. For example, to minimize the number of PFL zones algorithms, such as optimization algorithms, may analyze the spatial distribution and classification of all spatial elements within the robot’s operational environment. By identifying regions with similar safety requirements and proximity, the algorithms may consolidate adjacent or overlapping zones that may share uniform safety parameters. This consolidation may reduce the complexity of managing multiple zones, streamlines the safety framework, and ensures that safety measures are applied consistently across the operational environment. To maximize the operational efficiency of the robot, these algorithms may optimize the robot’s movement patterns and task execution strategies within the defined safety boundaries, for example. By intelligently navigating through the workspace, avoiding redundant or overly restrictive safety zones, and identifying optimal paths that minimize movement time and energy consumption, the robot may perform tasks more swiftly and effectively, for example. Furthermore, optimization algorithms may set the safety parameters as high as possible without compromising safety, for example. These algorithms may meticulously calibrate safety thresholds such as maximum speed, force limits, and pressure constraints by balancing the need for operational performance with stringent safety requirements. Utilizing image data from sensor inputs, environmental mapping, and human-robot interaction patterns, the algorithms may determine the highest feasible safety parameters that do not exceed risk levels defined by safety standards like ISO / TS 15066. This calibration may ensure that the robot operates at optimal performance levels, enhancing productivity while maintaining a safe working environment for human collaborators.

[0032] According to a second aspect of this disclosure, there are provided one or more computer program products comprising instructions which, when executed by one or more data processing apparatuses, cause the one or more data processing apparatuses to carry out the method of the first aspect of this disclosure.

[0033] The computer program products may be a computer program or computer programs as such, meaning a computer program consisting of or comprising program code to be executed by the data processing apparatus, in particular computer.

[0034] Alternatively, the one or more computer program products may be products such as data storages, in particular computer-readable data storage mediums, on which the computer programs may be temporarily or permanently stored.

[0035] According to a third aspect of this disclosure, there is provided a data processing system configured to carry out the method according to the first aspect of this disclosure.

[0036] According to a fourth aspect, there is provided a robot configured to carry out the method according to the first aspect of this disclosure. The term robot in the patent application is understood broadly and can include a variety of autonomous or semi- autonomous machines used in different environments. In the context of the invention described, the robot could take various forms such as Industrial Robots, Collaborative Robots (Cobots), Autonomous Mobile Robots (AMRs), Service Robots, Logistics and Delivery Robots, Agricultural Robots, Construction Robots, Robots for use in public spaces, for example.

[0037] It is noted that the above aspects, examples, and features may be combined with each other irrespective of the aspect involved. The above and other aspects of the present disclosure will become apparent from and elucidated with reference to the examples described hereinafter.

[0038] BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Exemplary embodiments will be further described with reference to Figures, wherein: Figure 1 shows a method for generating a power and force limiting, PFL, zone for a robot;

[0040] Figure 2 shows a data processing system;

[0041] Figure 3 shows a schematic example of a segmented spatial representation into spatial elements; and

[0042] Figures 4A and 4B show a schematic example of a classification of the spatial elements into two contact types; and

[0043] Figure 5 shows a schematic example of a PFL zone configuration according to a method of the present invention.

[0044] DETAILED DESCRIPTION OF THE INVENTION

[0045] Figure 1 shows a method for generating a power and force limiting, PFL, zone for a robot. In a first step 102 hazard zone data of the robot I obtained, wherein the hazard zone data is indicative of a potential contact area 11 between the robot 60 and a person 1. This potential contact area 11 may be within an operational environment 20 of the robot 60, for example. Alternatively, the hazard zone data may be determined by a swept volume of a trajectory 12 of the robot 60 and a safety margin 13. The swept volume of a trajectory 13 may be computed by taking the dimensions of the robot 60 body and the variability in trajectory into account, for example. Optionally, the robot’s 60 stopping distance may be computed based on the robot’s 60 velocity at each point throughout or spatial segments 22 of the trajectory.

[0046] In a second step 103, image data is obtained, wherein the image data is indicative of an operational environment 20 of the robot 60. The image data may be provided by an imaging sensor 62, more precisely by a 3D camera mounted on the robot. The 3D camera may deliver 3D information in a reliable and efficient manner. A preferred data format may be point clouds, more preferably 3D point clouds, which can be displayed in real time, for example. The sensor 62 may be temporarily mounted on the robot 60 or picked up by the robot 60 for generating image data. The image data may be pre- processed. For example, noise may be filtered out. The use of 3D cameras may particularly advantageous, as the resulting 3D point clouds can not only be displayed in real time but are also supported by a wealth of existing algorithms and software libraries, for example.

[0047] In a third step 104, as exemplary shown in Fig. 3, a spatial representation 21 or a grid of the operational environment 20 of the robot 60 may be generated, based on the image data, wherein the spatial representation 21 is segmented into spatial elements 22, wherein each spatial element 22 comprises information about an occupancy state 01 to 03 and F1 to F3, depending on whether the spatial element 22 is occupied 01 to 03 by an object or unoccupied 01 to 03. For this, a spatial representation 21 a grid that comprises the robot’s 60 operational environment 20 and its immediate surroundings may be generated. Each point in the point cloud may be assigned to a corresponding spatial element 22 based on its spatial coordinates, for example. Each spatial element 22 within the grid 21 may be classified based on its occupancy state 01 to 03 and F1 to F3. A spatial element 22 may be marked as occupied 01 to 03 if at least one point from the point cloud resides within it. Point clouds may not provide any information about areas in which no measurements were taken, for example due to visual occlusion or range limitations of the sensor 62. The field of view of the 3D camera may therefore also considered to ensure that spatial elements 22 that appear as free F1 to F3, occupied 01 to 03 or unknown in the camera view are correctly marked. Accurate classification of spatial elements 22 may be facilitated through the application of algorithms. For example, threshold-based methods may be employed, where the number of points within a spatial element 22 determines its state. Additionally, machine learning techniques may be integrated to train models that recognize complex patterns within the point clouds, thereby improving the precision of occupancy determinations. Sensor fusion, which combines data from multiple sensors, may also be utilized to mitigate uncertainties and enhance the reliability of the occupancy state 01 to 03 and F1 to F3.

[0048] In a fourth step 105, a safety parameter for each of the spatial elements 22 based on the spatial representation 21 of the operational environment 20 of the robot 60 and the hazard zone data, wherein a higher potential risk for the person 1 is assumed for an occupied area 01 to 03 and a lower potential risk for the person 1 is assumed for an unoccupied area F1 to F3. As further shown in Fig. 3, for each spatial element 22 within the spatial representation 21 , the occupancy state 01 to 03 and F1 to F3 in conjunction with the hazard zone data to determine the associated safety parameters may be assessed. The underlying principle may be that occupied areas 01 to 03, which indicate e.g., the presence of objects 5 or a human body part 2, pose a higher potential risk for accidental contact or collision. In this example, three different 01 to 03 states for occupied areas by an object 5 and three different states for unoccupied F1 to F3 areas by an object 5 are defined. Occupancy state 01 in this example means that spatial elements 22 are classified as occupied within a swept volume of a trajectory or in other words workspace. Thus, occupancy state 01 may refer to areas that fall directly within an operational environment 20 traversed by the robot's 60 planned movement path, for example. The swept volume may encompass the entire volume that the robot 60 will occupy as it moves from one point to another along its trajectory, for example. If any spatial element 22 within this swept volume is found to be occupied, whether by an object 5, obstacle, or person 1 presence, it may signify an immediate collision risk. This classification may necessitate stringent safety measures, such as halting the robot's 60 movement, altering its path, or adjusting its speed to prevent potential accidents. The identification of occupied spatial elements 22 within the swept volume may ensure that the robot 60 can operate safely without impeding or endangering a person 1 such as a human collaborator. Occupancy state 02 may pertain to spatial elements 22 located within a predefined safety margin 13 surrounding the robot's 60 operational workspace 20. The safety margin 13 may serve as a buffer zone that accounts for uncertainties in sensor data, potential person 1 movements, and dynamic changes in the operational environment 20, for example. Spatial elements 22 within this margin that are occupied may indicate areas where there is a heightened potential for interaction or proximity between the robot 60 and a person 1 such as a human operator. While these areas may not lie directly within the robot's immediate swept volume of a trajectory 12, their occupancy still may pose a risk. Safety parameters for these areas may involve reduced robot 60 speed, limited force application, and enhanced sensitivity in obstacle detection to ensure that the robot 60 maintains a safe distance from occupied areas with an occupancy state 02 within the safety margin 13. Spatial elements 22 simply classified as occupied 03 may denote areas within the robot's 60 operational environment 20 that are occupied by objects 5, obstacles, or a person 1 , without specific reference to their proximity to the robot's swept volume of a trajectory 12 or safety margin 13, for example. The classified spatial elements 22 may be continuously updated within an occupancy state 01 to 03 and F1 to F3, reflecting real-time changes and movements within the operational environment 20 of the robot 60.

[0049] Occupancy state F1 in this example describes spatial elements 22 classified as free of objects 5 within the swept volume of a trajectory 12. This may refer to areas that lie entirely within the operational environment 20 the robot 60 will traverse along its planned movement path, for example. The swept volume of a trajectory 12 may comprise all points in space that the robot 60 may occupy as it moves from its starting position to its target location. When a spatial element 22 within this swept volume 12 may be determined to be free of objects 5, it indicates that the robot 60 can move along its trajectory without encountering any objects 5 such as obstacles or hazards in that specific region. This classification may imply a lower risk for a person 1 such as human collaborators, as the absence of objects 5 within the swept volume 12 eliminates dangers such as clamping or crushing body parts 2, allowing the robot 60 to operate safely and efficiently without posing immediate physical threats to a nearby person 1.

[0050] The free of objects 5 within a safety margin 13 occupancy state F2 may pertain to spatial elements 22 located outside the immediate swept volume of its trajectory 12. The safety margin 13 may act as a buffer zone that accounts for uncertainties, such as unexpected human movements or dynamic changes in the environment. Spatial elements 22 within this margin that are free of objects 5 may be considered to present a moderate risk level. While these areas are not directly on the robot's 60 swept volume of its trajectory 12, maintaining them free of objects 5 reduces the likelihood of accidental interactions. This classification may consider that even if a person 1 inadvertently enters the safety margin 13, the absence of objects 5 minimizes risks like clamping or crushing body parts 2, allowing the robot to respond appropriately. Spatial elements 22 classified as free of objects 5 and outside the safety margin 13 and thus occupancy state F3 may be ignored for risk assessment. The occupancy states 01 to 03 and F1 to F3 described herein are exemplary and may include additional or fewer states depending on the specific circumstances.

[0051] Additionally as shown exemplary in Figs. 4A and 4B, the classification of the spatial elements 22 based on their occupancy state (01 to 03; F1 to F3) further comprises identifying a human body part 2 within each spatial element 22 based on the obtained image data and considering the identified human body part 2 in the classification of each spatial element 22 into free contacts (FC) and constrained contacts (CO), wherein different human body parts 2 are associated with different risk levels and corresponding safety parameters. The identification process of a human body part 2 may be employed by algorithms to accurately detect and classify a human body part 2 within the captured image data. This may include automated detection such as machine learning algorithms which utilizes convolutional neural networks (CNNs) or other deep learning models trained on extensive datasets to recognize and segment specific human body parts 2 from the image data, or image processing techniques such as edge detection, color segmentation, and shape recognition to identify and classify human body parts 2. The identification process of a human body part 2 may be performed manual, for example by user-based annotations or customizations. Once human body parts 2 are identified, they may be mapped onto the spatial elements 22. Different human body parts 2 may possess varying degrees of vulnerability and risk exposure when interacting with the robot 60. For instance, hands and fingers may be more susceptible to injury compared to the torso or legs. Additionally, it may be considered if the body part 2 could be clamped between the object 5 and the robot 60. In the first scenario in Fig. 4A a situation is depictured where a person 1 is operating with a human body part 2, in this case the arm, outside of an occupied area. The absence of immediate proximity to occupied areas reduces the likelihood of incidents such as clamping or crushing body parts 2, thereby enabling more efficient and productive robotic operations without compromising safety. Given the FC classification, the body part 2 may be associated with a lower risk level, allowing the robot to operate with higher speed and force parameters within this area. In the second scenario in Fig. 4B, the human body part 2, more precisely a hand of a person 1 is located between an end-effector 61 of the robot 60 and some objects 5. Spatial elements 22 classified as CC may be either occupied or located within a predefined distance from adjacent occupied areas such that a body part 2 may pose the risk of being clamped between the object 5 and in this case end-effector 61 of the robot 60. Thus, these areas may have a higher potential risk for contact-related hazards due to the proximity of objects 5 such as obstacles or occupied zones that could lead to accidental interactions, for example. This higher potential risk for a person 1 may mean that stricter safety measures, including reduced maximum speeds, lower force limitations, tighter pressure constraints, and restricted directional movements, may be imposed to ensure safety in these high-risk zones. Additionally, the classification of the spatial elements 22 may further comprise considering the robot's pose and motion direction.

[0052] In a next step 106, a PFL zone 10 based on the determined safety parameter may be generated. Figure 5 shows a schematic example of a PFL zone 10 configuration according to a method of the present invention. In the embodiment of Fig. 5, a sensor 62, more precisely a 3D camera continuously monitors the environment, providing the necessary image data for the robot 60 to classify spatial elements 22 and establish appropriate safety parameters. The person 1 or human operator utilizes a teach pendant to configure these safety parameters, ensuring that the robot 60 operates within defined safety margins 13 while interacting with the application periphery and performing its designated tasks, for example. This may not only enhance operational efficiency but also ensures the safety of a person 1 such as human collaborators by dynamically adjusting the PFL zones 10 based on real-time environmental data, for example.

[0053] Figure 2 schematically shows a data processing system 50, which may comprise one or more data processing apparatuses 30, e.g., on board computers. The data processing system 50, in particular the data processing apparatuses 30, in particular their processor 32, may be used to carry out the method 100 for generating a PFL 10 zone for a robot as schematically illustrated in Fig. 1. The data processing apparatus 30 comprises at least one processing unit or processor 32, e.g., a CPU, and at least one computer program product 34, e.g., in the form of a computer-readable storage medium. Computer program 40 is stored on the computer program product 34. In this example the processing component 42 of the computer program 40 is provided within the data processing system 50, which may form parts of the computer program 40, e.g., different program code or algorithms for different functions or steps of the method 100. Specifically, a computer program 40 of one of the data processing apparatuses 30 may be comprising one processing component 42, which may be in the form of software codes or instructions for the processors 32, such as but not limited to filtering algorithms, estimation algorithms, optimization algorithms, clustering algorithms, for example. The data processing system 50 may be a distributed computing environment with different processing apparatuses 30, or executed by the same data processing apparatus 30, which may be part of the robot, for example.

[0054] While the invention has been illustrated and described in detail in the drawings and foregoing description, such illustration and description are to be considered illustrative or exemplary and not restrictive. The invention is not limited to the disclosed embodiments. Other variations to the disclosed embodiments can be understood and effected by those skilled in the art and practicing the claimed invention, from a study of the drawings, the disclosure, and the claims.

[0055] As used herein, the word “comprising” does not exclude other elements or steps, and the indefinite article “a” or “an” does not exclude a plurality. The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage. Further, as used herein, the phrase “at least one” or similar, e.g., “one or more of”, in reference to a list of one or more entities should be understood to mean at least one entity selected from any one or more of the entities in the list of entities, but not necessarily including at least one of each and every entity specifically listed within the list of entities and not excluding any combinations of entities in the list of entities. This definition also allows that such entities may optionally be present other than the entities specifically identified within the list of entities to which the phrase “at least one” or similar refers, whether related or unrelated to those entities specifically identified. Thus, as a non-limiting example, “at least one of A and B” (or, equivalently, “at least one of A or B” or, equivalently “at least one of A and / or B” or, equivalently “one or more of A and B”, “one or more of A or B”, or “one or more of A and / or B”) may refer, in one example, to at least one, optionally including more than one, A, with no B present (and optionally including entities other than B); in another example, to at least one, optionally including more than one, B, with no A present (and optionally including entities other than A); in yet another example, to at least one, optionally including more than one, A, and at least one, optionally including more than one, B (and optionally including other entities). In other words, the phrases “at least one,” “one or more,” and “and / or” are open-ended expressions that are both conjunctive and disjunctive in operation. For example, each of the expressions “at least one of A, B, and C,” “at least one of A, B, or C,” “one or more of A, B, and C,” “one or more of A, B, or C,” and “A, B, and / or C” may mean A alone, B alone, C alone, A and B together, A and C together, B and C together, A, B, and C together, and optionally any of the above in combination with at least one other entity.

[0056] As used herein, the phrase “being indicative of” may for example mean “reflecting” and / or “comprising”. Accordingly, an entity, element and / or step referred to herein as “being indicative of [...]” can be synonymously or interchangeably used herein with one, two or all of said entity, element and / or step “comprising [...]” and said entity, element and / or step “reflecting [...]”. Further, as used herein, phrases such as “based on”, “related” or “relating”, “associated” and similar are not to be seen exclusively in terms of the entities, elements and / or steps to which they are referring, unless otherwise stated. Instead, these phrases are to be understood inclusively, unless otherwise stated, in that, for example, an entity, element or step referring by any of these phrases or similar, e.g., being “based on”, an or another entity, element or step, does not exclude that the respective entity, element or step may be further or also “based on” any other entity, element or step than the one to which it refers.

[0057] The designation of methods and steps as first, second, etc. as provided herein is merely intended to make the methods and their steps referenceable and distinguishable from one another. By no means does the designation of methods and steps constitute a limitation of the scope of this disclosure. For example, when this disclosure describes a third step of a method, a first or second step of the method do not need to be present yet alone be performed before the third step unless they are explicitly referred to as being required per se or before the third step. Moreover, the presentation of methods or steps in a certain order is merely intended to facilitate one example of this disclosure and by no means constitutes a limitation of the scope of this disclosure. Generally, unless no explicitly required order is being mentioned, the methods and steps may be carried out in any feasible order. Specifically, the terms first, second, third or (a), (b), (c) and the like in the description and in the claims are used for distinguishing between similar elements and not necessarily for describing a sequential or chronological order. It is to be understood that the terms so used are interchangeable under appropriate circumstances and that the embodiments of the invention described herein are capable of operation in other sequences than described or illustrated herein.

[0058] In the context of the present invention any numerical value indicated is typically associated with an interval of accuracy that the person skilled in the art will understand to still ensure the technical effect of the feature in question. As used herein, the deviation from the indicated numerical value is in the range of ± 10%, and preferably of ± 5%. The aforementioned deviation from the indicated numerical interval of ± 10%, and preferably of ± 5% is also indicated by the terms “about” and “approximately” used herein with respect to a numerical value.

[0059] Any reference signs in the claims should not be construed as limiting the scope.

Claims

Claims:

1. Method (100) for generating a power and force limiting, PFL, zone (10) for a robot (60), the method comprising:- obtaining hazard zone data of the robot (60), wherein the hazard zone data is indicative of a potential contact area (11) between the robot (60) and a person (1),- obtaining image data, wherein the image data is indicative of an operational environment (20) of the robot (60);- generating a spatial representation (21) of the operational environment (20) of the robot (60), based on the image data, wherein the spatial representation (21) is segmented into spatial elements (22), wherein each spatial element (22) comprises information about an occupancy state (01 to 03; F1 to F3), depending on whether the spatial element (22) is occupied (01 to 03) by an object or unoccupied (F1 to F3);- determining a safety parameter for each of the spatial elements (22) based on the spatial representation (21) of the operational environment (20) of the robot (60) and the hazard zone data, wherein a higher potential risk for the person (1) is assumed for an occupied (01 to 03) area and a lower potential risk for the person (1) is assumed for an unoccupied (F1 to F3) area; and- generating a PFL zone (10) based on the determined safety parameter, wherein the safety parameter comprises at least one or more of the following:■ a maximum speed within the PFL zone (10),■ a maximum force limitation within the PFL zone (10),■ a maximum pressure limit within the PFL zone (10),■ a directional movement restriction within the PFL zone (10),■ a response time for hazard detection and reaction within the PFL zone (10).

2. The method (100) according to claim 1 , wherein the hazard zone data is determined by a swept volume of a trajectory (12) of the robot (60) and a safety margin (13).

3. The method (100) according to claim 1 or 2, wherein the method further comprises:- classifying the spatial elements (22) based on their occupancy state (01 to 03; F1 to F3) and their spatial proximity to adjacent occupied (01 to 03) areas into at least two contact types:■ free contacts (FC): determining (22) spatial elements as free contacts (FC) when the spatial element (22) is unoccupied;■ constrained contacts (CC): determining spatial elements (22) as constrained contacts (CC) when the spatial element (22) is occupied or is located within a predefined distance from an adjacent occupied spatial element (22);- assigning the contact types based on the classified spatial elements (22), wherein free contacts are associated with a lower potential risk level for a person (1) and constrained contacts are associated with a higher potential risk level for a person (1).

4. The method (100) according to claim 3, wherein the classification of the spatial elements (22) further comprises:- identifying a human body part (2) within each spatial element (22) based on the obtained image data; and- considering the identified human body part (2) in the classification of each spatial element (22) into free contacts (FC) and constrained contacts (CC), wherein different human body parts (2) are associated with different risk levels.

5. The method (100) of claim 3, wherein the classification of the spatial elements (22) further comprises considering the robot's (60) pose and motion direction.

6. The method according to any one of the preceding claims, wherein generating the PFL zone (10) comprises grouping spatial elements (22) having similar safety parameters.

7. The method (100) according to any one of the preceding claims, wherein a person (1) demonstrates human poses manually or through predefined movements within the robot's (60) operational environment for spatial mapping of where a human body part (2) is likely to interact with the robot’s (60) potential contact area (11).

8. The method (100) according to any one of the preceding claims, wherein generating the spatial representation (21) of the robot's (60) operational environment (20) comprises using point cloud data and the sensor's (62) field ofview to classify spatial elements (22) into unoccupied (F1 to F3) spatial elements (22), occupied (01 to 03) spatial elements (22), and further unknown spatial elements (22).

9. The method (100) according to any one of the preceding claims, wherein the occupancy state (01 to 03; F1 to F3) of the spatial elements (22) is continuously or periodically updated.

10. The method (100) according to any one of the preceding claims, wherein the method further comprises validating the generated PFL zone (10).

11. The method (100) according to any one of the preceding claims, wherein generating the spatial representation (21) of the operational environment (20) of the robot (60) includes using digital modelling data of the operational environment (20), wherein the digital modelling data comprises at least one or more of: computer-aided design (CAD) data, digital maps, and three-dimensional (3D) models.

12. The method (100) according to any one of the preceding claims, wherein the method further comprises applying one or more algorithms to:- minimize the number of PFL zones (10);- maximize the operational efficiency of the robot (60); and- set the safety parameters as high as possible without compromising safety.

13. One or more computer program products (34, 40) comprising instructions which, when executed by one or more data processing apparatuses (30), cause the one or more data processing apparatuses (30) to carry out the method (100) of any one of the previous claims.

14. A data processing system (50) configured to carry out the method (100) of any one of claims 1 to 12.

15. A robot (60) configured to carry out the method (100) of any one of claims 1 to 12.

Images