Integrated Safety Management System for Real-Time Hazard Recognition, Prediction, and Avoidance Control in Robot Workcells

Abstract

According to one embodiment of the present invention, a system for performing real-time risk recognition and avoidance control in a robot work cell in which a worker and a robot are deployed is provided, and the system comprises: a sensor group that acquires environmental information around the worker and the robot, including RGB sensors and LiDAR sensors; a matching module that generates position information and depth information of an object by matching data acquired from the sensor group; a risk detection module that analyzes the posture of the worker or the operating state of the robot using the matched information and calculates the risk of collision between the worker and the robot; and a robot work cell risk control system that generates a stop command or an avoidance operation command for the robot based on the risk detection result. The matching module is configured to generate three-dimensional object information by matching RGB images and LiDAR point clouds based on coordinates, and the robot work cell risk control system operates to generate a control command to execute at least one of stopping, decelerating, or avoidance operations of the robot according to the calculated risk of collision.

Description

Technology Field

[0001] The present invention relates to a technology for ensuring safety in manufacturing and production processes where workers and industrial robots coexist in a robot workcell environment. Specifically, it relates to an integrated safety management system for real-time risk recognition, prediction, and avoidance control of a robot workcell that guarantees worker safety by fusing data acquired from multiple sensors, such as RGB cameras, LiDAR, and proximity sensors, in real time to recognize the operating status and risk factors of workers, robots, and equipment, predicting the possibility of future collisions through digital twin-based spatial analysis, and controlling the robot to perform deceleration, stopping, and avoidance actions according to the results. Background Technology

[0003] Robot workcells equipped with industrial robots are widely utilized to automate manufacturing processes and improve productivity, and recently, collaborative work environments where workers and robots work together in the same space are on the rise. In these environments, the risk of collision accidents is increasing due to various factors, such as unexpected approaches by workers, rapid robot movements, and the presence of surrounding equipment and irregular objects.

[0004] Conventionally, fixed safety systems utilizing single safety devices such as safety fences, light curtains, and proximity sensors have been primarily used to ensure the safety of robot workcells. However, this approach has limitations, such as reduced workspace flexibility and the inability to accurately detect fine movements of workers or the approach of irregular objects, making it unsuitable for collaborative robot environments.

[0005] In addition, camera-based object recognition technology or distance measurement technology using a single LiDAR has recently been introduced, but these single-sensor-based methods have the problem that it is difficult to accurately recognize changes in the operator's posture or the complex trajectory of a robot due to issues such as changes in lighting, occlusion, the occurrence of blind spots, and inaccuracy of depth information.

[0006] Furthermore, conventional risk detection systems are limited to simply assessing the likelihood of collision based on the current state and lack the capability to predict future risks by analyzing the movement patterns of workers and robots. Additionally, they cannot explain the basis for their prediction results, which presents limitations in terms of system reliability and compliance with safety certifications.

[0007] Furthermore, since most existing robot control systems generate stop commands based on fixed safety distances or simple distance thresholds, there is a problem in that active avoidance control that reflects the operator's movement speed, the robot's motion characteristics, and dynamic changes in the surrounding environment is difficult. Prior art literature

[0009] Republic of Korea Published Patent 2021-0105562 (2021.08.27) The problem to be solved

[0010] The present invention is devised to solve the aforementioned problems and aims to provide an integrated risk perception technology capable of accurately and reliably recognizing workers, robots, and unstructured objects by fusing multiple sensor data, such as RGB cameras, LiDAR, and proximity sensors, in real time.

[0011] In addition, the present invention aims to enable the prediction of potential risks that may occur within a workspace in advance by predicting the possibility of future collisions in a digital twin environment based on dynamic information including changes in the position, speed, and attitude of a worker and a robot.

[0012] In addition, the present invention aims to enhance the reliability of safety decision-making by applying explainable artificial intelligence (XAI)-based risk analysis technology to intuitively present the basis for risk judgments derived by the system.

[0013] In addition, the present invention aims to improve safety in a collaborative work environment by automatically controlling the deceleration, stopping, and avoidance movements of a robot according to the risk assessment results and risk level, and by applying a variable protective separation distance (Variable PSD) that responds to worker approach situations.

[0014] Furthermore, the present invention aims to provide an integrated safety management system capable of immediately responding to unexpected access by workers or sudden environmental changes by linking the entire safety process—leading from sensor recognition to risk analysis to risk prediction to robot control—in real time. means of solving the problem

[0016] According to one embodiment of the present invention, a system for performing real-time risk recognition and avoidance control in a robot work cell in which a worker and a robot are deployed is provided, and the system comprises: a sensor group that acquires environmental information around the worker and the robot, including RGB sensors and LiDAR sensors; a matching module that generates position information and depth information of an object by matching data acquired from the sensor group; a risk detection module that analyzes the posture of the worker or the operating state of the robot using the matched information and calculates the risk of collision between the worker and the robot; and a robot work cell risk control system that generates a stop command or an avoidance operation command for the robot based on the risk detection result. The matching module is configured to generate three-dimensional object information by matching RGB images and LiDAR point clouds based on coordinates, and the robot work cell risk control system operates to generate a control command to execute at least one of stopping, decelerating, or avoidance operations of the robot according to the calculated risk of collision.

[0017] In the aforementioned embodiment, the risk detection module is configured to estimate the position, joint angle, or body posture of a worker to determine whether there is an abnormal posture, and to reflect the result of the abnormal posture determination in the calculation of collision risk.

[0018] In addition, in any one of the aforementioned embodiments, the risk detection module is configured to calculate a protective separation distance between a worker and a robot in real time based on the worker's movement speed, the robot's joint speed, the motion trajectory of the robot's end, or the relative speed between objects, and to dynamically adjust the collision risk based on the calculated protective separation distance.

[0019] In addition, in any one of the aforementioned embodiments, the robot work cell risk control system is configured to calculate a plurality of risk grades in response to the collision risk input from the risk detection module, and to selectively perform at least one control operation among stopping, decelerating, speed limiting, generating an avoidance trajectory, or switching to a safety mode of the robot according to the risk grade.

[0020] In addition, in any one of the aforementioned embodiments, the robot work cell risk control system is configured to calculate the robot's avoidance movement direction based on the worker's position, movement direction, or relative approach speed when the collision risk or risk grade is greater than or equal to a preset standard, and to generate the robot's joint trajectory or work trajectory based on the avoidance movement direction and provide it to the robot controller. Effects of the invention

[0022] According to the present invention, by utilizing RGB sensors and LiDAR sensors together to collect environmental information around a worker and a robot in a multimodal manner and matching it based on coordinates in a matching module, it is possible to achieve more accurate 3D object recognition than existing single-sensor-based systems. As a result, the worker's position and the situation around the robot can be reliably identified even under environmental factors such as changes in lighting, occlusion, and blind spots, and reliable collision risk assessment can be made in real time within the robot work cell.

[0023] In addition, the present invention includes a posture-based risk recognition function that includes the position, joint angle, and posture change of a worker's body joints, and a protective separation distance calculation function that considers the relative speed and direction of movement between the worker and the robot, thereby enabling more precise risk assessment than simple distance-based collision judgment. As a result, actual dangerous behaviors such as a worker suddenly extending their arm or bending at the waist can be detected early, and since the risk is dynamically adjusted according to the movement characteristics of the robot, it has the effect of enabling more appropriate safety control in various work situations.

[0024] In addition, the present invention is configured to selectively perform safety controls at different stages, such as deceleration, speed limiting, avoidance maneuvers, and stop commands, based on the calculated collision risk. Therefore, it is possible to immediately avoid danger or stop when necessary without significantly reducing the productivity of the robot. In particular, by calculating the avoidance direction considering the approach direction and speed of the operator and automatically generating a robot joint trajectory based on this, it is possible to reduce the frequency of unnecessary stopping of the robot while effectively avoiding danger in actual collision risk situations. Brief explanation of the drawing

[0026] FIG. 1 is a diagram showing the overall configuration of an integrated safety management system for real-time risk recognition, prediction, and avoidance control of a robot work cell according to an embodiment of the present invention. FIG. 2 is a block diagram showing the internal configuration of an RGB / LiDAR matching module according to an embodiment of the present invention. FIG. 3 is a block diagram showing the detailed configuration of a risk detection module according to an embodiment of the present invention. FIG. 4 is a block diagram showing the internal configuration of an AI main server according to an embodiment of the present invention. FIG. 5 is a block diagram showing the internal configuration of a digital twin server according to an embodiment of the present invention. FIG. 6 is a block diagram showing the detailed configuration of a robot work cell risk control system according to an embodiment of the present invention. Specific details for implementing the invention

[0027] The advantages and features of the present invention and the methods for achieving them will become clear by referring to the embodiments described below in detail together with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below but may be implemented in various different forms.

[0028] The embodiments described herein are provided to ensure that the disclosure of the invention is complete and to fully inform those skilled in the art of the scope of the invention. The invention is defined only by the scope of the claims. Accordingly, in some embodiments, well-known components, well-known operations, and well-known techniques are not specifically described to avoid the invention being interpreted ambiguously.

[0029] Throughout the specification, the same reference numerals refer to the same components. Furthermore, the terms used (mentioned) in this specification are for describing the embodiments and are not intended to limit the invention. In this specification, the singular form includes the plural form unless specifically stated otherwise in the text. Additionally, components and operations referred to as "comprising (or comprising)" do not exclude the presence or addition of one or more other components and operations.

[0030] Unless otherwise defined, all terms used in this specification (including technical and scientific terms) may be used in a meaning commonly understood by those skilled in the art to which the present invention pertains. Additionally, terms defined in commonly used dictionaries are not to be interpreted ideally or excessively unless otherwise defined.

[0032] Embodiments of the present invention will be described below with reference to the attached drawings.

[0033] Referring to FIG. 1, the integrated safety management system for real-time risk recognition, prediction, and avoidance control of a robot work cell according to the present invention comprises: an RGB / LiDAR sensor group (100) composed of a plurality of sensors; a LiDAR fusion sensor real-time monitoring system (10) that collects and fuses data acquired from the RGB / LiDAR sensor group (100) in real time; a main system (20) that analyzes the status of a worker and a robot based on results transmitted from the monitoring system (10) and implements a work environment on a digital twin; a robot work cell risk control system (30) that controls the operation of a robot according to a risk judgment result provided from the main system (20); and a relationship system (40) that visually provides the risk analysis result to a manager.

[0035] First, the RGB / LiDAR sensor group (100) consists of a plurality of RGB cameras and LiDAR sensors installed within the robot work cell, and simultaneously acquires the position and posture of the worker, the movement trajectory of the robot, and the presence of surrounding equipment and irregular objects as 3D spatial information and image information. The data acquired from the sensor group (100) is transmitted to the LiDAR fusion sensor real-time monitoring system (10) via wired or wireless communication.

[0037] The LiDAR fusion sensor real-time monitoring system (10) includes an RGB / LiDAR alignment module (12), a risk detection module (14), and an AI GPU server (16) for performing operations of these modules. The RGB / LiDAR alignment module (12) aligns the RGB image and the LiDAR point cloud into the same reference coordinate system by correcting different coordinate systems, time delays, resolutions, etc. for each sensor, and accordingly generates an aligned RGB / LiDAR data set. The risk detection module (14) receives the aligned RGB / LiDAR data and uses a deep learning-based object detection and segmentation algorithm to separate and recognize workers, robots, fixed equipment, irregular obstacles, etc., and calculates the position, speed, direction of movement, and mutual separation distance of each object. At this time, the complex operations performed by the risk detection module (14) are accelerated by the AI ​​GPU server (16), and may further include computational resources such as BPC SRL and log recording devices as needed. The risk detection module (14) calculates the real-time protection separation distance (PSD) between the worker and the robot based on the above calculation results, and generates primary risk information by comparing it with a preset threshold value and transmits it to the main system (20) and the robot work cell risk control system (30).

[0039] The main system (20) includes an AI main server (22) and a digital twin server (24). The AI ​​main server (22) is a central computing unit that operates a worker / robot pose estimation AI (PAI) and a collision risk recognition XSI module, and receives matched RGB / LiDAR data and primary risk information transmitted from the monitoring system (10). The worker / robot pose estimation AI module estimates the joint angles, pose types, and movement patterns of the worker and the joint positions and poses of the robot arm using human pose estimation and robot joint state recognition algorithms, and tracks the results continuously over time. The collision risk recognition XSI module evaluates the possibility of collision by synthesizing the pose information, separation distance information, and movement speed, and applies explainable artificial intelligence (XAI) techniques to analyze which parts and paths are likely to be dangerous in the form of a heatmap, highlighting of attention areas, etc. The AI ​​main server (22) organizes these analysis results and transmits them to the digital twin server (24) and the robot work cell risk control system (30).

[0041] The digital twin server (24) includes a digital twin server (DT) and a digital twin implementation and other risk situation AI analysis module (DXA: Digital Twin Analysis SW). The digital twin server (DT) implements a three-dimensional virtual model of a robot workcell using ultra-high resolution 3D LiDAR information, design information of the workspace, robot models, equipment layout information, etc. The digital twin analysis module (DXA) updates the position and orientation of objects corresponding to actual sensor data in real time on the virtual model, predicts the future movement paths of the worker and the robot, and simulates the potential collision and risk level after a certain period of time. At this time, the DXA module can compare and analyze the risk reduction effects according to various scenarios (robot deceleration, stopping, change of avoidance trajectory, etc.) and provides the analysis results to other modules within the main system (20) and the robot workcell risk control system (30).

[0043] The robot work cell risk control system (30) includes a risk avoidance robot control module (RC), a robot control algorithm (RCA: Robot Control Algorithm SW), a robot arm (RA) that is the actual robot control target, a warning light (WL), etc. The risk avoidance robot control module (RC) receives risk recognition results and digital twin-based prediction information provided from the main system (20) and the monitoring system (10), and executes the robot control algorithm (RCA) to determine how to control the robot. The robot control algorithm (RCA) calculates commands such as a robot deceleration command, a temporary stop command, an emergency stop command, and a trajectory change command to an avoidance path by considering the current and predicted risk levels, the position and direction of movement of the worker, the working status of the robot, and the variable protection separation distance (Variable PSD). The calculated commands are transmitted to the controller of the robot arm (RA) via communication methods such as USB, TCP / UDP, etc., and the robot arm (RA) performs speed adjustment, path change, or stop actions accordingly. In addition, the risk control system (30) can activate visual and auditory alarm devices, such as a warning light (WL) or a buzzer, according to the risk level to immediately notify the worker of the danger.

[0045] The relationship system (40) receives risk analysis results, digital twin simulation results, and XAI-based explanatory information transmitted from the main system (20) and the robot workcell risk control system (30), and visualizes them on a monitor or large display device via an interface such as HDMI. Accordingly, an administrator or safety officer can intuitively check areas with high current risk, potential collision paths, and robot avoidance motion plans within the robot workcell, and, if necessary, adjust system settings or perform additional safety measures.

[0047] As such, the system according to the present invention is configured such that the multi-sensor data acquisition step starting from the RGB / LiDAR sensor group (100), the alignment and primary risk detection step by the LiDAR fusion sensor real-time monitoring system (10), the attitude estimation, XAI analysis, and digital twin prediction step by the main system (20), the robot avoidance control step by the robot workcell risk control system (30), and the risk visualization step by the related system (40) are linked in real time. Accordingly, the safety of the robot workcell can be greatly improved by recognizing and predicting risks in a rapid and explainable manner even for unexpected approaches by workers or changes in the environment, and by actively controlling the robot's operation.

[0049] FIG. 2 is a block diagram specifically illustrating the internal configuration of an RGB / LiDAR alignment module (12). Referring to FIG. 2, the RGB / LiDAR alignment module (12) of the LiDAR fusion sensor real-time monitoring system (10) according to the present invention is composed of a sensor collection unit (121), a time synchronization unit (122), a calibration parameter storage unit (123), a coordinate system transformation unit (124), a point cloud-image mapping unit (125), a noise removal and sampling unit (126), an alignment quality evaluation unit (127), and an alignment data output unit (128). The RGB / LiDAR alignment module (12) performs the role of aligning raw data acquired from an RGB camera and a LiDAR sensor on a common coordinate system and time axis, and generating aligned RGB / LiDAR data in a form that can be immediately utilized by a risk detection module (14).

[0051] The sensor acquisition unit (121) performs the function of receiving raw image data and point cloud data from a plurality of RGB cameras and lidar sensors included in the RGB / lidar sensor group (100). At this time, the sensor acquisition unit (121) may be configured to buffer data according to the format of the data stream output from each sensor and the communication protocol (TCP / UDP, GigE, USB, etc.), and to store sensor-specific timestamps and metadata (exposure value, laser scan period, etc.) together.

[0053] The time synchronization unit (122) performs the function of matching RGB image frames and LiDAR scan data acquired at the same time based on the timestamp assigned to each sensor data received from the sensor acquisition unit (121). The time synchronization unit (122) can pre-correct the time references of the sensors using NTP, PTP, or an external trigger signal, and in the event of a frame drop or communication delay, it selects data from the nearest time point or performs interpolation to generate temporally consistent data pairs.

[0055] The calibration parameter storage unit (123) performs the function of storing internal parameters of the RGB camera (focal length, principal point, distortion coefficient, etc.) and external parameters between the LiDAR sensor and the RGB camera (rotation matrix, translation vector, etc.). This storage unit (123) stores the results of camera-LiDAR calibration performed offline or during the initial setup phase in non-volatile memory, and allows the coordinate system transformation unit (124) to read and use the corresponding parameters when necessary. Additionally, it may include a function to update and store new parameters when calibration is re-performed due to changes in the work cell environment or sensor replacement.

[0057] The coordinate system conversion unit (124) performs the function of converting LiDAR point cloud data into an RGB camera coordinate system or a robot reference coordinate system using external parameters stored in the calibration parameter storage unit (123). To this end, the coordinate system conversion unit (124) applies a rotation matrix and a translation vector to the 3D coordinates of each point and is configured to support conversion to various reference coordinate systems, such as a robot base coordinate system and a work coordinate system, as needed. The coordinate system conversion unit (124) may use a library dedicated to floating-point operations or matrix operations for precise conversion.

[0059] The point cloud-image mapping unit (125) projects three-dimensional points, which have been converted to the camera coordinate system by the coordinate system transformation unit (124), onto the camera image plane to calculate the pixel location corresponding to each point and combine RGB color values. This mapping unit (125) performs projection transformation using camera intrinsic parameters and removes points if they are located outside the image range. Additionally, when multiple points are projected onto the same pixel, it forms a consistent mapping between the RGB image and the point cloud by applying depth-based selection rules, such as prioritizing the point with the smallest depth value.

[0061] The noise removal and sampling unit (126) has the function of removing sensor noise, reflection anomalies, incorrect points caused by disturbances, etc., from the mapping result generated through the point cloud-image mapping unit (125), and performing downsampling to improve computational efficiency. For example, the noise removal and sampling unit (126) can uniformly adjust the point density by applying a radius-based filter, a statistical outlier removal filter, voxel grid sampling, etc., and generate a preprocessed high-quality point cloud. In addition, points with low reliability can be removed by utilizing additional information such as the reflection intensity and reliability score of the LiDAR.

[0063] The alignment quality evaluation unit (127) performs the function of evaluating the quality of the result of the alignment process. The alignment quality evaluation unit (127) estimates the alignment error by analyzing the correlation between the image projection position of the point cloud and the edges, feature points, corner points, etc. of the image, or by measuring the position error of a pre-set reference marker. And if the alignment quality is less than a pre-set reference value, it can generate a flag indicating that the calibration parameter needs to be re-corrected, or notify the upper control unit of the lidar fusion sensor real-time monitoring system (10) to switch the alignment mode (fixed parameter mode, correction mode, etc.).

[0065] The alignment data output unit (128) receives the processing results of the noise removal and sampling unit (126) and the alignment quality evaluation unit (127), and performs the function of generating and outputting aligned RGB / LiDAR data in a format that can be immediately utilized by the risk detection module (14). The alignment data output unit (128) defines an integrated data format including a point cloud and RGB color values, depth information, reflectance, reliability score, etc., corresponding to each point, and configures this in the form of a memory buffer or network packet and transmits it to the risk detection module (14) and the AI ​​main server (22) of the main system (20).

[0067] As such, the RGB / LiDAR matching module (12) illustrated in FIG. 2 can perform the processes of collecting sensor data, time synchronization, coordinate system transformation, point-image mapping, noise removal, matching quality evaluation, and integrated data output in stages, thereby integrating multiple sensor data obtained within the robot work cell into one consistent three-dimensional information and improving the accuracy and reliability of risk recognition and digital twin-based prediction algorithms performed in subsequent stages.

[0069] FIG. 3 is a block diagram illustrating the internal configuration of a risk detection module (14) according to one embodiment of the present invention.

[0071] Referring to FIG. 3, the risk detection module (14) is composed of an object recognition unit (141), a body posture / joint estimation unit (142), an object position / velocity estimation unit (143), a distance and separation distance calculation unit (144), a collision risk calculation unit (145), a behavior pattern analysis unit (146), and a risk judgment / notification generation unit (147) to determine the risk of collision by analyzing the status of a worker, a robot, and surrounding objects in real time from the matched RGB / LiDAR data and to generate a risk notification or avoidance control command to the robot work cell control unit.

[0073] The object recognition unit (141) receives RGB images and 3D LiDAR point clouds as input, identifies objects such as workers, robots, transport pallets, and irregular obstacles, and calculates the ID and boundary area of ​​each object. A deep learning-based recognition model (YOLO, Mask-RCNN, PointNet, etc.) can be applied, and by combining RGB image-based 2D detection and LiDAR-based 3D detection, the problem of blind spots and lighting dependency of a single sensor is resolved.

[0075] The body posture / joint estimation unit (142) estimates the positions and angles of the worker's major joints by applying a 2D / 3D skeletal pose model to the recognized worker. This allows for the interpretation of changes in the worker's posture, arm protrusion, waist bending, abnormal movements, etc. Since this configuration interprets the worker's body risk posture itself rather than a simple location-based risk judgment, it can more precisely determine the behavior of high-risk workers.

[0077] More specifically, the body posture / joint estimation unit (142) calculates joint points such as the head, neck, shoulders, elbows, wrists, waist, knees, and ankles in a three-dimensional coordinate system based on a human skeleton extracted from matched RGB / LiDAR data, and calculates the bending angle, twisting angle, and elongation degree of each joint using vector relationships between each joint point. Then, based on this joint angle information and the rate of change over time, it calculates a posture risk score that reflects industrial safety standards or ergonomic evaluation indicators (REBA, RULA, etc.) and classifies specific posture patterns with a high probability of accident, such as excessive waist flexion, excessive lifting of the upper limbs, and sudden forward bending of the head and upper body, as high-risk postures.

[0079] In addition, the body posture / joint estimation unit (142) is configured to determine whether there is posture-based spatial interference by considering the relative positions between the worker's joint points, the robot end-effector, and the robot link, for example, whether the worker's arm enters the robot's work trajectory, or whether the waist or upper body crosses the boundary of the robot work area. In this process, by generating a heatmap or attention map that highlights areas with high risk contribution for each joint, it is possible to provide not only a result that it is simply “dangerous” but also “dangerous due to which joint’s posture” in an explainable form (XSI; XAI for Safety Intelligence).

[0081] With this configuration, the problem that conventional distance- and area-based collision judgment methods rely only on the simple distance between the worker and the robot and do not sufficiently reflect the safety of the actual work posture can be resolved, and advanced risk recognition reflecting the worker's joint condition and posture pattern becomes possible. Furthermore, by providing the posture risk calculation result and XAI-based heatmap information together, safety personnel or workers can intuitively understand the basis for risk judgment and improve their work posture, so the body posture / joint estimation unit (142) of the present invention is important in terms of responding to safety certification and intelligent robot safety control.

[0083] The object position and velocity estimation unit (143) calculates the direction of movement; movement velocity; acceleration; inertial movement, etc. based on the 3D coordinates and time change amount of the recognized object. This result serves as basic data for calculating the real-time protection distance (PSD) that reflects the worker approach speed, robot end-stage speed, and dynamic movement of the machine equipment.

[0085] The distance and separation distance calculation unit (144) calculates the three-dimensional distance and relative separation distance between the worker, the robot, and the obstacle. In particular, it calculates a variable protective separation distance (variable PSD) by considering the speed of the worker, the speed of the robot's end, the position of the robot's joints, and the motion trajectory of the robot. Unlike conventional fixed safety distances (e.g., 1m safety distance), the present invention applies a method in which the separation distance standard changes in real time according to the speed characteristics of the object.

[0087] The distance and separation distance calculation unit (144) calculates the three-dimensional distance and relative separation distance between the worker, the robot, and the obstacle. In particular, it calculates a variable protective separation distance (variable PSD) by considering the speed of the worker, the speed of the robot's end, the position of the robot's joints, and the motion trajectory of the robot. Unlike conventional fixed safety distances (e.g., 1m safety distance), the present invention applies a method in which the separation distance standard changes in real time according to the speed characteristics of the object.

[0089] More specifically, the distance and separation distance calculation unit (144) receives inputs for the worker's movement speed, the approach direction between the worker and the robot, the current speed of the robot's end effector, and the change in speed per joint, and calculates a dynamic protective separation distance based on the formula of ISO / TS 15066 or Speed ​​and Separation Monitoring (SSM). At this time, the protective distance is not calculated simply by distance alone,

[0090] - As the worker's movement speed increases, the required minimum protection distance increases,

[0091] - Acceleration and deceleration capabilities of the robot end effector,

[0092] - Maximum stopping distance per robot joint,

[0093] - Robot controller response delay,

[0094] - It is dynamically adjusted by including weights for approach vectors and other factors related to the direction component with a high probability of collision.

[0096] In addition, the distance and separation distance calculation unit (144) creates a safe space (envelope) for each robot joint and estimates the probability of a worker entering the space, taking into account that the actual collision risk point changes depending on the joint position due to the multi-joint structure characteristics of the robot. Through this, it is possible to predict early situations where a collision risk may occur not only due to the robot's end effector but also due to rotation of the middle part of the robot arm, such as the upper or lower arm link.

[0098] Furthermore, the distance and separation distance calculation unit (144) is configured to detect the risk of a part of the body rapidly approaching the robot's trajectory in advance, rather than simply calculating the point-to-point distance, by reflecting the worker's posture information (e.g., upper body bending, arm protrusion) and direction of action. For example, if the user's arm moves rapidly forward in the direction of the robot's trajectory, the local joint speed of the arm is considered first over the overall movement speed to separately calculate the partial protection distance (sub-PSD) for a specific body part.

[0100] As such, unlike the conventional method using a fixed safety distance standard, the distance and separation distance calculation unit (144) of the present invention can calculate a variable PSD that takes into account the dynamic speed characteristics of the worker and the robot, the structural characteristics of the robot joints, the approach direction, and the situation of rapid approach of a part of the body, so it can precisely reflect various movements occurring in the actual work environment and enable safer avoidance control.

[0102] The collision risk assessment unit (145) quantitatively calculates the probability of collision between objects by synthesizing distance, speed, and attitude information. In particular, by applying a Time-To-Collision (TTC) based analysis technique that predicts the time remaining until collision using relative distance and relative speed, it is possible to make a more precise risk assessment than with a simple distance criterion.

[0104] More specifically, the collision risk calculation unit (145) calculates the relative distance D(t), relative velocity V(t), and relative acceleration A(t) between the worker and the robot in real time, and calculates the following Time-To-Collision (TTC) value.

[0105]

[0107] In cases where the relative velocity is not constant, the following acceleration-based prediction formula can be applied:

[0108]

[0110] This enables stable collision time prediction even in irregular situations where workers or robots rapidly accelerate or brake.

[0112] When the TTC decreases below a threshold value, the collision risk calculation unit (145) evaluates the risk level upward, and determines the risk level to be the highest level as the TTC approaches 0. At the same time, by analyzing whether there is an intersection between the worker's movement direction vector and the robot's motion trajectory, a comprehensive collision risk is calculated that considers not only the possibility of collision but also the existence of a collision path (path intersection probability).

[0114] Additionally, the collision risk calculation unit (145) considers the body joint velocity and local movement information provided by the posture / joint estimation unit (142). For example, if a worker's arm rapidly extends toward the robot's end, a local rapid approach occurs where the arm's joint velocity is much greater than the body's overall velocity; therefore, the TTC can recalculate based on the arm joint point to separately evaluate the local risk. Through this, it is possible to detect local situations with a high probability of collision early on, such as when only the arm approaches rapidly even though the entire human body is far away from the robot.

[0116] The collision risk calculation unit (145) also calculates the actual stopping time of the robot, taking into account the maximum deceleration per robot joint, braking time, controller delay time, etc., and compares it with the TTC to determine whether the robot can actually stop before a collision. If the following conditions are satisfied, an immediate danger warning or emergency stop command can be generated:

[0117]

[0119] In this way, the collision risk assessment unit (145) can perform prediction-based and time-based risk analysis that cannot be detected by distance-based judgment alone, and can make immediate risk judgments in situations such as rapid approach by a worker, high-speed operation of a robot, and unpredictable rapid rotation of joints. Therefore, it is effective for improving safety by reflecting complex interactions that may occur in an actual robot work cell in real time.

[0121] The behavior pattern analysis unit (146) is configured to analyze the continuous posture changes and movement trajectories of the worker to predict not only the current dangerous state but also dangerous situations that may occur in the near future. To this end, the behavior pattern analysis unit (146) utilizes time-series information as input, such as the 3D coordinates of joint points and the rate of change of joint angles calculated by the body posture / joint estimation unit (142), and the movement speed and approach direction information calculated by the object position / velocity estimation unit (143). Since this time-series information reflects the patterns of body movements that change over time, it enables prediction of how the worker's movement will change in the future, going beyond simply what state the worker is currently in.

[0123] The behavior pattern analysis unit (146) can use time-series prediction models such as LSTM, GRU, or Transformer to estimate how a worker's movement will change in a very short time in the future and whether such movement will lead to dangerous behavior involving a rapid approach to the robot's workspace. For example, even if the worker is currently maintaining a sufficient distance from the robot, if the analysis of joint velocity changes or body center movement trends over the past few frames indicates a high probability that the arm or upper body will rapidly enter the direction of the robot's work trajectory in the near future, the behavior pattern analysis unit (146) classifies the movement as a predicted dangerous behavior. This function of detecting future dangerous behavior early can provide much higher safety than a method that judges based only on the worker's current position.

[0125] In addition, the behavior pattern analysis unit (146) is configured to take into account even minute movements at the joint level to detect early signs of danger that are difficult to perceive with the naked eye. Even when minute movements accumulate and potentially develop into dangerous behavior, such as when a part of the worker's body, like the elbow or wrist, shows a slight recoil or when the upper body momentarily leans forward, the time series analysis model can determine these patterns as early danger signals. This enables preemptive control before the risk of collision materializes by detecting movements in which a part of the body changes direction before the worker actually moves toward the robot.

[0127] In this way, the behavior pattern analysis unit (146) can perform a probabilistic analysis of what kind of dangerous situation the worker's movement pattern will lead to in the future, rather than simply determining whether there is a risk based on the current distance. Through this, the predicted probability of collision, the speed of progression of dangerous behavior, and the remaining time until the risk actually occurs are comprehensively calculated and used as a basis for safety controls such as robot deceleration, avoidance, or emergency stop performed in subsequent stages. Consequently, the behavior pattern analysis unit (146) is important for securing preemptive safety in the robot workcell environment by predicting future risks in advance.

[0129] The risk judgment and notification generation unit (147) is configured to determine the risk situation that may occur in the present or near future within the robot work cell by synthesizing the risk level calculated by the collision risk calculation unit (145) and the future risk information predicted by the behavior pattern analysis unit (146). The risk judgment and notification generation unit (147) evaluates the overall risk level in stages by considering the distance between the worker and the robot, relative speed, estimated collision time, the risk level of the worker's posture, and the behavior prediction results together. At this time, the risk judgment is not based on a single indicator but is made by reflecting the characteristics of the actual work environment where various risk factors interact in a complex manner, so the determined risk level can more accurately represent the safety level according to the worker's behavior, the robot's operating state, and changes in the surrounding environment.

[0131] The risk judgment and notification generation unit (147) determines the control commands to be transmitted to the robot based on this comprehensive risk analysis. For example, if the risk level is low, it may provide a visual or auditory warning to alert the operator, and if the risk level is high and it is determined that the likelihood of a collision is high, it may generate a control command to reduce the robot's speed or temporarily stop its operation. Additionally, if the risk is rapidly increasing or it is determined that a collision is imminent, it may be configured to generate an emergency stop command to immediately stop all operations of the robot. This control process is performed in real time, and as the risk level and behavior prediction results are continuously updated, the robot's operating state is also adjusted.

[0133] In particular, the risk judgment and notification generation unit (147) may include an explainable artificial intelligence (XAI)-based processing function to clearly explain the basis for the risk judgment to a worker or manager. This allows for visual representation of why the collision risk occurred, which body part is causing the risk, and which robot movement is increasing the risk. For example, if a worker's arm moves rapidly in the direction of the robot's trajectory, a risk heatmap is generated centered on the arm joint, and if the possibility of a collision increases due to a sudden change in the angle of a specific joint of the robot, the robot link part is highlighted. This visual explanation allows a manager or worker to intuitively understand the reason for the risk judgment and can also be used for safety training or work environment adjustment.

[0135] As such, the risk judgment and notification generation unit (147) is configured to perform a comprehensive risk judgment that includes not only the current state but also the near future, and to perform avoidance, deceleration, and stop control of the robot in stages based on the result, which is important for ensuring safety in a robot work cell environment.

[0137] Referring to FIG. 4, the AI ​​main server (22) is configured to include a series of functions for precisely analyzing the status of workers and robots based on multimodal data received from sensors, and for performing risk assessment and explainable AI-based risk judgment. The AI ​​main server (22) first receives aligned RGB / LiDAR data and related metadata from the LiDAR fusion sensor real-time monitoring system (10) through the data receiving unit (221). The received data is preprocessed in the data preprocessing and normalization unit (222) to be suitable for the input format of the AI ​​model, and normalization processes such as noise filtering, coordinate system alignment, and size transformation are performed during this process.

[0139] The preprocessed data is transferred to a multimodal feature extraction unit (223) to extract meaningful high-dimensional features from RGB images and 3D point clouds. The multimodal feature extraction unit can use a structure combining a CNN-based image feature extractor and a point cloud processing network, thereby reducing the bias of information associated with single-sensor-based analysis and enabling stable operation in various environments.

[0141] The AI ​​main server (22) operates the worker pose estimation unit (224) and the robot state estimation unit (225) respectively based on this feature information. The worker pose estimation unit (224) estimates the worker's body joints in a 2D or 3D skeleton form using multimodal features and can precisely analyze the worker's motion patterns, such as joint positions, angle changes, and body center movements. Meanwhile, the robot state estimation unit (225) estimates the robot's current link position, end effector trajectory, and motion state by combining joint angle, velocity, and torque information transmitted from the robot controller with sensor data, and generates a normalized state vector to analyze the robot's motion characteristics and predicted trajectory.

[0143] Worker pose information and robot state information are combined into a single integrated feature by the risk feature integration unit (226). This integrated feature includes various elements such as the relative distance between the worker and the robot, approach direction, speed change, and joint movement, and is used as a key input in future risk analysis. In particular, the risk feature integration unit also considers changes in state over time to form a time-series-based feature that can be used not only for the current situation but also for determining future risks.

[0145] The integrated features generated in the risk feature integration unit are transmitted to the XAI-based risk recognition unit (227) to make a final risk judgment. The XAI-based risk recognition unit calculates whether there is a collision risk and the risk level using a deep learning-based risk judgment model, and simultaneously generates visual or numerical importance information to explain the basis for the risk judgment. For example, if rapid acceleration of a specific joint or rapid rotation of a robot link affects the risk judgment, transparency of the risk judgment can be provided by generating a heatmap that highlights the corresponding joint or link area.

[0147] The AI ​​main server (22) also includes a digital twin linkage interface unit (229) and is configured to synchronize the actual worker's movements and robot status with the digital twin environment and to receive the prediction results analyzed in the virtual environment back into the system. Through this, high-risk analysis based on the digital twin and AI judgment based on real-environment data can be utilized complementarily. In addition, the model management and online learning unit (228) performs version management, parameter updates, and retraining based on field data of the risk judgment model to continuously maintain and improve the performance of the AI ​​model.

[0149] As such, the AI ​​main server (22) performs a series of AI-based advanced analysis functions, from receiving sensor data to multimodal feature extraction, worker and robot status analysis, risk judgment, explanation generation, digital twin linkage, and model maintenance, and is a key component for real-time risk recognition and safety control in a robot workcell environment.

[0151] FIG. 5 is a block diagram illustrating the configuration of a digital twin server (24) according to an embodiment of the present invention. The digital twin server (24) receives real-world environment information, such as RGB cameras, LiDAR, robot controllers, and worker status data, in real time, and creates a virtual model of a robot work cell based on this information to perform an analysis function for predicting future risks.

[0153] First, the digital twin data collection unit (241) receives various data in real time, such as point clouds, RGB images, robot joint information, worker coordinates, and obstacle locations, which are acquired in the real environment. The data is processed in a time-synchronized form so that the digital twin can be updated based on spatial components at the same point in time in a subsequent step.

[0155] Next, the digital twin space reconstruction unit (242) reconstructs the three-dimensional structure of the workspace in a virtual environment using collected point clouds and location data. This reconstruction can reflect high-resolution LiDAR-based point clouds, allowing environmental elements such as the movement of workers, the appearance of obstacles, and the placement of surrounding equipment to be reflected in the virtual space in real time. The reconstructed digital twin space can immediately update dynamic changes in the work environment.

[0157] The robot motion / work environment simulation unit (243) operates a virtual robot model in real time based on joint angles, velocity, and end-face position data provided by the robot controller. In addition, it can calculate the robot motion state after a certain period of time by simulating the estimated future trajectory of the robot. Through this, the possibility of collision with a worker or object can be checked in advance during the simulation stage.

[0159] Subsequently, the risk factor analysis and prediction unit (244) analyzes the possibility of collision between the worker and the robot based on the digital twin space and the robot simulation results. It calculates the risk over time by considering the worker's movement direction and speed, the robot's joint movement characteristics, and the robot's expected trajectory. This configuration predicts the risk of collision at a future point in time using relative distance change, relative speed, and time-to-collision (TTC) indicators.

[0161] Additionally, the environment change event processing unit (245) receives sudden environmental changes occurring within the workspace and reflects them in the digital twin space. For example, when an event occurs such as the sudden appearance of an obstacle, an atypical movement of a worker, or the falling of an object, it detects this and updates the digital twin space, and also reflects it in the subsequent risk analysis process.

[0163] The digital twin-AI integrated interface unit (246) performs bidirectional data exchange between the AI ​​main server (22) and the digital twin server (24). AI computation results, such as worker pose information and posture-based risk analysis results, are applied to the digital twin environment, and risk prediction information based on the robot's future trajectory calculated in the digital twin can be provided back to the AI ​​server. Through this, a comprehensive risk assessment that considers both sensor-based analysis and simulation-based prediction becomes possible.

[0165] Next, the visualization and UI / monitoring unit (247) visualizes the digital twin environment in the form of 3D graphics and provides it to the user in real time. The positions of the robot and the worker, the configuration of the surrounding environment, and potential hazard locations are displayed on the screen, and if the risk level increases, the user can intuitively understand the work site through color changes or highlighting.

[0167] Finally, the digital twin log / scenario storage unit (248) stores robot motion records, hazardous events, worker movement trajectories, environmental change history, etc. The stored data can be used for accident reconstruction, scenario-based simulations, system performance verification, model updates, etc.

[0169] The digital twin server (24) configured in this way combines real-time sensor data and predictive analysis functions to preemptively determine risks that may occur within the robot workcell and provides basic information for safety control.

[0171] FIG. 6 is a block diagram illustrating the internal components of a robot workcell risk control system (30) according to an embodiment of the present invention. The robot workcell risk control system (30) performs a function to ensure worker safety by controlling the actual operation of the robot based on risk information input from an AI main server (22) or a digital twin server (24). This system includes a continuous function of receiving a risk signal, determining a risk level, generating a control command corresponding to the level of risk, and transmitting it to a robot controller.

[0173] The risk recognition signal receiver (301) receives risk recognition signals transmitted from the LiDAR fusion sensor system (10), AI main server (22), digital twin server (24), etc. in real time. The received information may include the worker's location, posture analysis results, robot's future collision prediction information, risk level (caution, warning, emergency), etc., and is input via USB, TCP / UDP, or industrial fieldbus-based protocols.

[0175] The risk level determination unit (302) analyzes the received risk signal and quantitatively classifies the risk level of the current situation. The classification criteria may include the distance between objects, relative speed, Time-To-Collision (TTC), the joint speed of the robot, the motion energy of the robot, and the dangerous posture of the worker. The determined level may be set in a multi-level structure, such as a caution level (Level 1), a deceleration level (Level 2), an avoidance level (Level 3), and an emergency stop level (Level 4).

[0177] Unlike conventional methods that rely on simple distance-based thresholds, the risk classification used in this invention utilizes the Time-to-Collision (TTC) concept, which can interpret the relative movement between a worker and a robot in terms of time. TTC represents the estimated time until a collision occurs when both entities maintain the same state of motion, based on the current relative distance and relative velocity. Since this value reflects not only whether the two objects are close but also the speed and direction in which they are approaching, it enables a precise understanding of the dynamic relationship between the worker and the robot.

[0179] The TTC is calculated by first determining the distance vector between the robot end-effector or a specific risk link and the worker center point, and then projecting the relative velocity components of the two onto this vector. If the relative velocity is directed toward the danger direction, the distance reduction rate appears as a positive value; in this case, the TTC can be calculated by dividing the current distance (d) by the relative distance reduction rate (v_rel). Conversely, if the worker moves away from the robot, v_rel appears as a negative value, rendering the TTC value meaningless; in such cases, the probability of collision is deemed low, and the situation is excluded from the TTC-based risk rating calculation.

[0181] The TTC value calculated in this manner is used as a key judgment criterion for transitioning risk levels. Generally, if the TTC is large and has sufficient margin, the risk of collision between the operator and the robot is judged to be low; however, if the TTC decreases below a certain threshold, the probability of a collision is judged to increase rapidly. For example, if the TTC is maintained for several seconds or more, it is classified as a low-level risk requiring a warning, and if the TTC drops to the 1-second range or less, it may transition to a stage requiring the robot to decelerate. If the TTC decreases to a very short time, such as 0.5 seconds or less, it transitions to a stage requiring avoidance maneuvers, and if the TTC is 0.2 seconds or less or approaches 0, indicating that a collision is virtually unavoidable, an immediate emergency stop command is issued.

[0183] However, the TTC-based risk rating determination in the present invention does not simply rely on a time threshold, but is dynamically adjusted by reflecting the operator's direction of movement, the robot's joint and end-body speeds, the robot's current work stage, the presence of surrounding obstacles, and the results of future behavior prediction through a digital twin. For example, if the robot is working in high-speed mode, it may be judged to have a higher risk level even with the same TTC value, while conversely, if the robot is already decelerating or performing avoidance movements, the increase in the rating may be somewhat delayed. In addition, when calculating the TTC, whether the operator's posture or changes in joint angles coincide with the collision direction—that is, whether the operator is performing movements such as extending arms or tilting the body toward the robot—is simultaneously evaluated and may influence the adjustment of the TTC threshold.

[0185] As such, risk level switching conditions centered on TTC can precisely reflect the actual probability of collision compared to conventional methods based on static safety distances, and enable more reliable risk assessment by considering the operator's motion characteristics and the robot's operational context. Consequently, the TTC-based switching conditions of the present invention reduce the problem of the robot stopping unnecessarily frequently while enabling immediate control intervention in situations where the risk of collision increases significantly.

[0187] The avoidance motion generation unit (304) automatically generates an avoidance motion to reduce the risk of collision of the robot when the risk level determined by the risk level determination unit (302) is above a certain standard. The generated avoidance motion can be performed in various forms as follows.

[0188] - Adjust the movement direction of the robot end-effector to the opposite side of the danger area

[0189] - Change in evasion trajectory at the joint level

[0190] - Generates active avoidance paths that infer the direction of movement of workers or objects and reacts accordingly

[0192] More specifically, the avoidance motion generation unit (304) performs the function of calculating a new movement trajectory of the robot to reduce the possibility of collision between the robot and the worker when the risk level determination unit (302) determines that the avoidance stage or higher is determined. The avoidance motion generation unit (304) first uses information transmitted through the risk recognition signal receiving unit (301) and the robot's current state information transmitted through the robot controller interface unit (307) as inputs. The input information may include the current position and velocity of the robot end-effector, the position, velocity, and acceleration of each joint, the worker's current position and direction of movement, relative distance and relative velocity, predicted collision time and expected collision point, etc.

[0194] The avoidance motion generation unit (304) first defines a spatial area requiring avoidance. To this end, a danger area based on a variable protection distance (PSD) is set around the worker, and a restricted area where the robot must not move is set around the robot's existing work trajectory. At this time, the variable protection distance is calculated by considering the worker's movement speed, the speed of the robot's end, the relative approach direction, and the predicted collision time, and the avoidance motion generation unit (304) updates the avoidance target area in real time using these variable distance values. In other words, when the worker approaches quickly, the danger area is set larger, and when the worker's movement is gradual, a relatively smaller avoidance area is set.

[0196] Once the danger zone is defined in this way, the avoidance motion generation unit (304) calculates the avoidance direction and target position of the robot end effector. Generally, a vector is calculated from the current position of the robot end to the worker position, and a direction of movement of a certain distance in the opposite direction of this vector can be set as the basic avoidance direction. However, since fixed equipment, safety fences, and surrounding structures exist within the robot work cell, the avoidance motion generation unit (304) adjusts the avoidance direction by considering the available space within the work cell. To do this, after generating several candidate avoidance directions, the possibility of collision between the robot link and surrounding equipment, obstacles, and safety zones is evaluated for each candidate direction, and among them, a direction that satisfies both the joint limit and the speed limit of the robot without causing a collision can be selected as the final avoidance direction.

[0198] When the avoidance direction is determined, the avoidance motion generation unit (304) generates an avoidance trajectory over time in the robot's joint space or workspace. For trajectory generation, third-order or fifth-order polynomial interpolation, an S-curve velocity profile, or a trajectory planning algorithm based on minimum acceleration or minimum jerk may be used. The generated trajectory is defined so that the robot end moves continuously from the current position to the avoidance target position, and constraints are applied so that the joint velocity and acceleration limits are not exceeded at each point in time. In addition, for each point on the avoidance trajectory, the distance from the worker and the obstacle and whether the variable protection separation distance is satisfied are verified, and if the conditions are not satisfied during the verification process, the avoidance trajectory may be repeatedly recalculated.

[0200] The avoidance motion generation unit (304) also considers continuity with the existing work trajectory being performed by the robot. When an avoidance command occurs while the robot is performing a specific process, the avoidance motion generation unit (304) temporarily suspends the existing work trajectory and switches to an avoidance trajectory, and calculates a return trajectory that returns to the original work trajectory when the dangerous condition is resolved. At this time, the avoidance trajectory and the return trajectory are designed to satisfy the same speed and acceleration limits, thereby minimizing the mechanical burden on the robot while ensuring worker safety.

[0202] In addition, the avoidance motion generation unit (304) continuously monitors the relative position and relative speed of the worker and the robot even while the avoidance motion is being performed. If the environment changes, such as when the worker continues to approach along the avoidance direction or when a new obstacle appears, the avoidance motion generation unit (304) does not maintain the already generated trajectory as is, but can recalculate the avoidance target position and trajectory according to the current situation. Through this closed-loop avoidance control method, it is possible to verify whether safety conditions are continuously satisfied even while the avoidance motion is being performed.

[0204] As a result, the avoidance motion generation unit (304) determines the direction of movement and target position of the robot end based on the risk level and prediction information provided by the risk grade determination unit (302), and generates a continuous trajectory over time and provides it to the robot controller interface unit (307), thereby enabling the robot to be controlled to reduce the risk of collision with the worker without completely stopping the work.

[0206] The robot stop command unit (305) generates a command to immediately stop the operation of the robot when the risk level is determined to be an emergency stop level (Level 4). The stop command is transmitted to the robot controller as an E-Stop signal and can be configured in various safety stop modes such as Category 0 (immediate power cutoff) and Category 1 (control-based stop).

[0208] The robot controller interface unit (307) serves as an interface that transmits control commands generated by the avoidance motion generation unit (304) and the robot stop command unit (305) to the actual robot controller (RC). The communication method may be configured to include USB, TCP / UDP, EtherCAT, RS485, etc., and may be configured to support various command transmission protocols depending on the robot controller manufacturer and model.

[0210] The danger situation notification unit (306) performs the function of visually and audibly notifying a worker or manager of a danger state determined by the system. For example, a real-time alert may be provided through a warning light, warning sound, screen pop-up, or field display (HMI) depending on the danger level.

[0212] The logging and control history storage unit (308) stores danger signals, danger judgment results, robot control command history, avoidance action history, etc., in chronological order. The stored information can be used for accident analysis, system verification, safety certification response, future algorithm improvement, etc.

[0214] The robot work cell risk control system (30) according to the present invention includes a structure that calculates a multi-stage risk level by comprehensively considering the behavioral characteristics of the worker, the operating state of the robot, the relative speed between the robot and the worker, the time to collision (TTC), and the prediction results based on a digital twin, unlike the conventional binary method that simply determines whether there is a possibility of collision when a received risk signal is received. This risk level is used as a criterion for switching the control mode of the robot, and different control strategies such as warning, deceleration, avoidance, and stopping may be applied depending on the degree of risk.

[0216] The multi-stage risk level determination is primarily based on a risk index calculated based on the instantaneous distance between the worker and the robot, as well as the rate of change in distance, approach speed, direction vector, speed profile of the robot joints, the worker's posture, and joint angles. Additionally, through a simulation performed on the digital twin server (24), the possibility of a collision within a certain period of time is predicted, and this prediction result is also reflected as a factor in determining the risk level.

[0218] For example, if the speed at which a worker approaches the robot's work area is gradual and the probability of a collision is low, it is classified as a 'caution stage,' and in this stage, only visual notifications, such as the illumination of a warning light, are provided. On the other hand, if the worker's approach speed increases or the TTC falls below a set threshold, it is switched to a 'deceleration stage,' at which time the robot's joint speed is automatically limited and the robot's movement is adjusted to a smoother mode. Subsequently, if the relative approach direction between the robot and the worker changes to include the possibility of a direct collision, it is classified as an 'avoidance stage,' and in this stage, the avoidance motion generation unit (304) is activated to generate an avoidance trajectory so that the robot's end or joint can move out of the danger area. Finally, if the worker approaches the robot proximity limit line or a collision is almost certain, it is determined as an 'emergency stop stage,' and the robot stop command unit (305) immediately transmits an emergency stop signal to the robot controller.

[0220] As such, unlike conventional methods that simply stop robot operation by classifying risk levels into multiple stages, the present invention enables flexible control that ensures safety while maintaining the robot's ongoing task as much as possible. This structure plays a crucial role in maintaining a balance between the robot's productivity and safety by reflecting the fact that the required intensity of control may vary depending on the situation and the operator's behavioral patterns.

[0222] Furthermore, risk levels are not determined solely by fixed criteria; instead, situational factors such as environmental changes, changes in operator posture, robot joint speeds, robot mission phases, and the presence of surrounding obstacles are updated in real time, and the levels are continuously adjusted accordingly. This dynamic risk level-based control structure is significant in that, unlike existing methods that relied on static safety boundaries, it can reflect changes in the actual working environment.

[0224] As explained above, according to the present invention, since collision risk can be evaluated from multiple angles by utilizing RGB-LiDAR-based alignment information together with the operator's attitude and velocity information, it becomes possible to more accurately recognize actual dangerous situations that are difficult to identify using a simple distance-based method. Furthermore, since step-by-step control such as deceleration, avoidance, and stopping is automatically performed according to the level of risk, unnecessary robot stops are reduced while enabling immediate response when necessary; consequently, it is possible to simultaneously ensure the safety of the operator and the operational efficiency of the robot in the robot workcell.

[0226] Although preferred embodiments of the present invention have been described in detail above, this is merely an example to aid in understanding the invention, and the present invention is not limited to the above embodiments. Those skilled in the art to which the present invention pertains can make various modifications, variations, and applications without departing from the technical spirit or scope of the present invention, and it is obvious that such modified embodiments are also included within the scope of protection of the present invention. Accordingly, the scope of protection of the present invention is determined by the appended claims, and all technical ideas within the scope equivalent to the claims should be interpreted as being included within the scope of rights of the present invention. Explanation of the symbols

[0228] 10: LiDAR Fusion Sensor Real-time Monitoring System 12: RGB / LiDAR alignment module 14: Risk detection module 16: AI GPU Server 20: Main System 22: AI Main Server 24: Digital Twin Server 30: Robot workcell risk control system 40: Control system 100: RGB / LiDAR Sensor Group

Claims

Claim 1 A system for performing real-time risk recognition and avoidance control in a robot workcell where a worker and a robot are deployed comprises: a sensor group for acquiring environmental information around the worker and the robot, including RGB sensors and LiDAR sensors; a matching module for generating position and depth information of an object by matching data acquired from the sensor group; a risk detection module for analyzing the posture of the worker or the motion state of the robot using the matched information and calculating the collision risk between the worker and the robot; and a robot workcell risk control system for generating a stop command or an avoidance action command for the robot based on the risk detection result. The matching module is configured to generate 3D object information by matching RGB images and LiDAR point clouds based on coordinates, and the robot workcell risk control system generates a control command to execute at least one of stopping, decelerating, or avoidance actions of the robot according to the calculated collision risk. The risk detection module or the AI ​​main server calculates a risk contribution per joint point-robot link pair based on the relative position, relative velocity, and relative approach direction between a plurality of body joint points of the worker and a plurality of links or end effectors of the robot, and the risk contribution is greater than or equal to a reference A real-time risk recognition and avoidance control system for a robot workcell, characterized by being configured to calculate a partial protection separation distance or a local collision prediction time for a joint point-robot link pair, reflect the partial protection separation distance or the local collision prediction time in the collision risk calculation, and generate a heatmap, a region of attention, or visual importance information indicating the basis for risk judgment based on the risk contribution. Claim 2 A real-time risk recognition and avoidance control system for a robot workcell according to claim 1, characterized in that the risk detection module estimates the body joint position, joint angle, or body posture of a worker to calculate a posture risk score, and is configured to reflect the posture risk score as a weight of the risk contribution per joint point-robot link pair. Claim 3 A real-time risk recognition and avoidance control system for a robot workcell according to claim 1, wherein the risk detection module calculates the partial protection separation distance based on the local velocity of a body joint point included in a joint point-robot link pair in which the risk contribution is greater than or equal to a standard, and the relative approach velocity, stopping distance, or control response delay time of the corresponding robot link or end effector, and is configured to increase the collision risk level if the partial protection separation distance is not satisfied. Claim 4 A real-time risk recognition and avoidance control system for a robot workcell according to claim 1, wherein the robot workcell risk control system calculates a plurality of risk grades based on the risk contribution per joint point-robot link pair, the partial protection separation distance, or the local collision prediction time, and is configured to selectively perform at least one control operation among stopping, decelerating, speed limiting, generating an avoidance trajectory, or switching to a safety mode of the robot according to the risk grade. Claim 5 A real-time risk recognition and avoidance control system for a robot workcell according to claim 1, characterized in that the robot workcell risk control system is configured to set a joint point-robot link pair having a risk contribution greater than or equal to a standard as an avoidance reference object, and to select one of a plurality of avoidance candidate trajectories satisfying a partial protection separation distance or a local collision prediction time for the avoidance reference object and provide it to a robot controller.

Images