Real-time collision prevention method for preventing collision between person and robot, and real-time collision prevention system using same

Abstract

According to the present invention, a real-time collision prevention system for preventing collisions between a person and a robot can be provided, the system comprising: a human skeleton extraction unit for extracting skeleton information of a person; a robot skeleton extraction unit for extracting skeleton information of a robot; a motion vector generation unit for generating a motion vector of the person and a motion vector of the robot by using the skeleton information of the person and the skeleton information of the robot, respectively; and a collision risk determination unit for determining a collision risk between the person and the robot on the basis of the motion vector of the person and the motion vector of the robot, wherein each of the skeleton information of the person and the skeleton information of the robot includes a plurality of pieces of three-dimensional coordinate information for a plurality of points.

Description

Real-time collision avoidance method for preventing collisions between humans and robots and real-time collision avoidance system using the same

[0001] The present invention relates to a real-time collision prevention method for preventing collisions between a person and a robot, and a real-time collision prevention system using the same. More specifically, the present invention belongs to the field of robotics and human-robot interaction and is a technology for improving safety during physical interaction between a person and a robot, and in particular, relates to a method and system for preventing real-time collisions by controlling the movement of a robot using three-dimensional skeleton data of a person and a robot.

[0002] With the recent advancement of robotic technology leading to increased utilization of robots in industrial and service sectors, instances of humans and robots working in the same space are becoming more common. Consequently, as physical interaction between robots and humans increases, there is a risk of collision, making the assurance of safety a critical challenge.

[0003] Existing robot collision avoidance systems have limitations in accuracy and response methods as they primarily rely on simple distance sensors or 2D vision systems. In particular, these methods reduce robot operational efficiency and present challenges in accurately predicting collisions in complex work environments. Therefore, more sophisticated and effective collision avoidance and stability enhancement technologies are required. Specifically, technologies that track human movements in real time and adjust robot movements are essential.

[0004] The present invention aims to provide a method and system capable of preventing collisions between a person and a robot by analyzing the motion vectors of the person and the robot to determine the collision risk and controlling the robot's movements.

[0005] In addition, the present invention aims to provide a method and system that can more efficiently determine the risk of collision between a person and a robot and control the movement of the robot by utilizing skeleton data of a person and a robot.

[0006] In addition, the present invention aims to provide a more sophisticated and effective collision prevention and safety enhancement system by adjusting the robot's motion vectors to reduce the risk of collision while tracking human movement in real time.

[0007] In addition, the present invention aims to provide a method and system that can improve safety in the work and service environment of robots and humans and maintain the work efficiency of robots.

[0008] The problems to be solved by the present invention are not limited to those mentioned above, and other unmentioned technical problems will be clearly understood by a person skilled in the art from the description below.

[0009] According to one embodiment of the present invention, a real-time collision prevention system for preventing collisions between a person and a robot comprises: a person skeleton extraction unit configured to extract skeleton information of a person; a robot skeleton extraction unit configured to extract skeleton information of a robot; a motion vector generation unit configured to generate a motion vector of the person and a motion vector of the robot, respectively, using the skeleton information of the person and the skeleton information of the robot; and a collision risk determination unit configured to determine a collision risk between the person and the robot based on the motion vector of the person and the motion vector of the robot, wherein the skeleton information of the person and the skeleton information of the robot each include a plurality of three-dimensional coordinate information for a plurality of points.

[0010] In addition, the collision risk determination unit may be configured to extract the shortest distance between the person and the robot using a plurality of three-dimensional coordinate information included in the person's skeleton information and a plurality of three-dimensional coordinate information included in the robot's skeleton information, and to determine the collision risk based on the magnitude of the shortest distance.

[0011] In addition, the motion vector generation unit may be configured to generate the motion vector of the person and the motion vector of the robot by using the difference between the previous frame and the current frame, using three-dimensional coordinate information for a plurality of points stored according to a preset time interval.

[0012] Additionally, the system may further include a vector filtering unit configured to filter at least one specific motion vector among a plurality of motion vectors generated by the motion vector generation unit, and the collision risk determination unit may be configured to determine the collision risk between the person and the robot using the motion vector filtered by the vector filtering unit.

[0013] In addition, the vector filtering unit may be configured to filter motion vectors for points in a preset specific area among a plurality of points.

[0014] Additionally, the vector filtering unit may be configured to filter the motion vector based on at least one of the distance between the point of the person and the robot, the magnitude of the vector, and area information.

[0015] In addition, the area information is information related to whether it is a hand or face area, and can be configured to calculate the motion vector by applying weights based on the area information.

[0016] In addition, the collision risk determination unit may be configured to determine the collision risk using at least one of the distance, speed, and direction of the person's movement vector and the robot's movement vector.

[0017] In addition, the robot movement adjustment unit may further include a robot movement adjustment unit configured to change the movement path of the robot to reduce the collision risk when the collision risk is greater than or equal to a predetermined standard.

[0018] In addition, the robot movement adjustment unit may be configured to reduce the movement speed of the robot when the collision risk level is at a first level, and to change both the movement speed and direction of the robot when the collision risk level is at a second level higher than the first level.

[0019] In addition, the robot movement control unit may be configured to change the movement path of the robot based on the robot's movement vector adjusted using the human movement vector.

[0020] In addition, the robot skeleton extraction unit may be configured to acquire three-dimensional skeleton data of the robot by using a transformation between the robot coordinate system and the camera coordinate system based on the joint angles of the robot.

[0021] According to another embodiment of the present invention, a real-time collision prevention method for preventing a collision between a person and a robot comprises: a step of extracting skeleton information of a person by a person skeleton extraction unit; a step of extracting skeleton information of a robot by a robot skeleton extraction unit; a step of generating a motion vector of the person and a motion vector of the robot, respectively, using the skeleton information of the person and the skeleton information of the robot by a motion vector generation unit; and a step of determining a collision risk between the person and the robot based on the motion vector of the person and the motion vector of the robot by a collision risk determination unit, wherein the skeleton information of the person and the skeleton information of the robot each include a plurality of three-dimensional coordinate information for a plurality of points.

[0022] According to the present invention, a method and system can be provided to prevent collisions between a person and a robot by analyzing the motion vectors of the person and the robot to determine the collision risk and controlling the robot's motion.

[0023] In addition, according to the present invention, a method and system can be provided to more efficiently determine the risk of collision between a person and a robot and control the movement of the robot by utilizing skeleton data of a person and a robot.

[0024] In addition, according to the present invention, a more sophisticated and effective collision prevention and safety enhancement system can be provided by adjusting the motion vector of a robot that can reduce the risk of collision while tracking human movement in real time.

[0025] In addition, according to the present invention, a method and system can be provided to improve safety in the work and service environment of robots and humans and to maintain the work efficiency of robots.

[0026] The problems to be solved by the present invention are not limited to those mentioned above, and other unmentioned technical problems will be clearly understood by a person skilled in the art from the description below.

[0027] FIG. 1 is a block diagram illustrating the configuration of a real-time collision prevention system between a person and a robot according to one embodiment of the present invention.

[0028] FIGS. 2a to 2c are illustrative diagrams for explaining a process for extracting a human skeleton according to an embodiment of the present invention.

[0029] FIGS. 3a to 3d are exemplary diagrams illustrating a process for extracting a skeleton of a robot according to an embodiment of the present invention.

[0030] FIG. 4 is an illustrative diagram for explaining a vector generation process according to an embodiment of the present invention.

[0031] FIG. 5 is a flowchart illustrating a real-time collision prevention method for preventing collisions between a person and a robot in one embodiment of the present invention.

[0032] Embodiments of the present invention are described below with reference to the attached drawings so that those skilled in the art can easily implement the invention. However, the present invention may be embodied in various different forms and is not limited to the embodiments described herein. Furthermore, in order to clearly explain the embodiments of the present invention in the drawings, parts unrelated to the explanation have been omitted.

[0033] The terms used in this specification are used merely to describe specific embodiments and are not intended to limit the invention. Singular expressions may include plural expressions unless the context clearly indicates otherwise.

[0034] In this specification, terms such as “comprising,” “having,” or “having” are intended to indicate the existence of the features, numbers, steps, actions, components, parts, or combinations thereof described in the specification, and should be understood as not excluding in advance the existence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof.

[0035] Furthermore, the components shown in the embodiments of the present invention are illustrated independently to represent different characteristic functions and do not imply that each component consists of separate hardware or a single software unit. That is, for convenience of explanation, each component is described as a separate component, and at least two of the components may be combined to form a single component, or a single component may be divided into multiple components to perform a function. Such integrated and separated embodiments of each component are also included within the scope of the present invention as long as they do not deviate from the essence of the invention.

[0036] In addition, the following embodiments are provided to explain more clearly to those with average knowledge in the industry, and the shapes and sizes of the elements in the drawings may be exaggerated for clearer explanation.

[0037] Hereinafter, a preferred embodiment according to the present invention will be described with reference to the attached drawings.

[0038]

[0039] FIG. 1 is a block diagram illustrating the configuration of a real-time collision prevention system between a person and a robot according to one embodiment of the present invention.

[0040] Referring to FIG. 1, a real-time collision prevention system (100) for preventing collisions between a person and a robot may include a communication unit (110), a camera unit (120), and a processing unit (130), and as needed, some components may be omitted or additional components may be added.

[0041] Here, the communication unit (110) is configured to perform communication between each component within the real-time collision avoidance system (100) through a network, and is also configured to receive necessary information from an external server or external device through the network or to transmit acquired information to an external server or external device, wherein the network may be a wired or wirelessly connected network. Additionally, the connection network may be a network directly connected between an external device and a mobile robot, or it may be a private network created by a repeater. If the network is a wireless communication network, it may include cellular communication or short-range communication. For example, cellular communication may include at least one of LTE (Long-Term Evolution), LTE-A (LTE Advanced), 5G (5th Generation), 6G (6th Generation), CDMA (Code Division Multiple Access), WCDMA (Wideband CDMA), UMTS (Universal Mobile Telecommunications System), WiBro (Wireless Broadband), or GSM (Global System for Mobile Communications). In addition, short-range communication may include at least one of Wi-Fi (Wireless Fidelity), Bluetooth, Zigbee, NFC (Near Field Communication), or RFID (Radio Frequency Identification). However, the communication method is not limited to these and will also include wireless communication technologies developed in the future.

[0042] The camera unit (120) may be configured to photograph a person or robot in the workspace in real time, acquire an image, and transmit it to the processing unit (130) to perform image processing. For example, it may be configured to acquire RGB images and depth images of each object, and may be configured with multiple cameras as needed.

[0043] The processing unit (130) is configured to perform a process of determining and preventing the risk of collision between a person and a robot by extracting three-dimensional skeleton data of a person and a robot, analyzing motion vectors over time to determine the risk of collision, and adjusting and controlling the current motion vector of the robot accordingly. It may include a central processing unit (CPU), an application processor (AP), etc., and may be installed within an order product delivery system or configured within a separate device or server to be connected through a communication unit (110).

[0044] The processing unit (130) may include, for example, a human skeleton extraction unit (131), a robot skeleton extraction unit (132), a motion vector generation unit (133), a vector filtering unit (134), a collision risk determination unit (135), and a robot motion adjustment unit (136). The programs or program modules included in each of these processing units may be configured in the form of an operating system, an application program, or a program, and may be physically stored on various types of widely used storage devices. Such programs or program modules may include one or more routines, subroutines, programs, objects, components, instructions, data structures, and various forms for performing specific tasks or executing specific data types, but are not limited to these forms. Below, the operation of each component of the processing unit (130) will be examined in order.

[0045]

[0046] human skeleton extraction part (131)

[0047] The human skeleton extraction unit (131) is configured to extract human skeleton information using an image obtained through the camera unit (120), and performs the role of detecting major joint points of the human and estimating their positions. There may be various methods for human skeleton estimation, such as using sensor data (mainly images or depth information), and the present invention is not limited to a specific method, and various skeleton estimation methods can be applied.

[0048] As an example of a human skeleton estimation method, three-dimensional joint coordinates can be estimated by combining a pose estimation vision AI model such as YOLOv8 with depth information from a depth camera.

[0049] First, in the data collection stage, RGB images and depth information are acquired through the camera unit (120), and, for example, as in FIG. 2a, color images of a person are collected in real time, and depth information can be obtained for each pixel using a depth camera. This depth information represents the distance from the camera to the object, and the RGB images and depth information are synchronized so that they are captured at the same time.

[0050] Next, 2D pose estimation can be performed using an artificial intelligence model such as YOLOv8. YOLOv8 is a state-of-the-art deep learning model for real-time object detection and pose estimation that processes input RGB images to detect people and estimate 2D joint positions. This AI model outputs 2D coordinates (u_i, v_i) for each joint i, and major joints include the head, shoulders, elbows, wrists, hips, knees, and ankles. As shown in Fig. 2b, a total of N = 17 joints are estimated; in the default settings of this 2D pose model, there are 17 keypoints, and each keypoint represents a specific part of the human body. For example, the mapping of each index to the corresponding body joint is as follows: 0 is nose, 1 is left eye, 2 is right eye, 3 is left ear, 4 is right ear, 5 is left shoulder, 6 is right shoulder, 7 is left elbow, 8 is right elbow, 9 is left wrist, 10 is right wrist, 11 is left hip, 12 is right hip, 13 is left knee, 14 is right knee, 15 is left ankle, and 16 is right ankle.

[0051] Finally, a transformation into 3D coordinates can be performed. A depth value d_i corresponding to each 2D joint position (u_i, v_i) is extracted from the depth image, and this depth value indicates how far the joint is from the camera. In this way, by combining the 2D coordinates obtained through the artificial intelligence model with the depth information extracted from the depth image to finally obtain 3D coordinates for each point, 3D skeleton data of a person can be obtained.

[0052]

[0053] Robot skeleton extraction unit (132)

[0054] The robot skeleton extraction unit (132) is configured to extract skeleton information of the robot and performs the role of detecting major joint points of a multi-jointed robot and estimating their positions. Robot skeleton estimation can be performed using camera images similar to the human skeleton extraction unit (131), but methods such as using the robot's API (application programming interface) can be used more accurately. The present invention is not limited to a specific robot or method, and various robot skeleton estimation methods can be applied. The sensors or libraries used may be changed according to the system requirements and the type of robot.

[0055] As an example, joint-related data such as joint angles can be obtained using the RTDE (Real-Time Data Exchange) library of Universal Robots (UR) robots, and then converted into 3D coordinates of each joint using Denavit-Hartenberg (DH) parameters as illustrated in Fig. 3a. Additionally, for the fingertip coordinates of the robot arm, the Tool Center Point (TCP) can be directly retrieved from the API and used. Furthermore, the robot coordinate system can be converted to a camera coordinate system for comparison with a human 3D skeleton.

[0056] More specifically, a robot's joint angle is a value indicating how much each joint has rotated or moved. Denavit-Hartenberg (DH) parameters are a method that mathematically expresses the relationship between each link and joint of a robot and are used to construct a robot's kinematic model. Forward kinematics is a method for calculating the position and orientation of each part of a robot when the joint angles are known, and calibration is the process of determining the relationship between different coordinate systems (e.g., the robot coordinate system and the camera coordinate system) to enable coordinate transformation.

[0057] First, in the data collection phase, joint angles are obtained, and the angle value θ_j of each joint can be collected in real time using the robot's real-time library. The robot generally has multiple joints (e.g., 6), and each joint performs rotational motion.

[0058] Next, a transformation to 3D coordinates is performed. Denavit-Hartenberg (DH) parameters are used to represent the geometric relationship between each joint and link of the robot, and through this, forward kinematics is performed, and for each link, the DH parameters (aj , α j , d j , θ j ) can be defined. Here, a j is the link length, α j is the twist angle of the link, d j is the link offset, θ j is the joint angle (value measured in real time). The Tool Center Point (TCP) coordinates of the robot arm can be calculated using the transformation matrix up to the last joint, or the Tool Center Point (TCP) coordinates obtained directly using the RTDE library of the UR robot can be used. Through this process, the 3D position of each joint can be obtained relative to the robot base coordinate system as shown in Fig. 3b.

[0059] Next, coordinate system transformation is performed. The robot's coordinate system is transformed into the camera coordinate system for comparison with human skeleton data. To do this, an Eye To Hand Calibration is performed as shown in Fig. 3c to obtain a transformation matrix between the two coordinate systems.

[0060] Finally, skeleton data of the robot can be extracted by obtaining 3D coordinates for each joint of the robot based on the camera coordinate system, and this data can then be compared with human skeleton data and used in a collision avoidance algorithm.

[0061]

[0062] Motion vector generation unit (133)

[0063] The motion vector generation unit (133) is configured to generate a motion vector of a person and a motion vector of a robot using skeleton information of a person and skeleton information of a robot, respectively, and can be configured to generate a motion vector of a person and a motion vector of a robot using the difference between a previous frame and a current frame using 3D coordinate information of a plurality of points stored according to a preset time interval.

[0064] The motion vector generation unit (133) stores three-dimensional skeleton data of a person and a robot over time and performs the role of tracking the motion vector of each joint point based on this, stores the skeleton data input in real time in frames, and maintains data from a certain period of time in the past to be used for motion analysis.

[0065] For example, looking at the data storage structure, the Time Window can store skeleton data for N past frames relative to the current frame. For instance, when N = 10, it can hold a total of 10 frame data points, including the current frame, as shown in Fig. 4. Here, the data structure consists of Person Skeleton Data Pi(t) and Robot Skeleton Data Rj(t). Here, Pi(t) is the 3D coordinate of the i-th joint point at time t, and Pi(t) = (x i (t), y i (t), z i It is expressed as (t). Similarly, Rj(t) is the 3D coordinate of the j-th joint point at time t, where Rj(t) = (x j (t), y j (t), z j It is expressed as (t)).

[0066] The method for generating motion vectors is as follows. First, two time points t_1 and t_2 are selected for comparison to generate vectors by setting a time interval. Generally, the current frame time t and a specific past time t - Δt are used. For example, the current frame time t and the past frame time t' = tk · Δt are compared, where k is the frame interval and Δt is the time interval between frames.

[0067]

[0068] [Mathematical Formula: Correlation between each frame and time]

[0069] The generation of skeleton motion vectors consists of calculating the vector V_i(t) for each joint point i. The formula for calculating the vector is Vi(t) = Pi(t) - Pi(t'), where Vi(t) is the motion vector of the i-th joint from time t' to t, and Pi(t) and Pi(t') are the positions of the i-th joint at times t and t', respectively.

[0070]

[0071]

[0072] [Mathematical Formula: Calculation of Skeleton Motion Vector]

[0073] An example of application is comparing current data with data from 5 frames prior. When the frame rate is 30fps, k = 5 and Δt = 1 / 30 second, and the past time is t' = t - 5 · 1 / 30 = t - 1 / 6 second. Through this, the movement vector V_i(t) for each joint of a person can be calculated to identify movement during the last 1 / 6 second.

[0074]

[0075] [Mathematical Formula: Frame Time Example]

[0076] The operation of the motion vector generation unit (133) first involves storing three-dimensional skeleton data of a person and a robot, which is input in real-time during the data collection and storage stage, in frame units. It maintains up to N frames of data, and deletes the oldest data when new data is input. Next, during the vector generation stage, it generates motion vectors using past and present skeleton data according to the time interval set. It calculates and stores vectors for each joint point. Finally, during the data output stage, the generated vectors are used for risk assessment or robot vector adjustment and control in subsequent stages, and the vector data can be utilized for attribute analysis such as the magnitude and direction of the movement.

[0077] In this way, by expressing the positional changes of joint points over time in vector form using motion vectors generated by skeleton extraction, the direction and velocity of movement can be identified, which can then be utilized for collision risk prediction, motion pattern analysis, and robot motion planning.

[0078]

[0079] Vector filtering unit (134)

[0080] The vector filtering unit (134) may be configured to filter or apply weights to at least one specific motion vector deemed important among a plurality of motion vectors generated by the motion vector generation unit. The collision risk determination unit (135) may be configured to determine the collision risk between a person and a robot using the motion vector filtered by the vector filtering unit (134) or using the weighted vector value. Here, the vector filtering unit may be configured to filter motion vectors for points in a specific area that is pre-set among a plurality of points. For example, the specific area may be a pre-set area with a high probability of collision or a dangerous area in case of collision, such as a person's hand or face area, a robot's fingertips, or an area with a large radius of rotation.

[0081] Additionally, the vector filtering unit (134) may be configured to filter the motion vector based on at least one of the distance between the person and the robot, the magnitude of the vector, and area information, or to apply weights to the motion vector based on at least one of the distance between the person and the robot, the magnitude of the vector, and area information.

[0082] In this way, the vector filtering unit (134) performs an operation to more efficiently determine the risk of collision and improve safety by selecting representative vectors that are deemed important in the skeleton data of a person and a robot.

[0083] In this invention, three methods for filtering specific vectors are described, and these methods may be used in combination.

[0084] As a first example, the shortest distance-based selection method is intended to identify the most dangerous point in terms of distance between the human and the robot. The Euclidean distance D_ij(t) between the human skeleton point P_i(t) and the robot skeleton point R_j(t) is calculated using the equation below to find the pair of points (i) with the minimum distance. * , j * By selecting ), motion vectors of points with the shortest distance can be filtered. Additionally, motion vectors can be filtered by filtering pairs of points with a distance within a predetermined criterion.

[0085]

[0086]

[0087] [Mathematical Formula: Shortest Distance-Based Selection Formula Based on Euclidean Distance Calculation]

[0088] As a second example, the momentum-based selection method is intended to identify the risk of collision in the most active parts of movement. By calculating the magnitude (velocity) of each skeleton vector using the equation below and selecting the joint point with the maximum vector magnitude, the vectors with the largest magnitudes can be filtered. Additionally, corresponding motion vectors whose vector magnitude is greater than or equal to a predetermined threshold can be filtered.

[0089]

[0090]

[0091] [Mathematical Formula: Vector Momentum-based Selection Formula]

[0092] As a third example, the weighted selection method involves increasing the importance of specific areas with a high risk of collision—such as a human hand, face region, or a robot's fingertips—and reflecting this in the risk assessment. Importance is adjusted by assigning a weight to each joint point corresponding to a specific area, such as the hand or face region, and the vector magnitude S to which the weights are applied i Calculate (t). The corresponding mathematical formula is as follows.

[0093]

[0094]

[0095] [Mathematical Formula: Weighted Selection Formula]

[0096]

[0097] Collision risk judgment unit (135)

[0098] The collision risk determination unit (135) may be configured to determine the collision risk and potential risk level between a person and a robot based on the movement vector of a person and the movement vector of a robot using vectors selected from the vector filtering unit (134) or all movement vectors. For example, it may be configured to determine the collision risk using at least one of the distance, speed, and direction of the movement vector of a person and the movement vector of the robot.

[0099] The collision risk judgment unit (135) comprehensively analyzes the movement and position information of the person and the robot to evaluate the risk level for predicting the possibility of a collision, and then adjusts the robot's operation or generates an alarm according to the risk level.

[0100] For example, collision risk can be evaluated by analyzing the interaction between a person's motion vector V_i(t) and a robot's motion vector U_j(t). Various algorithms can be applied to calculate the risk level by considering the direction, velocity, and distance of the motion vectors.

[0101] As an example, we will explain a method to evaluate the likelihood of movements colliding by calculating the dot product between motion vectors and applying a formula that numerically expresses the risk level.

[0102] First, human movement vector V i (t) and the robot's motion vector U j The inner product between (t) is calculated to determine whether the movements are in a direction of collision. The mathematical formulas for calculating the inner product and the cosine angle are as follows.

[0103]

[0104] [Mathematical Formula: Dot Product and Cosine Angle between Vectors]

[0105] Here, as the value of cosθ approaches 1, it moves in the same direction, and as it approaches -1, it moves in the opposite direction.

[0106] In addition, the risk R(t) can be calculated by combining distance, speed, and directionality, and can be expressed as shown in the following mathematical formula.

[0107]

[0108] [Mathematical Formula: Formula for Calculating Risk R(t)]

[0109] Here, f is a function that calculates the risk level, and various forms of functions can be applied. For example, it can be defined by a mathematical formula of the following form.

[0110]

[0111] [Mathematical Formula: Example of Risk Calculation Function Formula]

[0112] Here w i , w j is the weight of the human and robot joints, and D ij (t) is the distance between the points, and cosθ is the directionality between the motion vectors.

[0113] Here, the risk level is classified based on the calculated risk value R(t), and corresponding countermeasures are taken. The classification of risk levels and the response methods may vary depending on the system requirements and settings. For example, risk levels can be classified as 'Safe', 'Caution', or 'Danger', and robot motion control or notification methods appropriate to each level can be applied.

[0114] For example, a first boundary value for distinguishing between the safe and caution stages and a second boundary value for distinguishing between the caution and danger stages can be set for the risk level R(t), and the risk level can be classified into danger, caution, and safety levels based on R(t).

[0115] For example, at the safety level, the robot continues to perform its existing planned actions. At the caution level, for instance, the robot's speed can be reduced, or visual or auditory alerts can be provided to the user simultaneously. Additionally, at the danger level, the robot's movement path can be altered to move in a safe direction, and a loud alarm can be generated to alert the user of the danger.

[0116] In addition to these exemplary methods, such risk assessment methods can apply various safety evaluation techniques, including machine learning algorithms, statistical modeling, and rule-based systems. The risk level is calculated according to the selected method, and based on the results, the robot's operation can be controlled or notifications provided to the user.

[0117]

[0118] Robot movement control unit (136)

[0119] The robot movement control unit (136) may be configured to change the movement of the robot to reduce the collision risk when the collision risk is above a predetermined standard. For example, the robot movement control unit (136) may be configured to reduce the movement speed of the robot when the collision risk is at a first level, which is a caution level, and may be configured to change both the movement speed and direction of the robot when the collision risk is at a second level, which is a risk level higher than the first level.

[0120] Additionally, the robot movement control unit (136) may be configured to change the existing movement path by adjusting the robot's current movement vector based on the robot's movement vector adjusted using the human's movement vector in order to more effectively avoid collisions between the human and the robot. Safety can be ensured by controlling the robot's movements in real time in this manner.

[0121] There are various methods for adjusting a robot's motion vectors and they are not limited to a specific method. As an example, we will explain a method in which vector operations are performed on the current robot's motion vector using a human motion vector, and the adjusted robot is controlled using the motion vector.

[0122] First, let Urobot(t) be the motion vector associated with the robot's original movement, and Vhuman(t) be the motion vector of the estimated human movement. If the risk level in the risk assessment module exceeds a certain threshold, the robot's motion vector can be adjusted as shown in the following mathematical formula.

[0123]

[0124] [Mathematical Formula: Robot Movement Vector Adjustment Formula]

[0125] Here, U'robot(t) is the motion vector of the adjusted robot, and α is the adjustment coefficient. This equation allows for effective prevention of collisions by adjusting the robot's motion vector in a direction that increases the probability of avoiding collisions by considering the human motion vector.

[0126] The robot movement control unit (136) controls the actual movement of the robot based on the adjusted robot movement vector U'robot(t), and can adjust the speed, direction, etc. of the robot in real time based on the adjusted vector. Through this method, the robot can adjust its movement path in a way that minimizes the risk by reflecting the movement of a person in a dangerous situation, and can prevent collisions.

[0127]

[0128] FIG. 5 is a flowchart illustrating a real-time collision prevention method for preventing collisions between a person and a robot in one embodiment of the present invention.

[0129] First, human skeleton information can be extracted by the human skeleton extraction unit (131) using an image obtained through the camera unit (120) and an artificial intelligence model, and three-dimensional coordinates for each point of the skeleton can be obtained. (S510)

[0130] In addition, real-time joint information of the robot can be collected by the robot skeleton extraction unit (132), and skeleton information of the robot can be extracted by using the transformation between the robot coordinate system and the camera coordinate system, thereby obtaining the 3D coordinates of each joint of the robot. (S520)

[0131] Next, the motion vector generation unit (133) can generate a motion vector of a person and a motion vector of a robot, respectively, using the skeleton information of a person and the skeleton information of a robot. (S530) For example, the generation of the motion vector of a skeleton can be utilized to analyze attributes such as the magnitude and direction of movement by calculating and storing vector values ​​for each point of the skeleton.

[0132] Additionally, specific motion vectors of high importance can be filtered by the vector filtering unit (134). (S540) For example, vectors of high importance among the motion vectors of a person and a robot can be selected and filtered, or weights can be applied, and methods such as a selection method based on the shortest distance between points, a selection method based on the magnitude of the vector, or a method of applying weights to a specific area can be used.

[0133] Next, the collision risk determination unit (135) can determine the collision risk between the person and the robot based on the movement vector of the person and the movement vector of the robot (S550). At this time, the movement and position information of the person and the robot is comprehensively analyzed to evaluate the risk level for predicting the possibility of a collision, and motion control or notification control of the robot can be performed according to the risk level.

[0134] Next, the robot's movement can be modified by the robot movement adjustment unit (136) to reduce the risk of collision (S560). For example, the robot's movement can be adjusted in real time by considering the direction of movement of the person by adding a value obtained by applying an adjustment coefficient to the person's movement vector to the current robot's movement vector.

[0135]

[0136] The various embodiments described herein may be implemented by hardware, middleware, microcode, software, and / or combinations thereof. For example, the various embodiments may be implemented in one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), processors, controllers, microcontrollers, microprocessors, other electronic units designed to perform the functions presented herein, or combinations thereof.

[0137] Additionally, for example, various embodiments may be stored or encoded on a computer-readable medium containing instructions. Instructions stored or encoded on a computer-readable medium may enable a programmable processor or another processor to perform a method, for example, when the instructions are executed. A computer-readable medium includes both a computer storage medium and a communication medium including any medium that facilitates the transfer of a computer program from one place to another. A storage medium may be any available medium accessible by a computer. For example, such a computer-readable medium may include RAM, ROM, EEPROM, CD-ROM or other optical disc storage media, magnetic disc storage media or other magnetic storage devices, or any other medium that can be used to transport or store desired program code in the form of instructions or data structures accessible by a computer.

[0138] Such hardware, software, firmware, etc., may be implemented within the same device or in individual devices to support the various operations and functions described in this specification. Additionally, components, units, modules, components, etc., described as “parts” in this invention may be implemented together or individually as separate but interoperable logic devices. Descriptions of different features for modules, units, etc., are intended to highlight different functional embodiments and do not necessarily imply that they must be realized by individual hardware or software components. Rather, functions associated with one or more modules or units may be performed by individual hardware or software components or integrated within common or individual hardware or software components.

[0139] Although operations are depicted in a specific order in the drawings, it should not be understood that these operations must be performed in the specific order depicted or in a sequential order to achieve the desired result, or that all depicted operations must be performed. In any environment, multitasking and parallel processing may be advantageous. Furthermore, the distinction between the various components in the above-described embodiments should not be understood as requiring such distinction in all embodiments, and it should be understood that the described components may generally be integrated together into a single software product or packaged into multiple software products.

[0140] The present invention has been described with reference to the embodiments illustrated in the drawings, but this is merely illustrative, and those skilled in the art will understand that various modifications and equivalent alternative embodiments are possible therefrom. Accordingly, the true technical scope of protection of the present invention should be determined by the technical spirit of the appended claims.

Claims

1. In a real-time collision avoidance system for preventing collisions between humans and robots, A human skeleton extraction unit configured to extract human skeleton information; A robot skeleton extraction unit configured to extract skeleton information of a robot; A motion vector generation unit configured to generate a motion vector of the person and a motion vector of the robot, respectively, using the skeleton information of the person and the skeleton information of the robot; and A collision risk determination unit configured to determine the collision risk between the person and the robot based on the motion vector of the person and the motion vector of the robot. Includes, A real-time collision avoidance system in which the skeleton information of the person and the skeleton information of the robot each include multiple three-dimensional coordinate information for multiple points.

2. A real-time collision prevention system according to claim 1, wherein the collision risk determination unit is configured to extract the shortest distance between the person and the robot using a plurality of three-dimensional coordinate information included in the skeleton information of the person and a plurality of three-dimensional coordinate information included in the skeleton information of the robot, and to determine the collision risk based on the magnitude of the shortest distance.

3. A real-time collision avoidance system according to claim 1, wherein the motion vector generation unit is configured to generate the motion vector of the person and the motion vector of the robot by using the difference between the previous frame and the current frame using three-dimensional coordinate information for a plurality of points stored according to a preset time interval.

4. In claim 1, further comprising a vector filtering unit configured to filter at least one specific motion vector among a plurality of motion vectors generated by the motion vector generating unit, A real-time collision prevention system in which the above collision risk determination unit is configured to determine the collision risk between the person and the robot using motion vectors filtered by the above vector filtering unit.

5. A real-time collision prevention system according to claim 4, wherein the vector filtering unit is configured to filter motion vectors for points in a preset specific area among a plurality of points.

6. A real-time collision avoidance system according to claim 4, wherein the vector filtering unit is configured to filter motion vectors based on at least one of the distance between the point of the person and the robot, the magnitude of the vector, and area information.

7. A real-time collision avoidance system according to claim 6, wherein the area information is information related to whether it is a hand or face area, and is configured to calculate the motion vector by applying weights based on the area information.

8. A real-time collision prevention system according to claim 1, wherein the collision risk determination unit is configured to determine the collision risk using at least one of the distance, velocity, and direction of the person's motion vector and the robot's motion vector.

9. A real-time collision avoidance system according to claim 1, further comprising a robot movement adjustment unit configured to change the movement path of the robot to reduce the collision risk when the collision risk is greater than or equal to a predetermined standard.

10. A real-time collision avoidance system according to claim 9, wherein the robot movement control unit is configured to reduce the movement speed of the robot when the collision risk level is at a first level, and is configured to change both the movement speed and direction of the robot when the collision risk level is at a second level higher than the first level.

11. A real-time collision avoidance system according to claim 9, wherein the robot motion control unit is configured to change the movement path of the robot based on the robot's motion vector adjusted using the human motion vector.

12. A real-time collision avoidance system according to claim 1, wherein the robot skeleton extraction unit is configured to acquire three-dimensional skeleton data of the robot by utilizing a transformation between a robot coordinate system and a camera coordinate system based on the joint angles of the robot.

13. In a real-time collision avoidance method for preventing collisions between a person and a robot, A step of extracting human skeleton information by a human skeleton extraction unit; A step of extracting skeleton information of a robot by a robot skeleton extraction unit; A step of generating a motion vector of the person and a motion vector of the robot, respectively, using the skeleton information of the person and the skeleton information of the robot by a motion vector generation unit; and A step of determining the collision risk between the person and the robot based on the movement vector of the person and the movement vector of the robot by the collision risk determination unit A real-time collision prevention method comprising, wherein the skeleton information of the person and the skeleton information of the robot each comprise a plurality of three-dimensional coordinate information for a plurality of points.

Images