Robotic system
By using the distance image sensor and image processing device of the robot system, blind spots are detected and calculated, solving the problem of blind spots in robot system monitoring and enabling effective monitoring of workers and other personnel and prevention of entry.
Patent Information
- Application Number
- CN201911035999.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2019-10-29
- Publication Date
- 2024-11-29
- Estimated Expiration
- 2039-10-29
AI Technical Summary
In existing robot systems, obstructions in the field of view of distance image sensors prevent the monitoring of workers entering through blind spots, creating monitoring blind zones.
A robotic system is used to acquire and process distance image information around the action area through distance image sensors and image processing devices. The system uses the reference distance image information to detect whether stationary objects are obstructing the boundary, calculates the size of the blind spot space, and implements countermeasures to prevent entry.
Even when there are obstructions in the field of view, it can effectively monitor and prevent workers from entering through blind spots, thus improving safety and reliability.
Smart Images

Figure CN111113374B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to robotic systems. Background Art
[0002] In the past, a robot system is known as follows: in the distance image continuously acquired by using a distance image sensor arranged above the workbench, a monitoring area with a boundary is set around the robot's action area, and the robot performs danger avoidance action by detecting workers etc. who are heading towards the monitoring area through the boundary 3 (for example, refer to patent document 1).
[0003] Prior art literature
[0004] Patent Literature
[0005] Patent Document 1: Japanese Patent Application Publication No. 2017-221985 Summary of the invention
[0006] Problem that the invention aims to solve
[0007] In the robot system of Patent Document 1, objects such as workbenches arranged in the monitoring area do not exceed the boundary of the monitoring area. Therefore, workers who enter the monitoring area through the boundary or workers who hide under the workbenches can be monitored reliably.
[0008] However, in actual work sites, objects such as workbenches are placed at the blocking boundary of the field of view of the range image sensor, forming a blind spot on the opposite side of the range image sensor across the object. Therefore, there is a disadvantage that the range image sensor cannot monitor the entry into the monitoring area through the blind spot.
[0009] An object of the present invention is to provide a robot system that can grasp the possibility of entry through a blind spot formed by an object even if an object is placed at a position away from the boundary of a monitoring area blocked in the field of view of an image sensor.
[0010] Solutions for solving problems
[0011] One embodiment of the present invention is a robot system, comprising: a robot; a range image sensor that temporally continuously acquires range image information of the periphery of the action area of the robot from above the action area; and an image processing device that processes the range image information acquired by the range image sensor, the image processing device comprising a storage unit that defines a monitoring area having a boundary that allows entry into the action area from the outside around the action area, and stores reference range image information, the reference range image information being the range image information of the boundary of the monitoring area when no object exists in the monitoring area, the image processing device detecting whether a stationary object existing in the monitoring area blocks the boundary in the field of view of the range image sensor based on the range image information acquired by the range image sensor and the reference range image information stored in the storage unit.
[0012] According to this scheme, if the distance image information of the surroundings of the action area is continuously acquired from the top of the action area of the robot by using the distance image sensor, the acquired distance image information is processed by the image processing device. The image processing device defines a monitoring area with a boundary that allows entry from the outside into the action area around the action area of the robot. Therefore, when an operator or the like crosses the boundary and enters the monitoring area, the distance information of any pixel in the distance image information changes with time, thereby being able to detect the operator or the like entering the monitoring area.
[0013] On the other hand, when a stationary object is disposed in the monitoring area, the distance information in the distance image information does not change over time, and the stationary object is not detected as an entering object.
[0014] In this case, according to the present scheme, reference distance image information is stored in the storage unit, wherein the reference distance image information is distance image information of the boundary of the monitoring area when there is no object in the monitoring area, and the image processing device detects whether a stationary object existing in the monitoring area blocks the boundary within the field of view of the distance image sensor based on the reference distance image information and the distance image information acquired by the distance image sensor.
[0015] In the case where a stationary object blocks a boundary, there is a possibility that a worker or the like can enter the monitoring area from the blocked boundary portion through a blind spot hidden by the stationary object and cannot be detected by the distance image sensor. That is, if the worker or the like passes through a blind spot from the boundary portion blocked by the stationary object, the distance information in the distance image information will not change over time, and the entry of the worker or the like cannot be detected. However, by detecting the presence or absence of a boundary blocked by a stationary object, the possibility of entry through a blind spot formed by the stationary object can be grasped. Then, if it can be determined that there is a possibility of entry, countermeasures to prevent entry can be implemented.
[0016] In the above scheme, the image processing device may further include a notification unit, which calculates the size of the space blocked by the stationary object when it is determined that the stationary object blocks the boundary, and notifies when the calculated size of the space exceeds a predetermined threshold.
[0017] According to this configuration, it is possible to determine whether a worker or the like can pass through or hide in a blind spot formed by a stationary object, and when the size of the space exceeds a predetermined threshold, the notification unit notifies the worker of the fact. If the notification is made, measures to prevent entry can be implemented.
[0018] In addition, in the above solution, the image processing device can also calculate the minimum cross-sectional area of the space as the size of the space.
[0019] According to this structure, it is possible to determine whether a worker or the like can pass through a blind spot formed by a stationary object or enter a part of the body into a blind spot formed by a stationary object, and when the size of the space exceeds a predetermined threshold, the notification unit notifies the worker of the fact. If the worker has been notified, measures to prevent entry can be implemented.
[0020] In addition, in the above scheme, the image processing device can also calculate the minimum size of the cross section of the space as the size of the space.
[0021] According to this structure, it is possible to determine whether a worker or the like can pass through a blind spot formed by a stationary object or enter a part of the body into a blind spot formed by a stationary object, and when the size of the space exceeds a predetermined threshold, the notification unit notifies the worker of the fact. If the worker has been notified, measures to prevent entry can be implemented.
[0022] In the above aspect, the image processing device may further include a display unit configured to display a space blocked by the stationary object when it is determined that the stationary object blocks the boundary.
[0023] According to this configuration, since the blind spot formed by the stationary object is displayed on the display unit, it is possible to easily implement a measure to prevent entry into the displayed blind spot.
[0024] In the above aspect, the image processing device may further include a three-dimensional model generating unit that generates a three-dimensional model of the space, and the display unit may display the three-dimensional model of the space generated by the three-dimensional model generating unit.
[0025] According to this configuration, since the blind spot formed by the stationary object is displayed on the display unit as a three-dimensional model, the angle of the blind spot is changed and displayed, and a measure to prevent entry into the blind spot can be easily implemented.
[0026] In the above aspect, the robot system may further include an imaging unit for imaging the robot and its surroundings, and the display unit may display the three-dimensional model by superimposing the image acquired by the imaging unit on the image.
[0027] According to this structure, the blind spot formed by the stationary object is displayed on the display unit as a three-dimensional model superimposed on the image of the robot, etc. actually obtained by the imaging unit, so it is easy to grasp the position of the robot, etc. relative to the blind spot, and it is easy to implement measures to prevent entry into the blind spot.
[0028] Effects of the Invention
[0029] According to the present invention, even if an object is placed at a position where the boundary of the monitoring area in the field of view of the image sensor is blocked, the possibility of entry through the blind spot formed by the object can be grasped. BRIEF DESCRIPTION OF THE DRAWINGS
[0030] Figure 1 It is a diagram showing the overall configuration of a robot system according to one embodiment of the present invention.
[0031] Figure 2 Yes means Figure 1 Side view of the robotic system.
[0032] Figure 3 It means by Figure 1 FIG. 1 is a diagram showing an example of a distance image of the outer side surface and bottom surface of a monitoring area acquired by a robot system.
[0033] Figure 4 It means in Figure 2 A side view of an example of a situation in which a stationary object is configured within a monitoring area in a robot system.
[0034] Figure 5 Yes means Figure 4 FIG. 1 is a diagram showing an example of a distance image of the outer side surface and bottom surface of the monitoring area in the case of .
[0035] Figure 6 Yes means Figure 5 The distance image and Figure 3 FIG. 1 is a diagram of a difference image of a distance image.
[0036] Figure 7 It means that the Figure 4 A stereogram of an example of a blind spot formed by a stationary object within the monitoring area.
[0037] Description of Reference Numerals
[0038] 1: Robotic system
[0039] 2: Robot
[0040] 3: Security Camera (Distance Image Sensor)
[0041] 4: Image processing device
[0042] A: Monitoring area
[0043] A1: Outer side (border)
[0044] A3: Inner side (border)
[0045] D: Minimum cross-sectional area
[0046] E: Minimum size
[0047] G: Action range (action area)
[0048] V: Field of view
[0049] X: Stationary object DETAILED DESCRIPTION
[0050] Hereinafter, a robot system 1 according to an embodiment of the present invention will be described with reference to the drawings.
[0051] like Figure 1 As shown, the robot system 1 of this embodiment includes: a robot 2; a safety camera (range image sensor) 3 disposed above an operating range (operating area) G of the robot 2; and an image processing device 4 connected to the safety camera 3.
[0052] exist Figure 1 In the example shown, the robot 2 is, for example, a six-axis articulated robot and has a spherical motion range G.
[0053] The security camera 3 can use a stereo camera or a TOF (Time of flight) type distance image sensor. The security camera 3 can acquire a distance image (distance image information) in which the distance from the center of the lens of the security camera 3 is detected for each pixel.
[0054] like Figure 1 as well as Figure 2 As shown in FIG. 1 , the security camera 3 has a field of view V, which does not interfere with the motion range G of the robot 2 and can capture a range surrounding the motion range G of the robot 2 in the entire circumference. That is, the field of view V of the security camera 3 extends radially from the center of the lens and expands to a ring-shaped range that does not include the motion range G of the robot 2. For example, the distance image acquired by the security camera 3 is as follows: Figure 3 The area indicated by diagonal lines in the figure is the range image.
[0055] The image processing device 4 is composed of a processor and a memory. The image processing device 4 processes the distance image acquired by the security camera 3, such as Figure 2 As shown, the monitoring area A is defined. For example, the monitoring area A has a certain cross-sectional shape of a rectangle, such as Figure 2 As shown in FIG. 1 , it is a square annular area arranged at a position outside the action range G of the robot 2 in the entire circumference. Figure 3 In the distance image and Figure 2 The outer side surface A1 of the monitoring area A shown and the bottom surface A2 of the monitoring area A correspond to each other.
[0056] The security camera 3 obtains the distance information to the object within the field of view V for each pixel, but excludes the area outside the outer side (boundary) A1 of the monitoring area A and the area below the bottom surface A2 of the monitoring area A from the monitoring object according to the size of the distance information.
[0057] That is, the image processing device 4, Figure 2 Point B1, which is further outside the outer side surface A1 of the monitoring area A, is arranged at a position farther than the outer side surface A1 of the monitoring area A in the straight line L1 connecting the point B1 and the lens center of the security camera 3. Therefore, the distance information of the corresponding pixel in the distance image is larger than that of the point B2 in the outer side surface A1 of the monitoring area A corresponding to the same pixel. Therefore, it can be excluded from the monitoring object as a point outside the monitoring area A.
[0058] In addition, Figure 2 Point C1 is located further below the bottom surface A2 of the monitoring area A. In the straight line L2 connecting point C1 and the lens center of the security camera 3, point C1 is arranged at a position farther than the bottom surface A2 of the monitoring area A. Therefore, the distance information of the corresponding pixel in the distance image is larger than that of point C2 in the bottom surface A2 of the monitoring area A corresponding to the same pixel. Therefore, it can be excluded from the monitoring object as a point outside the monitoring area A.
[0059] In addition, for stationary objects existing in the monitoring area A (refer to Figure 5 )X, for example, an object that maintains a stationary state, such as a workbench or a control device, is set as a stationary object X and is excluded from the monitoring object.
[0060] In this case, in this embodiment, the image processing device 4 uses the distance image acquired by the security camera 3 to detect whether the stationary object X existing in the monitoring area A blocks the outer side surface (boundary) A1 of the monitoring area A within the field of view (view) V of the security camera 3.
[0061] Specifically, the image processing device 4 has a storage unit (not shown) that stores a first reference distance image, which is a distance image of the outer side surface A1 and the bottom surface A2 of the monitoring area A when no stationary object X is configured, and can detect the stationary object X based on the difference between the distance image obtained by the security camera 3 and the reference distance image.
[0062] When a stationary object X is detected, the image processing device 4 detects whether the detected stationary object X is approaching a portion of the area that should be the outer side surface A1 or the inner side surface A3 of the monitoring area A.
[0063] Figure 3 represents an example of a reference distance image. Then, Figure 4 A side view showing a stationary object X disposed in a monitoring area A. Figure 5 An example of a distance image of the outer side surface A1 and the bottom surface A2 of the monitoring area A is shown. Figure 6 express Figure 3 The reference distance image and Figure 5 The difference image of the distance image.
[0064] exist Figure 4 In FIG. 1 , the hatched area in the monitoring area A is a blind spot formed by the stationary object X in the monitoring area A. Figure 4 In the example of FIG. 1 , the blind spot is formed close to a local portion of the outer side surface A1 of the monitoring area A.
[0065] In addition, Figure 6 In FIG. 8 , the hatched area represents the area where the outer side surface A1 of the monitoring area A is blocked by the stationary object X.
[0066] In this area, the stationary object X is located closer to the security camera 3 than the outer side surface A1 of the monitoring area A. Therefore, for each pixel in this area, smaller distance information than the outer side surface A1 can be obtained. Thus, the fact that the outer side surface A1 of the monitoring area A is blocked by the stationary object X can be detected.
[0067] That is, there is the possibility that when the field of view V is blocked by bringing the stationary object X close to the area that should be the outer side surface A1 of the monitoring area A, a worker or the like may enter the monitoring area A from the blocked outer side surface A1 through a blind spot hidden by the stationary object X, thereby being unable to be detected by the distance image sensor.
[0068] Therefore, by detecting the presence or absence of the outer side surface A1 blocked by the stationary object X, it is possible to grasp the possibility of entry through the blind spot formed by the stationary object X. Then, when it can be determined that there is a possibility of entry, the stationary object X is moved to a position where the outer side surface A or the inner side surface A3 is not blocked in the field of view V, or a wall is provided outside the outer side surface A1 of the monitoring area A to block the passage connected to the blind spot in advance, so that countermeasures to prevent the entry of workers and the like can be implemented.
[0069] Thus, the robot system 1 according to the present embodiment has the following advantages: it detects whether the field of view V of the security camera 3 is blocked by the stationary object X in the area where the stationary object X arranged in the monitoring area A overlaps with the outer side surface A1 of the monitoring area A, thereby being able to understand whether there is a possibility that an operator or the like may enter through a blind spot formed by the stationary object X.
[0070] In addition, in the present embodiment, the image processing device 4 can also calculate the size of the space blocked by the stationary object X, and the robot system 1 includes a notification unit that notifies the calculated size of the space when it exceeds a predetermined threshold. The notification unit can be used to notify by any method such as sound, light, or image display.
[0071] like Figure 7 As shown, as the size of the space, examples include the volume S of the space (hatched area) blocked by the stationary object X in the monitoring area A, the minimum cross-sectional area D of the space, the minimum dimension E of the cross-sectional area of the space, etc. Figure 7 In the figure, the space that becomes a blind spot is the space including the space below the bottom surface A2 of the monitoring area A.
[0072] When the stationary object X blocks the field of view V of the security camera 3 in the inner side surface A3 of the monitoring area A, and when the space has a large volume that can hide a person, the person hidden in the blind spot formed by the stationary object X before the robot system 1 is started cannot be detected by the security camera 3 and enters the action range G of the robot 2. Therefore, the volume S of the space needs to be kept small enough.
[0073] In addition, when the minimum cross-sectional area D of the space or the minimum dimension E of the cross section of the space is large enough to allow a person or a part of a person, such as a wrist or a foot, to pass through, the person can enter the action range G of the robot 2 without being detected by the security camera 3. Therefore, the minimum cross-sectional area D of the space or the minimum dimension E of the cross section also needs to be kept small enough.
[0074] In addition, the image processing device 4 may further include a display unit. The display unit is a monitor. The display unit displays the space blocked by the stationary object X. The space may be displayed in a different color in the distance image.
[0075] Furthermore, the image processing device 4 may include a three-dimensional model generating unit that generates a three-dimensional model of a space, and may cause a display unit to display the generated three-dimensional model of the space.
[0076] As a three-dimensional model of the space, a three-dimensional model of the space itself can be directly generated, or a three-dimensional model of the monitoring area A can be generated to indirectly display the space blocked by the stationary object X based on the presence or absence of areas missing from the three-dimensional model of the monitoring area A.
[0077] By displaying the three-dimensional model, the shape of the space blocked by the stationary object X can be confirmed from different viewing angles.
[0078] In addition, the robot system 1 may further include a photographing unit for photographing the robot 2 and its surroundings, and the display unit may superimpose a three-dimensional model on the image acquired by the photographing unit and display it. Thus, the blind spot formed by the stationary object X is actually superimposed on the image of the robot 2 etc. acquired by the photographing unit and displayed on the display unit as a three-dimensional model, so that the position of the robot 2 etc. relative to the blind spot can be easily grasped, and countermeasures to prevent entry through the blind spot can be easily implemented.
[0079] In addition, when there are two or more security cameras 3, when the space blocked by the stationary object X is defined as a blind spot for any security camera 3, the union of the spaces of the blind spots calculated by each security camera 3 may be calculated as the blind spot. In addition, when the space blocked by the stationary object X is defined as a blind spot for all security cameras 3, the intersection of the spaces of the blind spots calculated by each security camera 3 may be calculated as the blind spot.
Claims
1. A robot system, characterized in that: have: robot; A distance image sensor that continuously acquires distance image information within a range surrounding the action area in a full circumference from above the action area of the robot; as well as an image processing device for processing the distance image information acquired by the distance image sensor, The image processing device defines an annular monitoring area, the monitoring area has a certain rectangular cross-sectional shape, is arranged at a position surrounding the outside of the action area within the entire circumference, and has a boundary that can be entered from the outside into the action area. The image processing device has a storage unit, the storage unit stores reference distance image information, the reference distance image information is the distance image information of the boundary of the monitoring area when no object exists in the monitoring area. The image processing device detects whether a stationary object existing in the monitoring area blocks the boundary in the field of view of the distance image sensor based on the distance image information acquired by the distance image sensor and the reference distance image information stored in the storage unit.
2. The robot system according to claim 1, characterized in that: The image processing device includes a notification unit that calculates the size of a space blocked by the stationary object when determining that the stationary object blocks the boundary, and performs notification when the calculated size of the space exceeds a predetermined threshold.
3. The robot system according to claim 2, characterized in that: The image processing device calculates a minimum cross-sectional area of the space as the size of the space.
4. The robot system according to claim 2, characterized in that: The image processing device calculates a minimum dimension of a cross section of the space as the size of the space.
5. The robot system according to claim 1, characterized in that: The image processing device includes a display unit configured to display a space blocked by the stationary object when determining that the stationary object blocks the boundary.
6. The robot system according to claim 5, characterized in that: The image processing device includes a three-dimensional model generating unit that generates a three-dimensional model of the space. The display unit displays the three-dimensional model of the space generated by the three-dimensional model generation unit.
7. The robot system according to claim 6, characterized in that: The robot system includes a photographing unit for photographing the robot and its surroundings. The display unit displays the three-dimensional model by superimposing it on the image acquired by the imaging unit.
Citation Information
Patent Citations
Robot system
JP2017221985A
Object detection sensor
JP2011215772A
System and method for monitoring entry of object into surrounding area of robot
US20140207285A1
