Mobile robot safety monitoring system

Abstract

This system provides a safety monitoring system that can detect when a target, including a person, approaches a mobile robot, even while the robot is in motion, and can perform safety monitoring of the mobile robot. [Solution] The closest approach point of the detection target 82 is recognized based on multiple detection target points Pmd obtained from among multiple laser measurement points Pm by image recognition of the camera image PC. If it is estimated that at least a portion of the mobile robot 10 will reach the closest approach point within a predetermined determination time, an emergency stop signal is sent to stop the movement of the mobile robot 10. Therefore, it is possible to detect when the detection target 82 approaches the mobile robot 10 beyond a predetermined limit, regardless of whether the mobile robot 10 is moving or not. As a result, it is possible to monitor the safety of the mobile robot 10 even while it is moving.

Description

Technical Field

[0001] The present disclosure relates to a safety monitoring system for mobile robots.

Background Art

[0002] Patent Document 1 describes a mobile robot with a robotic arm mounted on a vehicle body that is a carriage. The mobile robot of this Patent Document 1 includes a slider and a non-contact area sensor attached to the slider. Then, during the operation of the mobile robot of Patent Document 1, the area sensor is deployed by the slider to detect the intrusion of a person into the working area of the mobile robot.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In the automation of factory production and the service field, the use of mobile robots combined with manipulators such as movable carriages and robotic arms is expanding. Since the manipulator moves in three-dimensional directions, it is necessary to detect the intrusion of a person into the surroundings. Here, in the case of a non-mobile robot, person detection by a laser scanner installed on a non-mobile object such as the floor or equipment near the robot has been carried out. However, in the case of a mobile robot, the working places are diverse, and sensors cannot be installed on non-mobile objects. Therefore, it is required to detect the intrusion of a detection target such as a person into the working area only by the sensors provided on the mobile robot.

[0005] Furthermore, while a function to cause sensors to ignore equipment and other objects within a specific area, i.e., a sensor monitoring range masking function, is common in safety monitoring for stationary robots, such a masking function cannot be used for mobile robots because their work locations are unpredictable. Therefore, mobile robots need to avoid treating equipment and other objects that do not need to be detected as obstacles without using a sensor monitoring range masking function.

[0006] For these reasons, adopting the safety monitoring technology for the mobile robot described in Patent Document 1 is one possible approach. However, with the mobile robot described in Patent Document 1, it takes time to extend the slider and deploy the area sensor. Furthermore, the area sensor cannot be deployed while the mobile robot is moving, so the mobile robot cannot perform robotic tasks while it is moving. The inventors found the above to be the result of their detailed investigation.

[0007] In view of the above points, this disclosure aims to provide a safety monitoring system that can detect when a target, including a person, approaches a mobile robot, even while the mobile robot is in motion, and to perform safety monitoring of the mobile robot. [Means for solving the problem]

[0008] To achieve the above objectives, the mobile robot safety monitoring system described in this disclosure, from one perspective, A safety monitoring system for a mobile robot (10) having a trolley (12) that travels along a travel path (81) and a manipulator (14) attached to the trolley, A camera (22) is installed on the mobile robot to take pictures of the area around the mobile robot, A laser sensor (24) is installed on the mobile robot and detects objects located within a laser detection range (RL) that at least partially overlaps with the shooting range (Rcm) captured by the camera and extends around the mobile robot. An image recognition unit (262) identifies the target image region (ARh) occupied by a detection target (82) including a person from the camera image (PC) captured by the camera, A detection target point extraction unit (263) extracts multiple detection target points (Pmd) that indicate the three-dimensional position on the surface of the object to be detected from among multiple laser measurement points (Pm) measured by a laser sensor, based on the target image region, The nearest point recognition unit (264) recognizes the nearest point position (Xnr), which is the three-dimensional position of the detected target that is closest to the mobile robot, based on multiple detected target points, The system includes an emergency stop determination unit (265) that estimates whether at least a portion of the mobile robot will reach the closest point position within a predetermined determination time (T1r) based on the relative velocity (Vr) of each part (12, 142, 143, 144) of the mobile robot with respect to the travel path, and if it is estimated that at least a portion of the mobile robot will reach the closest point position within the determination time, it sends an emergency stop signal (Ssp) to stop the movement of the mobile robot.

[0009] In this way, it is possible to detect when the target object approaches the mobile robot beyond a predetermined limit, regardless of whether the mobile robot is moving or not, thus enabling safety monitoring of the mobile robot even while it is in motion.

[0010] In addition, each element in the application documents may be given a reference numeral in parentheses. In this case, the reference numeral merely indicates one example of the correspondence between the element and the specific configuration described in the embodiments described later. Therefore, this disclosure is not limited in any way by the inclusion of such reference numerals. [Brief explanation of the drawing]

[0011] [Figure 1] This diagram schematically shows a mobile robot being monitored for safety and a person approaching the mobile robot as a detection target in the first embodiment. [Figure 2]This is a block diagram showing the input / output system of a safety monitoring system that performs safety monitoring around a mobile robot in the first embodiment. [Figure 3] This is a schematic plan view of the camera and laser sensor of the safety monitoring system, the mobile robot, and the surrounding area of ​​the mobile robot in the first embodiment. [Figure 4] This diagram schematically shows a camera image captured by a camera of a safety monitoring system and a target image region within that camera image in the first embodiment. [Figure 5] This flowchart shows the control process performed by the monitoring and control device of the safety monitoring system in the first embodiment. [Figure 6] This figure shows a plurality of laser measurement points measured by the laser sensor of the safety monitoring system in the first embodiment. [Figure 7] This is a magnified view of section VII in Figure 6, showing a selection of detection points extracted from multiple laser measurement points. [Figure 8] This is a flowchart showing the subroutine executed in step S106 of Figure 5. [Figure 9] In the second embodiment, this is a flowchart showing the control process performed by the monitoring and control device of the safety monitoring system, and corresponds to Figure 5. [Figure 10] In the second embodiment, the figure schematically illustrates the nearest detection point, correction amount, and detection point correction position along with the point cloud to be detected. [Figure 11] This figure shows the mathematical model used in step S208 of Figure 9. [Figure 12] In the second embodiment, the image shows multiple laser measurement points superimposed on an image of a person to be detected, and also indicates points on the target that have not been detected by the laser. [Figure 13] In the second embodiment, this figure shows the point cloud to be detected and points on the detection target that are not detected by the laser. [Figure 14]This is a flowchart showing the control process executed by the monitoring control device of the safety monitoring system in the third embodiment, which corresponds to FIG. 5. [Figure 15] This is a plan view schematically showing, in a plan view, the camera and laser sensor of the safety monitoring system, the mobile robot, and the periphery of the mobile robot in the fourth embodiment, which corresponds to FIG. 3.

Embodiments for Carrying Out the Invention

[0012] Hereinafter, each embodiment will be described while referring to the drawings. In each of the following embodiments, parts that are identical or equivalent to each other are denoted by the same reference numerals in the drawings.

[0013] (First Embodiment) As shown in FIGS. 1 and 2, the mobile robot 10 of the present embodiment is a work robot that autonomously travels unmanned, for example, inside a factory or other premises, and is controlled by a robot control device 11. For example, the mobile robot 10 travels unmanned toward a destination point inside the premises according to a control signal from the robot control device 11, and when it arrives at the destination point, it performs a predetermined task unmanned.

[0014] As shown in FIGS. 1 and 3, the mobile robot 10 has a carriage 12 that travels on a travel path 81 formed on a part of the floor surface 80 inside the premises, and a manipulator 14 attached to the carriage 12. For example, the carriage 12 has a plurality of wheels that roll and contact the travel path 81 at the lower part of the carriage 12, and travels on the travel path 81 by driving the plurality of wheels with an electric motor. In FIG. 1, the upper side on the paper surface is the upper side in the vertical direction Dg inside the premises. Also, as shown in FIG. 3, when viewed in the direction along the vertical direction Dg, the carriage 12 has a rectangular peripheral portion 121.

[0015] Note that the floor surface 80 inside the premises and the travel path 81 included in the floor surface 80 are in a planar shape that extends in a direction perpendicular to the vertical direction Dg. Also, in the description of the present embodiment, the view in the direction from the upper side to the lower side along the vertical direction Dg may be referred to as a plan view in some cases.

[0016] As shown in Figure 1, the manipulator 14 of the mobile robot 10 is specifically a robot arm, and has a base 141, an end 142, a plurality of joints 143, and a plurality of intermediate connecting parts 144. The manipulator 14 is connected to the upper part of the trolley 12 at the base 141. For example, the base 141 of the manipulator 14 is made immobile relative to the trolley 12 in the vertical direction Dg, but is rotatable about an axis along the vertical direction Dg. When this base 141 is rotated relative to the trolley 12, for example by an electric motor, the entire manipulator 14 rotates relative to the trolley 12.

[0017] The tip 142 of the manipulator 14 is located on the opposite side of the manipulator 14 from the base 141 and has a structure capable of gripping, for example, an object. The multiple joints 143 and multiple intermediate connecting parts 144 of the manipulator 14 are located between the tip 142 and the base 141. The multiple intermediate connecting parts 144 of the manipulator 14 have a shape that extends longitudinally, for example, and are connected in series between the tip 142 and the base 141 within the manipulator 14. In the manipulator 14, the tip 142, joints 143, intermediate connecting parts 144, joints 143, intermediate connecting parts 144, joints 143, and base 141 are connected in series in that order. The manipulator 14, which is a robot arm, has a structure that allows bending and straightening at each of the multiple joints 143.

[0018] As described above, the mobile robot 10 operates unmanned, so a safety monitoring system 20 is provided to monitor the safety of the area around the mobile robot 10, as shown in Figures 1 to 3. Specifically, the safety monitoring system 20 monitors for the intrusion of detection targets 82 into the monitoring range Rs that surrounds the mobile robot 10 all around. In this embodiment, the detection targets 82 are specifically people, and objects other than people are excluded from the detection targets 82.

[0019] The monitoring range Rs is, in more detail, a predetermined three-dimensional area that needs to be monitored to ensure human safety, and is formed on the floor surface 80 to cover the entire mobile robot 10, with the mobile robot 10 as the reference point. Therefore, the monitoring range Rs moves along with the mobile robot 10 when the mobile robot 10 moves.

[0020] Specifically, the safety monitoring system 20 comprises multiple cameras 22, multiple laser sensors 24, and a monitoring control device 26. The multiple cameras 22 and multiple laser sensors 24 are mounted on the mobile robot 10, and the safety monitoring system 20 monitors the entire surroundings of the mobile robot 10 using the multiple cameras 22 and multiple laser sensors 24. The safety monitoring system 20 then uses the detection results from the cameras 22 and laser sensors 24 to identify the detection target 82 and accurately determine the relative position and relative distance of the detection target 82 to the mobile robot 10.

[0021] Multiple cameras 22 are connected to a monitoring and control device 26 via wireless or wired connection, enabling information communication. Image signals representing the camera images PC captured by each of the multiple cameras 22 are sequentially output to the monitoring and control device 26. Each of the multiple cameras 22 can capture both moving and still images. An example of a camera image PC captured by a camera 22 is shown in Figure 4.

[0022] As shown in Figures 1 and 3, multiple cameras 22 are provided on the mobile robot 10. In detail, the multiple cameras 22 are fixed to the trolley 12 of the mobile robot 10, and as shown in Figure 3, in a plan view they are arranged at intervals from each other on the periphery 121 of the trolley 12 and facing outwards from the trolley 12. That is, the multiple cameras 22 are arranged to photograph the area around the mobile robot 10, and the shooting range Rcm of each of the multiple cameras 22 is formed to extend from the camera 22 outwards from the trolley 12 in a plan view. Furthermore, the union of the shooting range Rcm of each of the multiple cameras 22 is formed to surround the mobile robot 10 around its entire circumference in a plan view, and extends to cover the entire monitoring range Rs, excluding blind spots near the cameras 22. Note that in Figure 3 and Figure 15 described later, dot-shaped hatching is applied to the shooting range Rcm of the cameras 22 for clarity.

[0023] Multiple laser sensors 24 are provided on each of the mobile robots 10. Specifically, two sets of these laser sensors 24 are provided and fixed to the trolley 12 of the mobile robot 10. In a plan view, the multiple laser sensors 24 are positioned at the corner 121a located at one end of the diagonal of the rectangular periphery 121 of the trolley 12, and at the corner 121b located at the other end of the diagonal.

[0024] Multiple laser sensors 24 emit laser light while scanning three-dimensionally in the surrounding area, and detect objects located within the laser detection range RL that the laser light can reach. Objects detected by the laser sensors 24 include all objects that reflect laser light, such as detection targets 82 including people, and equipment and machinery fixed on the floor surface 80 of the premises.

[0025] Furthermore, the laser detection range RL in Figure 3 is the range in which the laser sensor 24 can detect objects, and is formed to extend around the mobile robot 10. The union of multiple laser detection ranges RL is formed to surround the mobile robot 10 in a plan view over its entire circumference and extends to cover the entire monitoring range Rs. For example, in this embodiment, the entire union of the camera 22's shooting range Rcm is contained within the union of the laser detection ranges RL, so each of the multiple laser detection ranges RL at least partially overlaps with the shooting range Rcm of one of the cameras 22.

[0026] Specifically, each of the multiple laser sensors 24 in this embodiment is composed of LiDAR. Therefore, each of the multiple laser sensors 24 can detect various information such as the three-dimensional position of a point on the surface of an object that reflected the laser light, and its relative velocity to the object, by receiving and analyzing the reflected light that is reflected by the irradiated laser light from an object. The multiple laser sensors 24 are connected to the monitoring and control device 26 via wireless or wired connection for information communication, and the laser detection signals indicating the detection information detected by each of the multiple laser sensors 24 are sequentially output to the monitoring and control device 26. LiDAR stands for Light Detection And Ranging.

[0027] The monitoring and control device 26 shown in Figure 2 is an electronic control device configured as a microcomputer, equipped with a CPU, RAM, ROM, and non-volatile rewritable memory (not shown). In other words, the monitoring and control device 26 reads and executes a computer program stored in the ROM or non-volatile rewritable memory, which are non-transitional physical recording media. When this computer program is executed, a method corresponding to the computer program is performed. That is, the monitoring and control device 26 performs various control processes, such as the control process shown in Figure 5, which will be described later, according to its computer program.

[0028] Furthermore, the robot control device 11, like the monitoring and control device 26, is an electronic control device having the configuration of a microcomputer as described above. The monitoring and control device 26 and the robot control device 11 are connected to each other so as to be able to communicate information wirelessly or via a wired connection. For example, the monitoring and control device 26 and the robot control device 11 are mounted on the mobile robot 10 and arranged inside the trolley 12.

[0029] Furthermore, as shown in Figure 2, the monitoring and control device 26 is functionally equipped with a measurement control unit 261, an image recognition unit 262, a detection target point extraction unit 263, a nearest point recognition unit 264, and an emergency stop determination unit 265 in order to execute the control process shown in Figure 5.

[0030] Figure 5 is a flowchart showing the control process executed by the monitoring and control device 26. This control process in Figure 5 is executed periodically and repeatedly while the mobile robot 10 is in operation, regardless of whether the mobile robot 10 is moving or stationary and performing work.

[0031] As shown in Figure 5, first, in step S101, the image recognition unit 262 receives image signals from each of the multiple cameras 22, thereby obtaining the camera image PC captured by each camera 22. Then, as shown in Figure 4, the image recognition unit 262 identifies the target image region ARh occupied by the detection target 82 from the obtained camera image PC.

[0032] As an image recognition method to identify the target image region ARh, techniques such as bounding box method, segmentation, or skeletal estimation can be employed. The bounding box method is a method of estimating the range in the camera image PC where a detection target 82, such as a person, exists as a rectangle, while segmentation is a method of estimating the range in the camera image PC where a detection target 82 exists in an arbitrary shape. Skeletal estimation is a method of estimating the joint positions of a person displayed in the camera image PC.

[0033] Furthermore, since the image recognition performed in step S101 is for the purpose of safety monitoring, high processing speed is generally required. Among the multiple image recognition methods described above, the bounding box method, which allows for high-speed processing, or segmentation using thermal images, as described later, are advantageous. On the other hand, segmentation and skeleton estimation using ordinary images, rather than thermal images, can calculate the range of a person more accurately, thus reducing the possibility of overdetection, which would result in the incorrect extraction of objects other than people, which are the detection target 82. Therefore, segmentation and skeleton estimation using ordinary images can also be candidates for the image recognition method adopted in step S101. All of the image recognition methods described above have sufficiently high accuracy in distinguishing between the target image region ARh and other image regions. In step S101, the target image region ARh and other regions are distinguished from the camera image PC.

[0034] As shown in Figure 5, steps S102 and S103 are executed in parallel with step S101 described above. In step S102, the measurement control unit 261 receives laser detection signals from each of the multiple laser sensors 24, thereby obtaining multiple laser measurement points Pm measured by the laser sensors 24, as shown in Figure 6. In obtaining the laser measurement points Pm, the relative three-dimensional position of the laser measurement points Pm with respect to the mobile robot 10 is also obtained. In this embodiment, the three-dimensional position of the laser measurement points Pm is described as being represented by coordinate values ​​in a Cartesian coordinate system having x, y, and z axes.

[0035] In step S103, following step S102 in Figure 5, the detection target point extraction unit 263 projects the multiple laser measurement points Pm obtained in step S102 onto the camera image PC, as shown in Figure 1, according to predetermined coordinate transformation conditions. Specifically, the multiple laser measurement points Pm are projected onto the uv plane PLuv, which is a two-dimensional projection plane that extends to include the camera image PC, which is a two-dimensional image.

[0036] As a result, the detection target point extraction unit 263 obtains multiple projected points Pmx by projecting multiple laser measurement points Pm onto the uv plane PLuv, as shown in Figures 1 and 4. These multiple projected points Pmx correspond to each of the multiple laser measurement points Pm in a one-to-one relationship. In this embodiment, the two-dimensional position of the projected points Pmx is described as being represented by coordinate values ​​in a Cartesian coordinate system having a u axis and a v axis. Although multiple cameras 22 and laser sensors 24 are provided, in Figure 1, for the sake of simplicity, only one camera 22 and one laser sensor 24 are shown, and only one of the multiple camera images PC is shown.

[0037] Furthermore, although various coordinate transformation conditions can be assumed as described above, in this embodiment, the following equation F1 is adopted as the coordinate transformation condition.

number

[0038] Following steps S101 and S103 in Figure 5, the process proceeds to step S104. In step S104, the detection target point extraction unit 263 extracts multiple detection target points Pmd, which indicate the three-dimensional positions on the surface of the detection target 82, from among the multiple laser measurement points Pm obtained in step S102, based on the target image region ARh identified in step S101. These multiple detection target points Pmd are illustrated in Figure 7.

[0039] In detail, in order to extract multiple detection target points Pmd, the detection target point extraction unit 263 first extracts from the multiple projection points Pmx obtained in step S103 that fall within the target image region ARh (see Figure 4) in the camera image PC. Figure 4 shows several examples of multiple projection points Pmx that fall within the target image region ARh.

[0040] Then, as shown in Figure 7, the detection target point extraction unit 263 extracts the laser measurement points Pm that formed the basis of the extracted multiple projection points Pmx, i.e., multiple projection points Pmx within the target image region ARh, as detection target points Pmd. Since there are multiple projection points Pmx within the target image region ARh, there are also multiple detection target points Pmd.

[0041] In this way, the detection target point extraction unit 263 extracts multiple detection target points Pmd from among the multiple laser measurement points Pm, where the projected point Pmx obtained by projecting the laser measurement point Pm onto the camera image PC according to the above coordinate transformation conditions falls within the target image region ARh. These multiple detection target points Pmd extracted in step S104 are collectively called the detection target point group Gpmd. In other words, these multiple detection target points Pmd constitute the detection target point group Gpmd.

[0042] In step S105, following step S104 in Figure 5, the nearest point recognition unit 264 recognizes the nearest point position Xnr, which is the three-dimensional position of the detection target 82 that is closest to the mobile robot 10, based on the multiple detection target points Pmd obtained in step S104.

[0043] Specifically, the nearest point recognition unit 264 selects one of the multiple detection target points Pmd obtained in step S104 that is closest to the mobile robot 10 as the nearest detection point P1md. The nearest point recognition unit 264 then recognizes the three-dimensional position of the nearest detection point P1md as the nearest point position Xnr of the detection target 82.

[0044] For example, the nearest point recognition unit 264 recognizes the position and orientation of the trolley 12 and the operating status of the manipulator 14 based on information obtained from the robot control device 11. In other words, the nearest point recognition unit 264 sequentially recognizes the robot interference range, which is a three-dimensional range that changes over time and interferes with the mobile robot 10. Therefore, the nearest point recognition unit 264 can select the nearest detection point P1md as described above.

[0045] Following step S105 in Figure 5, the process proceeds to step S106. In step S106, the emergency stop determination unit 265 estimates whether at least a portion of the mobile robot 10 will reach the closest point position Xnr of the detection target 82 within a predetermined determination time T1r, based on the relative velocity Vr of each part of the mobile robot 10 with respect to the travel path 81. If the emergency stop determination unit 265 estimates that at least a portion of the mobile robot 10 will reach the closest point position Xnr within the determination time T1r, it sends an emergency stop signal Ssp (see Figure 2) to the robot control device 11 to stop the operation of the mobile robot 10. The determination time T1r is experimentally set in advance so that, for example, it is determined to send an emergency stop signal Ssp if the detection target 82 approaches the mobile robot 10 and there is a possibility of contact with any part of the mobile robot 10.

[0046] In short, in step S106, the emergency stop determination unit 265 performs an emergency stop determination to determine whether or not to send an emergency stop signal Ssp, and sends the emergency stop signal Ssp according to the determination result of the emergency stop determination.

[0047] Specifically, the emergency stop determination unit 265 performs emergency stop determination and sends an emergency stop signal Ssp according to the flowchart in Figure 8. First, in step SA01 of Figure 8, the emergency stop determination unit 265 calculates the relative speed Vr to the travel path 81, which is the relative speed Vr of each part of the mobile robot 10 with respect to the travel path 81. Examples of the parts for which the relative speed Vr to the travel path is calculated include the trolley 12, the tip 142 of the manipulator 14, the multiple joints 143 of the manipulator 14, and the multiple intermediate connecting parts 144 of the manipulator 14. In this description of the embodiment, the parts of the mobile robot 10 for which the relative speed Vr to the travel path is calculated may be referred to as speed calculation target parts.

[0048] Furthermore, the relative velocity Vr of each part of the mobile robot 10 relative to the travel path is expressed as a vector quantity that includes the direction of travel. For example, the relative velocity Vr of the trolley 12 relative to the travel path is the same as the travel speed of the trolley 12. Also, if the manipulator 14 is not operating, the relative velocity Vr of each part of the manipulator 14 relative to the travel path, such as the tip 142 of the manipulator 14, is also the same as the travel speed of the trolley 12. On the other hand, if the manipulator 14 is operating, the relative velocity Vr of each part of the manipulator 14 relative to the travel path is the combined velocity obtained by combining the relative velocity of that part with respect to the trolley 12 and the travel speed of the trolley 12. After step SA01 in Figure 8, proceed to step SA02.

[0049] In step SA02, the emergency stop determination unit 265 calculates the required reach distance DS, which is the distance from each of the multiple speed calculation target units of the mobile robot 10, whose relative speed Vr relative to the travel path has been calculated, to the closest point position Xnr of the detection target 82. This required reach distance DS is the distance along the direction of travel of the speed calculation target unit indicated by the vector quantity relative speed Vr relative to the travel path. After step SA02 in Figure 8, the process proceeds to step SA03.

[0050] In step SA03, the emergency stop determination unit 265 calculates the estimated required time Tr for each of the multiple speed calculation target units based on the relative speed Vr to the road calculated in step SA01 and the required distance DS calculated in step SA02. This estimated required time Tr is an estimate of the time required for the speed calculation target unit to reach the closest point position Xnr of the detection target 82, and since there are multiple speed calculation target units, multiple estimated required times Tr are calculated. For example, the estimated required time Tr is calculated for each speed calculation target unit by dividing the required distance DS by the relative speed Vr to the road. After step SA03 in Figure 8, the process proceeds to step SA04.

[0051] In step SA04, the emergency stop determination unit 265 determines for each of the multiple estimated required time Tr values ​​whether the estimated required time Tr is within a predetermined determination time T1r.

[0052] If, as a result of the determination, one or more of the multiple estimated required times Tr are determined to be within the determination time T1r, the process proceeds to step SA05. In step SA05, the emergency stop determination unit 265 sends an emergency stop signal Ssp to the robot control device 11, as shown in Figure 2. For example, if the robot control device 11 receives the emergency stop signal Ssp, it immediately stops the mobile robot 10. In that case, the robot control device 11 stops the trolley 12 and also stops the operation of the manipulator 14. When the emergency stop determination unit 265 sends the emergency stop signal Ssp in step SA05, the flowchart in Figure 8 ends, and step S106 in Figure 5 ends.

[0053] On the other hand, if the determination in step SA04 determines that all of the multiple estimated required times Tr have exceeded the determination time T1r, the flowchart in Figure 8 terminates. In other words, step S106 in Figure 5 terminates without sending the emergency stop signal Ssp. If step S106 in Figure 5 terminates, the control process in Figure 5 restarts from steps S101 and S102.

[0054] In the embodiment described above, the following effects and advantages are achieved.

[0055] According to this embodiment, as shown in Figures 5, 7, and 8, the closest point position Xnr of the detection target 82 is recognized based on a plurality of detection target points Pmd obtained from a plurality of laser measurement points Pm by image recognition of the camera image PC. If it is estimated that at least a portion of the mobile robot 10 will reach the closest point position Xnr within a predetermined determination time T1r, an emergency stop signal Ssp is sent to stop the operation of the mobile robot 10.

[0056] Therefore, regardless of whether the mobile robot 10 is moving or not, it is possible to detect when the object to be detected 82 approaches the mobile robot 10 beyond a predetermined limit. As a result, it is possible to monitor the safety of the mobile robot 10 even while it is moving.

[0057] In particular, in this embodiment, whether or not at least a part of the mobile robot 10 reaches the closest point position Xnr within the determination time T1r is estimated based on the relative velocity Vr of each part of the mobile robot 10 relative to the travel path. For example, among the relative velocity Vr of each part of the mobile robot 10 relative to the travel path, the relative velocity Vr of each part of the manipulator 14 relative to the travel path is a composite velocity obtained by combining the relative velocity of that part with respect to the trolley 12 and the travel speed of the trolley 12.

[0058] Therefore, compared to the case where the relative speed Vr to the travel path is uniformly set to the travel speed of the trolley 12, it is possible to accurately estimate whether at least a portion of the mobile robot 10 will reach the closest point position Xnr within the determination time T1r. This makes it possible to accurately determine the possibility of contact between the mobile robot 10 and the detection target 82, and to reduce the transmission of unnecessary emergency stop signals Ssp.

[0059] Furthermore, in this embodiment, it is possible to estimate whether at least a portion of the mobile robot 10 will reach the closest point position Xnr within the determination time T1r with high distance accuracy using multiple laser measurement points Pm measured by the laser sensor 24.

[0060] (1) Furthermore, according to this embodiment, as shown in Figures 1, 4 to 7, multiple detection target points Pmd are extracted from multiple laser measurement points Pm based on the target image region ARh in the camera image PC. Specifically, this means that multiple detection target points Pmd are extracted from among the multiple laser measurement points Pm, where the projected points Pmx obtained by projecting the laser measurement points Pm onto the camera image PC according to predetermined coordinate transformation conditions fall within the target image region ARh. Therefore, for example, multiple detection target points Pmd can be automatically extracted quickly by computer processing.

[0061] (2) In addition, according to this embodiment, as shown in Figure 3, in plan view, the trolley 12 has a rectangular peripheral portion 121. Also in plan view, the multiple laser sensors 24 are arranged at the corner portion 121a provided at one end of the diagonal of the rectangular peripheral portion 121 of the trolley 12, and at the corner portion 121b provided at the other end of the diagonal.

[0062] Therefore, it is possible to minimize the number of laser sensors 24 installed while forming a laser detection range RL that covers the entire circumference around the mobile robot 10 in a plan view, such that the laser detection range RL is not affected by changes in the posture of the manipulator 14 on the trolley 12.

[0063] Furthermore, according to this embodiment, the union of the shooting ranges Rcm of the multiple cameras 22 is formed to surround the mobile robot 10 around its entire circumference in a plan view, and extends to cover the entire monitoring range Rs, excluding blind spots near the cameras 22. Similarly, the union of the multiple laser detection ranges RL is also formed to surround the mobile robot 10 around its entire circumference in a plan view, and extends to cover the entire monitoring range Rs. Therefore, it is possible to monitor the entire three-dimensional space surrounding the mobile robot 10 in a 360° radius.

[0064] The laser detection range RL of a single laser sensor 24 generally extends approximately 360° horizontally and 50° vertically Dg relative to the laser sensor 24. Therefore, even if a typical sensor is used as the laser sensor 24 in Figure 3, by arranging the laser sensors 24 at the corners 121a and 121b of the trolley 12, as described above, the union of the laser detection ranges RL can be formed to cover the entire circumference of the mobile robot 10.

[0065] (Second Embodiment) Next, a second embodiment will be described. In this embodiment, the differences from the first embodiment described above will be mainly explained. Furthermore, parts that are the same as or equivalent to the above embodiment will be omitted or simplified in their description. The same applies to the descriptions of the embodiments described later.

[0066] When multiple laser measurement points Pm are measured by the laser sensor 24, due to factors such as the low spatial density of measurement by the laser sensor 24, the short measurement time, or the speed of movement of the detection target 82, some points representing parts of the detection target 82 (for example, protruding parts of the human body) may not be included in the detection target point cloud Gpmd. In such cases, the nearest detection point P1md selected from the detection target point cloud Gpmd may not indicate the actual position closest to the mobile robot 10. This is partly due to the limited performance of the laser sensor 24, given the limited mounting space for the laser sensor 24.

[0067] In this embodiment, to address the above, the monitoring and control device 26 calculates the nearest point position Xnr of the detection target 82 by appropriately correcting the three-dimensional position of the nearest detection point P1md.

[0068] Specifically, the monitoring and control device 26 of this embodiment executes the control process shown in Figure 9 instead of the control process shown in Figure 5 in the first embodiment. This control process shown in Figure 9 is executed periodically and repeatedly while the mobile robot 10 is operating, regardless of whether the mobile robot 10 is moving or stationary and performing work, similar to the control process shown in Figure 5 in the first embodiment. In Figure 9, steps with the same content as in Figure 5 are denoted by the same reference numerals as in Figure 5, and the explanation of the steps in Figure 9 that are denoted by the same reference numerals as in Figure 5 is omitted. This method of explanation is also used in the flowchart described later.

[0069] As shown in Figure 9, in step S205 following step S104, the nearest point recognition unit 264 selects one of the multiple detection target points Pmd obtained in step S104 that is closest to the mobile robot 10 as the nearest detection point P1md. The selection of this nearest detection point P1md is the same as in step S105 in Figure 5 of the first embodiment. After step S205 in Figure 9, the process proceeds to step S206.

[0070] In step S206, the nearest point recognition unit 264 calculates the straight-line distance between each of the multiple detection target points Pmd, excluding the nearest detection point P1md, and the nearest detection point P1md. Based on the calculation results of these straight-line distances, the nearest point recognition unit 264 selects multiple nearby detection points Pct located near the nearest detection point P1md from among the multiple detection target points Pmd, as shown in Figure 10. For example, multiple points located within a predetermined distance from the nearest detection point P1md are selected as nearby detection points Pct. This predetermined distance is experimentally set in advance to reduce the computational load when calculating the correction amount AM described later, while ensuring accurate calculation of the correction amount AM. The multiple nearby detection points Pct selected in step S206 and the nearest detection point P1md together constitute the nearby detection point group Gct. Following step S206 in Figure 9, the process proceeds to step S207.

[0071] In step S207, the nearest point recognition unit 264 calculates the feature quantity CHn of the nearby detected point group Gct. The reason for calculating this feature quantity CHn is to suppress the influence of differences in the density of multiple laser measurement points Pm depending on the position and distance of the detection target 82 to the laser sensor 24, and differences in the distribution of laser measurement points Pm depending on the orientation of the detection target 82.

[0072] The difference in the density of multiple laser measurement points Pm depending on the position and distance of the detection target 82, as described above, means, for example, that the further the detection target 82 is from the laser sensor 24, the sparser the density of its laser measurement points Pm becomes. Furthermore, the difference in the distribution of laser measurement points Pm depending on the orientation of the detection target 82, as described above, means, for example, that the amount of laser light hitting the human body differs depending on whether the human hand, which is the detection target 82, is facing sideways or forward, resulting in areas with many laser measurement points Pm and areas with few laser measurement points Pm.

[0073] The feature quantity CHn of the neighboring point group Gct is a variety of quantities that represent the characteristics of the neighboring point group Gct. For example, the feature quantity CHn of the neighboring point group Gct includes all or any of the following: the principal component directions of the neighboring point group Gct, the directions perpendicular to those principal component directions, the number of points included in the neighboring point group Gct, the variance, kurtosis, and skewness of the neighboring point group Gct in the principal component directions, and the variance, kurtosis, and skewness of the above directions perpendicular to the principal component directions. The above principal component directions and the above directions perpendicular to those principal component directions are each represented by three-dimensional vectors, while the above number of points, variance, kurtosis, and skewness are each represented by one-dimensional quantities. After step S207 in Figure 9, proceed to step S208.

[0074] In step S208, as shown in Figure 11, the nearest point recognition unit 264 calculates a correction amount AM for correcting the three-dimensional position of the nearest detected point P1md using a mathematical model 27 that has been pre-trained by supervised learning, based on the feature quantity CHn of the nearest detected point group Gct. This mathematical model 27 is configured to take the feature quantity CHn of the nearest detected point group Gct as input and output the correction amount AM.

[0075] The hatched area B in Figure 10 represents a portion of the human body assumed to be corrected by this correction amount AM. Furthermore, the correction amount AM calculated in step S208 is for correcting the three-dimensional position and therefore has components in the x-axis, y-axis, and z-axis directions.

[0076] For example, in this embodiment, the mathematical model 27 outputs a correction amount AM as the positional difference between the tip position of the protruding part and the three-dimensional position of the closest detection point P1md when the detection target 82 has a partially protruding part and the closest detection point P1md is located on the surface of the protruding part. Therefore, the correction amount AM output by the mathematical model 27 can be said to be for compensating for the absence of laser measurement points Pm on the protruding part of the detection target 82 based on the closest detection point P1md when the detection target 82 has the aforementioned protruding part. Examples of the aforementioned protruding part of the detection target 82 include a person's arm or leg.

[0077] Furthermore, it is desirable that the mathematical model 27 be a machine learning model in order to correspond to the characteristics of the laser sensor 24 and the physique of the person being detected 82. A machine learning model is a supervised learning model that has been pre-trained using multiple sets of true values ​​of inputs and outputs as training data, and is capable of estimating an output for an unknown input. Examples of mathematical models 27 in this embodiment include logistic regression models, random forest regression models, and hist gradient boosting regression models. After step S208 in Figure 9, the process proceeds to step S209.

[0078] In step S209, as shown in Figure 10, the nearest point recognition unit 264 first corrects the three-dimensional position of the nearest detection point P1md using a correction amount AM and calculates the corrected three-dimensional position as the detection point correction position Xam. This correction using correction amount AM corrects the three-dimensional position in the x-axis, y-axis, and z-axis directions. For example, in this correction, if the x-axis coordinate value of the nearest detection point P1md is x1, the x-axis component of the correction amount AM is Δx, and the x-axis coordinate value of the detection point correction position Xam is x2, then the x-axis coordinate value x2 of the detection point correction position Xam is given as "x2 = x1 + Δx". This is an excerpt from the explanation of the correction in the x-axis direction using correction amount AM, but the corrections in the y-axis and z-axis directions are similar.

[0079] Next, the nearest point recognition unit 264 compares the detection point correction position Xam calculated by the above correction with the three-dimensional position of the nearest detection point P1md. Then, the nearest point recognition unit 264 recognizes the position closer to the mobile robot 10 between the detection point correction position Xam and the three-dimensional position of the nearest detection point P1md as the nearest point position Xnr. After step S209 in Figure 9, the process proceeds to step S106.

[0080] (1) As described above, according to this embodiment, the nearest point recognition unit 264 calculates a correction amount AM from the feature quantity CHn of the neighboring detection point group Gct, which includes a plurality of neighboring detection points Pct located in the vicinity of the nearest detection point P1md, using a pre-learned mathematical model 27. The nearest point recognition unit 264 then recognizes the nearest point position Xnr as the position closer to the mobile robot 10 between the detection point correction position Xam obtained by correcting the three-dimensional position of the nearest detection point P1md with the correction amount AM and the three-dimensional position of the nearest detection point P1md.

[0081] Therefore, the overall shape of the detection target 82, which is not obtained from the information indicated by the multiple laser measurement points Pm measured by the laser sensor 24, can be grasped through correction, thereby improving safety compared to when no correction is performed. For example, as shown in Figures 12 and 13, the overall shape of the detection target 82 can be grasped through correction, even if the detection target 82 extends to the location indicated as "not detected" where there is no laser measurement point Pm.

[0082] Furthermore, in this embodiment, since the above-mentioned feature quantity CHn is used for correction, it is possible to perform highly accurate correction using the correction quantity AM while avoiding the effects caused by differences and variations in the density of the detected point cloud Gpmd due to, for example, the position and posture of the person being detected 82.

[0083] Except as described above, this embodiment is the same as the first embodiment. In this embodiment, the effects obtained from the configuration common to the first embodiment can be obtained in the same way as in the first embodiment.

[0084] (Third embodiment) Next, a third embodiment will be described. This embodiment will primarily describe the differences from the second embodiment described above.

[0085] In this embodiment, the monitoring and control device 26 executes the control process shown in Figure 14 instead of the control process shown in Figure 9 in the second embodiment. This control process shown in Figure 14 is executed periodically and repeatedly while the mobile robot 10 is operating, regardless of whether the mobile robot 10 is moving or stationary and performing work, for example.

[0086] As shown in Figure 14, in step S307 following step S104, the nearest point recognition unit 264 calculates the feature quantity CHmd of the target point cloud Gpmd. In step S307, the point cloud used as the basis for calculating the feature quantity CHmd is the target point cloud Gpmd, instead of the nearest neighbor detection point cloud Gct used in step S207 of Figure 9 in the second embodiment. Except for this, step S307 is the same as step S207 in Figure 9.

[0087] Therefore, the feature quantity CHmd of the point cloud Gpmd to be detected is a variety of quantities that represent the characteristics of the point cloud Gpmd. For example, the feature quantity CHmd of the point cloud Gpmd to be detected includes all or any of the following: the principal component direction of the point cloud Gpmd, the direction perpendicular to the principal component direction, the number of points included in the point cloud Gpmd, the variance, kurtosis, and skewness in the principal component direction of the point cloud Gpmd, and the variance, kurtosis, and skewness in the above direction perpendicular to the principal component direction. After step S307 in Figure 14, proceed to step S308.

[0088] In step S308, the nearest point recognition unit 264 calculates a correction amount AM using a mathematical model 27 from the feature quantity CHmd of the point cloud Gpmd to be detected, as shown in Figure 11. In this step S308, the input to the mathematical model 27 is the feature quantity CHmd of the point cloud Gpmd to be detected, rather than the feature quantity CHn of the neighboring point cloud Gct. Except for this, step S308 is the same as step S208 in Figure 9.

[0089] Following steps S104 and S308 in Figure 14, the process proceeds to step S309. In step S309, the nearest point recognition unit 264 first selects one of the multiple detection target points Pmd obtained in step S104 that is closest to the mobile robot 10 as the nearest detection point P1md. The selection of this nearest detection point P1md is the same as in step S105 in Figure 5 of the first embodiment.

[0090] Then, as shown in Figure 10, the nearest point recognition unit 264 calculates the corrected three-dimensional position of the nearest detection point P1md by correcting it with a correction amount AM, and uses this corrected three-dimensional position as the detection point correction position Xam. This correction using the correction amount AM is the same as in step S209 in Figure 9 of the second embodiment.

[0091] Next, the nearest point recognition unit 264 compares the detection point correction position Xam calculated by the above correction with the three-dimensional position of the nearest detection point P1md. Then, the nearest point recognition unit 264 recognizes the position closer to the mobile robot 10 between the detection point correction position Xam and the three-dimensional position of the nearest detection point P1md as the nearest point position Xnr. After step S309 in Figure 14, the process proceeds to step S106.

[0092] Except as described above, this embodiment is the same as the second embodiment. In this embodiment, the effects obtained from the configuration common to the second embodiment can be obtained in the same way as in the second embodiment.

[0093] (Fourth Embodiment) Next, a fourth embodiment will be described. This embodiment will primarily describe the differences from the first embodiment described above.

[0094] As shown in Figure 15, in this embodiment, the safety monitoring system 20 is equipped with one laser sensor 24. The laser sensor 24 is fixed to the trolley 12 and is positioned, for example, near the center of the trolley 12 in a plan view. As a result, the laser detection range RL of the laser sensor 24 is formed to extend around the entire circumference of the trolley 12 (i.e., 360° around the trolley 12) in a plan view. Thus, one laser detection range RL extends to cover the entire monitoring range Rs.

[0095] Furthermore, to prevent the laser sensor 24 from interfering with the operation of the manipulator 14, and to ensure that the laser beam from the laser sensor 24 is not blocked by the manipulator 14, the laser sensor 24 is positioned, for example, above the operating range of the manipulator 14.

[0096] (1) As described above, according to this embodiment, the laser sensor 24 is positioned such that, in a plan view, the laser detection range RL of one laser sensor 24 extends around the entire circumference of the trolley 12. Therefore, while keeping the number of laser sensors 24 installed to a minimum of one, it is possible to monitor the detection target 82 around the mobile robot 10 in a plan view.

[0097] Except as described above, this embodiment is the same as the first embodiment. In this embodiment, the effects obtained from the configuration common to the first embodiment can be obtained in the same way as in the first embodiment.

[0098] Although this embodiment is a modification based on the first embodiment, it is also possible to combine this embodiment with the second or third embodiment described above.

[0099] (Fifth embodiment) Next, a fifth embodiment will be described. This embodiment will primarily describe the differences from the first embodiment described above.

[0100] The image recognition method employed in step S101 of Figure 5, that is, the method for identifying the target image region ARh within the camera image PC, can identify not only the human body but also various other objects. In this embodiment, in step S101 of Figure 5, the image region occupied by the human body within the camera image PC is identified as the target image region ARh. This is the same as in the first embodiment, but in addition, in this embodiment, the image region occupied by the group consisting of a person in a wheelchair and the wheelchair within the camera image PC is also identified as the target image region ARh. Furthermore, the image region occupied by the group consisting of a person using a predetermined mobility aid and the predetermined aid within the camera image PC is also identified as the target image region ARh.

[0101] Therefore, in this embodiment, not only the human body but also the group consisting of a person in a wheelchair and that wheelchair, and the group consisting of a person using a predetermined mobility aid and that predetermined aid, each fall under the category of detection target 82. The predetermined mobility aid may be, for example, a cane, a walking aid, or a prosthesis. In other words, examples of a person using a predetermined aid include a person with a cane, a person using a walking aid, and a person wearing a prosthesis.

[0102] (1) As described above, according to this embodiment, not only the human body but also the group consisting of a person riding in a wheelchair and that wheelchair, and the group consisting of a person using a predetermined device to assist with mobility and that predetermined device, each fall under the category of detection target 82. Therefore, it is possible to perform safety monitoring suitable for the intended use of the mobile robot 10, such as for care services.

[0103] Except as described above, this embodiment is the same as the first embodiment. In this embodiment, the effects obtained from the configuration common to the first embodiment can be obtained in the same way as in the first embodiment.

[0104] Although this embodiment is a modification based on the first embodiment, it is also possible to combine this embodiment with any of the second to fourth embodiments described above.

[0105] (Other embodiments) (1) In the first embodiment described above, as shown in Figure 3, the entire union of the camera 22's shooting range Rcm is contained within the union of the laser detection range RL, but this is just one example. As long as each of the multiple laser detection ranges RL overlaps at least partially with the shooting range Rcm of any of the cameras 22, for example, it is acceptable even if the union of the camera 22's shooting range Rcm and the union of the laser detection range RL only partially overlap each other.

[0106] (2) In each of the embodiments described above, the monitoring control device 26 and the robot control device 11 shown in Figure 2 are mounted on, for example, the mobile robot 10, but this is just one example. The monitoring control device 26 and the robot control device 11 may be located separately from the mobile robot 10.

[0107] (3) In the first embodiment described above, the camera 22 provided on the mobile robot 10 in Figure 1 cannot measure the temperature of the object being photographed, but it may be replaced with a thermal imaging camera capable of measuring the temperature of the object being photographed as a thermal image. When such a thermal imaging camera is used, it can sense the temperature of the person being detected 82, so it is possible to identify the area in the camera image PC shown in Figure 4 that is as hot as the body temperature of a person as the target image area ARh. This image recognition method using a thermal imaging camera can also be considered a type of segmentation described above.

[0108] (4) In the first embodiment described above, as shown in Figure 8, the movement speed of the nearest detection point P1md is not taken into account in the calculation of the estimated required time Tr in step SA03, but it may be taken into account in the calculation of the estimated required time Tr. If this is done, the estimated required time Tr can be calculated with higher accuracy. For example, the movement speed of the nearest detection point P1md can be determined from the difference in the three-dimensional position of the nearest detection point P1md at multiple time points.

[0109] (5) In the second embodiment described above, as shown in Figure 10, the neighbor detection point group Gct includes a nearest detection point P1md in addition to a plurality of neighbor detection points Pct, but this is just one example. For example, the neighbor detection point group Gct may consist of a plurality of neighbor detection points Pct without including the nearest detection point P1md.

[0110] (6) In the second embodiment described above, in the flowchart of Figure 9, after the calculation of the correction amount AM, the detection point correction position Xam is always calculated by correcting the three-dimensional position of the closest detection point P1md by the correction amount AM, but this is just one example. For example, if the correction amount AM corrects the three-dimensional position of the closest detection point P1md to move it away from the mobile robot 10, the detection point correction position Xam does not need to be calculated. In that case, the three-dimensional position of the closest detection point P1md is recognized as the closest point position Xnr. The same applies to the flowchart of Figure 14 in the third embodiment.

[0111] (7) In each of the embodiments described above, as shown in Figure 2, the robot control device 11 and the monitoring control device 26 are configured as separate control devices, but the robot control device 11 and the monitoring control device 26 may be configured as a single control device.

[0112] (8) In each of the embodiments described above, the processing of each step shown in the flowcharts of Figures 5, 8, 9, and 14 is implemented by a computer program, but it may also be implemented by hardware.

[0113] (9) In the first embodiment described above, in step SA03 of Figure 8, for example, the estimated required time Tr is calculated for each speed calculation target unit by dividing the required distance DS by the relative speed Vr to the travel path, but the estimated required time Tr can also be calculated by other methods. For example, in the mobile robot 10 of the first embodiment, the trolley 12 and the manipulator 14 are usually set to repeatedly perform predetermined set operations. In such cases, the trajectories of each part of the trolley 12 and the manipulator 14 can be predicted from the above set operations, so the estimated required time Tr can be calculated based on the above set operations.

[0114] (10) The present disclosure is not limited to the embodiments described above and can be implemented in various modified forms. Furthermore, the embodiments described above are not unrelated to each other and can be combined as appropriate, except in cases where the combination is clearly impossible.

[0115] Furthermore, it goes without saying that, in each of the above embodiments, the elements constituting the embodiment are not necessarily essential unless explicitly stated to be particularly essential or unless they are clearly considered essential in principle. Also, in each of the above embodiments, when numerical values ​​such as the number, numerical values, quantities, or ranges of the components of the embodiment are mentioned, the embodiment is not limited to those specific numbers unless explicitly stated to be particularly essential or unless it is clearly limited to a specific number in principle. Also, in each of the above embodiments, when the material, shape, positional relationship, etc. of the components are mentioned, the embodiment is not limited to those material, shape, positional relationship, etc. unless explicitly stated or unless it is clearly limited to a specific material, shape, positional relationship, etc. in principle. [Explanation of Symbols]

[0116] 10 Mobile Robots 20. Safety Monitoring System 22 cameras 24 Laser Sensors 262 Image Recognition Unit 263 Detection target point extraction unit 264 Closest point recognition unit 265 Emergency stop judgment section Rcm shooting range RL Laser Detection Range

Claims

1. A safety monitoring system for a mobile robot (10) having a trolley (12) that travels along a travel path (81) and a manipulator (14) attached to the trolley, The mobile robot is equipped with a camera (22) that photographs the area around the mobile robot, A laser sensor (24) is provided on the mobile robot and detects objects located within a laser detection range (RL) that at least partially overlaps with the shooting range (Rcm) captured by the camera and extends around the mobile robot. The camera image (PC) captured by the aforementioned camera includes an image recognition unit (262) that identifies the target image region (ARh) occupied by a detection target (82) including a person, A detection target point extraction unit (263) extracts a plurality of detection target points (Pmd) that indicate the three-dimensional position on the surface of the object to be detected from a plurality of laser measurement points (Pm) measured by the laser sensor, based on the target image region, A nearest point recognition unit (264) recognizes the nearest point position (Xnr), which is the three-dimensional position closest to the mobile robot among the detected targets, based on the plurality of detected target points, A safety monitoring system comprising: an emergency stop determination unit (265) that estimates whether at least a portion of the mobile robot will reach the closest point position within a predetermined determination time (T1r) based on the relative speed (Vr) of each part (12, 142, 143, 144) of the mobile robot with respect to the travel path, and if it is estimated that at least a portion of the mobile robot will reach the closest point position within the determination time, an emergency stop determination unit (265) that sends an emergency stop signal (Ssp) to stop the operation of the mobile robot.

2. The safety monitoring system according to claim 1, wherein the extraction of the plurality of detection target points from the plurality of laser measurement points based on the target image region means that the plurality of detection target points are selected from the plurality of laser measurement points in which the projection points (Pmx) obtained by projecting the laser measurement points onto the camera image according to a predetermined coordinate transformation condition (F1) fall within the target image region.

3. The nearest point recognition unit, From the plurality of detection target points, the closest detection point (P1md) that is closest to the mobile robot is selected. From the feature quantities (CHn) of the neighboring detection point group (Gct), which consists of multiple neighboring detection points (Pct) located within a predetermined distance from the nearest detection point among the multiple detection target points, or from the feature quantities (CHmd) of the detection target point group (Gpmd), which consists of the multiple detection target points, a correction amount (AM) for correcting the three-dimensional position of the nearest detection point is calculated using a mathematical model (27) that has been pre-trained by supervised learning. The safety monitoring system according to claim 1 or 2, wherein the system recognizes the position closer to the mobile robot between the three-dimensional position of the closest proximity detection point and the corrected three-dimensional position (Xam) obtained by correcting the three-dimensional position of the closest proximity detection point by the correction amount, the latter of which is recognized as the closest proximity point position.

4. The safety monitoring system according to claim 1 or 2, wherein an assembly consisting of a person riding in a wheelchair and the wheelchair, and an assembly consisting of a person using a predetermined device for assisting mobility and the predetermined device, each correspond to the detection target.

5. Multiple laser sensors are provided, The safety monitoring system according to claim 1 or 2, wherein, in a view along the vertical direction (Dg), the trolley has a rectangular peripheral edge (121), and the plurality of laser sensors are arranged at a corner (121a) provided at one end of the diagonal of the rectangular peripheral edge of the trolley and at a corner (121b) provided at the other end of the diagonal.

6. The safety monitoring system according to claim 1 or 2, wherein the laser sensors are arranged such that, when viewed in a direction along the vertical direction (Dg), the laser detection range of one of the laser sensors extends around the entire circumference of the trolley.

Images