Control system, control method, and program

Abstract

To provide a control system capable of improving the safety around the mobile robot which is capable of autonomous movement and user operation by preventing the safety to its surroundings from varying depending on the operator.SOLUTION: A disclosed control system executes a system control for controlling a system including a mobile robot 100 that is capable of autonomous movement and can be operated by a user depending on the characteristics of the mobile object around an interface for controlling the mobile robot 100. The system control includes changing operation restrictions for the operation interface when the feature satisfies the predetermined condition.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to a control system, a control method, and a program. [Background technology]

[0002] Patent Document 1 discloses a robot control system. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Special Publication No. 2021-508902 Summary of the Invention [Problem to be solved by the invention]

[0004] However, if a mobile robot can be controlled by an operator, it becomes difficult to ensure safety because the safety of the mobile robot's surroundings varies depending on the operator. However, this problem cannot be solved by the technology described in Patent Document 1.

[0005] The present disclosure has been made to solve such problems, and provides a control system, control method, and program that can prevent the safety of a mobile robot that can move autonomously and be operated by a user from varying depending on the operator, thereby improving safety. [Means for solving the problem]

[0006] A control system according to the present disclosure executes system control for controlling a system including a mobile robot in accordance with characteristics of moving objects present in the vicinity of an operation interface for operating an autonomously mobile and user-operable mobile robot, the system control including modification control for modifying operation restrictions on the operation interface when the characteristics satisfy a predetermined condition. This configuration of the control system prevents the safety of the mobile robot's surroundings from varying depending on the operator, thereby improving safety. The autonomous movement control can also use a learning model obtained by machine learning to cause the mobile robot to move autonomously.

[0007] The predetermined condition is that the characteristic is one of a characteristic indicating a non-staff member of a facility that operates the mobile robot and a characteristic indicating a child, and the change control is control to impose the operation restriction on the operation interface when the characteristic satisfies the predetermined condition. With this configuration, when an operator who would lower safety if allowed to operate the operation interface is present near the operation interface, the control system imposes an operation restriction more severe than when an operator who can maintain safety even if allowed to operate the operation interface is present near the operation interface, thereby preventing safety from varying depending on the operator.

[0008] The predetermined condition is that the feature is a feature indicative of a staff member of a facility that operates the mobile robot, and the change control is control to relax the operation restriction on the operation interface when the feature satisfies the predetermined condition. With this configuration, the control system imposes an operation restriction when an operator who would lower safety if allowed to operate the operation interface is present near the operation interface, and relaxes the operation restriction when an operator who would maintain safety even if allowed to operate the operation interface is present near the operation interface, thereby preventing safety from varying depending on the operator.

[0009] The operation interface includes at least one of an interface displayed on a display device installed on the mobile robot, a joystick device installed on the mobile robot, and an operation device for remotely controlling the mobile robot. With this configuration, the control system can prevent the safety of the operation interface installed on the mobile robot from varying depending on the operator.

[0010] The system control includes control for stopping the mobile robot when the predetermined condition is satisfied. With this configuration, the control system not only controls to change the operation limits but also controls to stop the mobile robot when the predetermined condition is satisfied, thereby further improving safety.

[0011] The control system recognizes the feature based on an image obtained by capturing an image of the periphery of the operation interface. With this configuration, the control system can accurately recognize the feature.

[0012] The control system determines whether the feature satisfies the predetermined condition by using a learning model that inputs an image obtained by capturing an image of the periphery of the operation interface and outputs whether the image satisfies the predetermined condition. With this configuration, the control system can accurately determine whether the predetermined condition is satisfied.

[0013] The operation restriction is a restriction that prohibits some or all of the operations on the operation interface. With this configuration, the control system can prohibit operations on the operation interface when a predetermined condition is met, thereby further preventing safety from varying depending on the operator.

[0014] The operation restriction is a restriction that makes it difficult to perform some or all of the operations in the operation interface. With this configuration, the control system can make it difficult to perform operations in the operation interface when a predetermined condition is met, thereby further preventing safety from varying depending on the operator.

[0015] The system control includes at least one of control for changing at least one of the content and method of notification by the operation interface when the feature satisfies the predetermined condition, and control for displaying an image of the moving object on the operation interface when the feature satisfies the predetermined condition. With this configuration, the control system can perform at least one of notification by the operation interface and displaying an image of the moving object when the predetermined condition is satisfied. Therefore, in the case where an operator who would lower safety if allowed to operate the operation interface is present near the operation interface, the control system can suppress the operator's intention to operate the interface, thereby further preventing safety from varying depending on the operator.

[0016] A control method according to the present disclosure executes system control for controlling a system including a mobile robot in accordance with characteristics of moving objects present in the vicinity of an operation interface for operating an autonomously mobile robot that is user-operable, the system control including modification control for modifying operation restrictions on the operation interface when the characteristics satisfy a predetermined condition. With this configuration, the control method can prevent the safety of the mobile robot from varying depending on the operator, thereby improving safety.

[0017] The predetermined condition is that the characteristic is one of a characteristic indicating a non-staff member of a facility that operates the mobile robot and a characteristic indicating a child, and the change control is control to impose the operation restriction on the operation interface when the characteristic satisfies the predetermined condition. With this configuration, when an operator who would lower safety if allowed to operate the operation interface is present near the operation interface, the control method imposes an operation restriction more severe than when an operator who can maintain safety even if allowed to operate the operation interface is present near the operation interface, thereby preventing safety from varying depending on the operator.

[0018] The predetermined condition is that the feature is a feature indicative of a staff member of a facility that operates the mobile robot, and the change control is control to relax the operation restriction on the operation interface when the feature satisfies the predetermined condition. With this configuration, the control method imposes an operation restriction when an operator who would lower safety if allowed to operate the operation interface is present near the operation interface, and relaxes the operation restriction when an operator who can maintain safety even if allowed to operate the operation interface is present near the operation interface, thereby preventing safety from varying depending on the operator.

[0019] The operation interface includes at least one of an interface displayed on a display device installed on the mobile robot, a joystick device installed on the mobile robot, and an operation device for remotely controlling the mobile robot. With this configuration, the control method can prevent the safety of the operation interface installed on the mobile robot from varying depending on the operator.

[0020] The system control includes control for stopping the mobile robot when the predetermined condition is satisfied. With this configuration, the control method not only controls to change the operation limit but also controls to stop the mobile robot when the predetermined condition is satisfied, thereby further improving safety.

[0021] The control method includes: detecting a position of the user's hand on the operation interface; detecting a position of the user's hand on the operation interface;

[0022] The determination of whether the feature satisfies the predetermined condition is performed using a learning model that inputs an image obtained by capturing an image of the periphery of the operation interface and outputs whether the image satisfies the predetermined condition. With this configuration, the control method can accurately determine whether the predetermined condition is satisfied.

[0023] The operation restriction is a restriction that prohibits some or all of the operations on the operation interface. With this configuration, the control method can prohibit operations on the operation interface when a predetermined condition is satisfied, thereby further preventing safety from varying depending on the operator.

[0024] The operation restriction is a restriction that makes it difficult to perform some or all of the operations in the operation interface. With this configuration, the control method can make it difficult to perform operations in the operation interface when a predetermined condition is satisfied, thereby further preventing safety from varying depending on the operator.

[0025] The system control includes at least one of control for changing at least one of the content and method of notification by the operation interface when the feature satisfies the predetermined condition, and control for displaying an image of the moving object on the operation interface when the feature satisfies the predetermined condition. With this configuration, the control method can perform at least one of notification by the operation interface and displaying an image of the moving object when the predetermined condition is satisfied. Therefore, with the control method, when an operator who would lower safety if allowed to operate the operation interface is present near the operation interface, the operator can be prevented from trying to operate it, thereby further preventing safety from varying depending on the operator.

[0026] A program according to the present disclosure causes a computer to execute a process for executing system control for controlling a system including an autonomously movable and user-operable mobile robot in accordance with characteristics of moving objects present in the vicinity of an operation interface for operating the mobile robot, the system control including change control for changing operation restrictions on the operation interface when the characteristics satisfy a predetermined condition. With this configuration, the program can prevent the safety of the mobile robot from varying depending on the operator, thereby improving safety.

[0027] The predetermined condition is that the characteristic is one of a characteristic indicating a non-staff member of a facility that operates the mobile robot and a characteristic indicating a child, and the change control is control to impose the operation restriction on the operation interface when the characteristic satisfies the predetermined condition. With this configuration, when an operator who would lower safety if allowed to operate the operation interface is present near the operation interface, the program imposes an operation restriction more severe than when an operator who can maintain safety even if allowed to operate the operation interface is present near the operation interface, thereby preventing safety from varying depending on the operator.

[0028] The predetermined condition is that the feature is a feature indicative of a staff member of a facility that operates the mobile robot, and the change control is control to relax the operation restriction on the operation interface when the feature satisfies the predetermined condition. With this configuration, the program imposes an operation restriction when an operator who would lower safety if allowed to operate the operation interface is present near the operation interface, and relaxes the operation restriction when an operator who can maintain safety even if allowed to operate the operation interface is present near the operation interface, thereby preventing safety from varying depending on the operator.

[0029] The operation interface includes at least one of an interface displayed on a display device installed on the mobile robot, a joystick device installed on the mobile robot, and an operation device for remotely controlling the mobile robot. With this configuration, the program can prevent the safety of the operation interface installed on the mobile robot from varying depending on the operator.

[0030] The system control includes control for stopping the mobile robot when the predetermined condition is satisfied. With this configuration, the program not only controls the change of the operation limit but also controls the stop of the mobile robot when the predetermined condition is satisfied, thereby further improving safety.

[0031] The feature is recognized based on an image obtained by capturing an image of the periphery of the operation interface. With this configuration, the program can accurately recognize the feature.

[0032] The determination of whether the feature satisfies the predetermined condition is performed using a learning model that inputs an image obtained by capturing an image of the periphery of the operation interface and outputs whether the image satisfies the predetermined condition. With this configuration, the program can accurately determine whether the predetermined condition is satisfied.

[0033] The operation restriction is a restriction that prohibits some or all of the operations in the operation interface. With this configuration, the program can prohibit operations in the operation interface when a predetermined condition is met, thereby further preventing safety from varying depending on the operator.

[0034] The operation restriction is a restriction that makes it difficult to perform some or all of the operations in the operation interface. With this configuration, the program can make it difficult to perform operations in the operation interface when a predetermined condition is met, thereby further preventing safety from varying depending on the operator.

[0035] The system control includes at least one of control for changing at least one of the content and method of notification by the operation interface when the feature satisfies the predetermined condition, and control for displaying an image of the moving object on the operation interface when the feature satisfies the predetermined condition. With this configuration, the program can perform at least one of notification by the operation interface and displaying an image of the moving object when the predetermined condition is satisfied. Therefore, when an operator who would lower safety if allowed to operate the operation interface is present near the operation interface, the program can suppress the operator's intention to operate the interface, thereby further preventing safety from varying depending on the operator. [Effects of the Invention]

[0036] According to the present disclosure, it is possible to provide a control system, a control method, and a program for a mobile robot that is capable of autonomous movement and can be operated by a user, which can prevent the safety of the mobile robot to its surroundings from varying depending on the operator, thereby improving safety. [Brief explanation of the drawings]

[0037] [Figure 1] 1 is a perspective view showing an example of the overall configuration of a mobile robot that is an object to be controlled by a control system according to an embodiment; [Figure 2] FIG. 2 is a perspective view showing an example of the overall configuration of a wagon transported by the mobile robot of FIG. 1. [Figure 3] 3 is a perspective view showing the mobile robot of FIG. 1 transporting the wagon of FIG. 2. FIG. [Figure 4] 2 is a block diagram showing an example of the configuration of a control computer in the mobile robot of FIG. 1. FIG. [Figure 5] FIG. 5 is a diagram showing an example of classification of non-staff and staff determined by the control computer of FIG. 4. [Figure 6] 5 is a diagram showing an example of a GUI image displayed on an operation unit under the control of the control computer of FIG. 4. FIG. [Figure 7] 7 is a diagram showing an example of a GUI image when operation restriction is applied to the GUI image of FIG. 6. FIG. [Figure 8] 7 is a diagram showing another example of a GUI image when operation restriction is applied to the GUI image of FIG. 6. FIG. [Figure 9] 5 is a flowchart illustrating an example of an operation restriction process executed by the control computer of FIG. 4. FIG. [Figure 10] 7 is a flowchart for explaining another example of the operation restriction process executed by the control computer of FIG. 4. [Figure 11] 1 is a schematic diagram illustrating an example of the overall configuration of a system including a mobile robot according to an embodiment; [Figure 12] FIG. 2 illustrates an example of a hardware configuration of the apparatus. DETAILED DESCRIPTION OF THE INVENTION

[0038] The present invention will be described below through embodiments of the invention, but the invention according to the claims is not limited to the following embodiments. Furthermore, not all of the configurations described in the embodiments are necessarily essential as means for solving the problems.

[0039] (Embodiment) The control system according to this embodiment executes system control for controlling a system including a mobile robot that is capable of autonomous movement and can be operated by a user. This mobile robot can be configured to transport an object, and such an example will be described below, but it may also be configured in a way that does not allow it to transport an object. If the mobile robot is configured to transport an object, it can also be called a transport robot, and the above system can be called a transport system. An example of the configuration of the mobile robot according to this embodiment will be described below using FIGS. 1 and 2. FIG. 1 is a perspective view showing an example of the overall configuration of a mobile robot that is the object of control by the control system according to this embodiment, and FIG. 2 is a perspective view showing an example of the overall configuration of a wagon transported by the mobile robot of FIG. 1.

[0040] The above-mentioned systems, such as the above-mentioned transport system, may include a mobile robot such as the mobile robot 100 shown in Fig. 1, but may also include other devices such as a host management device. However, for simplicity of explanation, an example in which the transport system is configured using the mobile robot 100 alone will first be given, and its main features will be explained. In this example, the control system can refer to the mobile robot 100 itself or the components of the control system provided in the mobile robot 100.

[0041] In the following explanation, an XYZ Cartesian coordinate system will be used where appropriate. The X direction is the front-to-back direction of the mobile robot 100 shown in FIG. 1, the Y direction is the left-to-right direction, and the Z direction is the vertical up-down direction. More specifically, the +X direction is defined as the front direction of the mobile robot 100, and the -X direction is defined as the rear direction of the mobile robot 100. The +Y direction is the left direction of the mobile robot 100, and the +Z direction is the vertical up direction.

[0042] The mobile robot 100 can move both forward and backward. That is, when the wheels are rotated forward, the mobile robot 100 moves forward, and when the wheels are rotated backward, the mobile robot 100 moves backward. By changing the rotation speed of the left and right wheels, the mobile robot 100 can turn left and right.

[0043] 1, the mobile robot 100 may include a chassis 110 on which an object to be transported is mounted, a stand 120, and an operation unit 130. The chassis 110 is equipped with wheels 111, axles, a battery, a control computer 101, a drive motor, and the like. Note that the description will be given assuming that the control computer 101 is mounted in the position shown in the figure on the chassis 110, but this is not limiting, and the control computer 101 may be mounted in another position on the chassis 110, or part or all of the control computer 101 may be mounted on at least one of the stand 120 and the operation unit 130.

[0044] The chassis 110 holds wheels 111 in a rotatable manner. In the example of FIG. 1, the chassis 110 is provided with four wheels 111. The four wheels 111 are left and right front wheels and left and right rear wheels. The direction and speed of rotation of the wheels 111 are controlled independently, allowing the mobile robot 100 to move along a desired route. Some of the four wheels 111 may be drive wheels, and the rest may be driven wheels. Furthermore, as shown in FIG. 1, additional driven wheels may be provided between the front and rear wheels 111, for example.

[0045] Furthermore, at least one of the chassis 110, the operating unit 130, and the stand 120 may be provided with various sensors such as a camera, a distance sensor, etc., for example, to prevent contact with obstacles or to confirm the route.

[0046] FIG. 1 shows an example in which the sensor includes a camera 104 facing the +X side on a stand 120 and a sensor 105 provided at the front of the chassis 110. A bumper is provided at the front of the chassis 110, and the sensor 105 can be disposed on the bumper to detect contact of an object with the bumper. When the sensor 105 detects contact of an object, i.e., an obstacle, with the mobile robot 100, the mobile robot 100 can be controlled to stop the mobile robot 100. Therefore, the sensor 105 can be referred to as a stop sensor. However, the sensor 105 is not limited to being located at the front, and can also be a sensor that detects contact of an object with a bumper provided on part or all of the outer periphery of the mobile robot 100. The sensor 105 can also be configured to detect the position at which the object contacts the bumper.

[0047] The mobile robot 100 is an autonomous mobile robot, but it has the ability to move by user operation, i.e., it is a mobile robot that can switch between an autonomous movement mode and a user-operated mode. By controlling the autonomous movement, the mobile robot 100 can move autonomously based on a route determined according to a set destination or a set route. In controlling the autonomous movement, the mobile robot 100 can also move autonomously by using a learning model obtained by machine learning to determine a route and avoid collisions.

[0048] Here, the user operation mode, in which movement is based on user operation, may be a mode in which the degree of user operation is relatively high compared to the autonomous movement mode in which movement is autonomous. In other words, the user operation mode does not need to be limited to a mode in which the user operates all of the movements of the mobile robot and autonomous control by the mobile robot is completely eliminated, and similarly, the autonomous movement mode does not need to be limited to a mode in which the mobile robot is completely autonomously controlled and does not accept any user operation. For example, the user operation mode and the autonomous movement mode may include the following first to third examples.

[0049] In a first example, in the autonomous movement mode, the mobile robot moves autonomously and makes the decision to stop and start moving, without any user operation; in the user operation mode, the mobile robot moves autonomously and the user performs the stop and start operations. In a second example, in the autonomous movement mode, the mobile robot moves autonomously and the user performs the stop and start operations; in the user operation mode, the mobile robot does not move autonomously and the user performs the stop and start operations as well as the movement operations. In a third example, in the autonomous movement mode, the mobile robot moves autonomously and makes the decision to stop and start moving, without any user operation; in the user operation mode, the mobile robot moves autonomously, adjusting speed and avoiding collisions, while the user performs the operation to change the direction of movement and route.

[0050] The user may also be an employee of the facility where the mobile robot 100 is operated, and if the facility is a hospital, the user may be a hospital employee.

[0051] The control computer 101 can be realized, for example, by an integrated circuit, and can be realized, for example, by a processor such as an MPU (Micro Processor Unit) or a CPU (Central Processing Unit), a working memory, and a non-volatile storage device. A control program to be executed by the processor is stored in this storage device, and the processor reads the program into the working memory and executes it, thereby fulfilling the function of controlling the mobile robot 100. The control computer 101 can be called a control unit.

[0052] The control computer 101 controls the autonomous movement of the mobile robot 100 to move toward a predetermined destination or along a predetermined route, based on pre-stored map data and information acquired by various sensors such as the camera 104. This autonomous movement control may include control of loading and unloading the wagon 500 shown in FIG. 2. The wagon 500 will be described later. The control computer 101 may be equipped with a movement control unit that performs this autonomous movement control.

[0053] In order to load and unload transported objects such as the wagon 500, the chassis 110 may be equipped with an elevator mechanism 140 for loading and unloading the transported objects. A portion of the elevator mechanism 140 may be housed inside the chassis 110, or the elevator mechanism 140 may be disposed on the upper surface of the chassis 110 with a loading surface on which the transported objects are placed exposed. The elevator mechanism 140 is an elevator stage that can be raised and lowered, and can be raised and lowered under control of the control computer 101. The chassis 110 is equipped with an elevator motor and a guide mechanism. The upper surface of the elevator mechanism 140 serves as a loading surface on which the wagon 500 as the transported object is placed. The wagon 500 is not limited to the configuration shown in FIG. 2, and may be any wagon of a predetermined size, shape, and weight that can be placed on the elevator mechanism 140 for transport. The elevator mechanism 140 has a lift mechanism that lifts the wagon 500. The space above the elevator mechanism 140 serves as a loading space on which the transported object is placed. It should be noted that the chassis 110 does not need to be provided with the lifting mechanism 140 if the user is limited to loading the wagon 500 .

[0054] The chassis 110 may also include a first light-emitting unit 11 at a position surrounding the lifting mechanism 140. The first light-emitting unit 11 may be configured to emit light for notification purposes, and may be configured, for example, with one or more LEDs (Light-Emitting Diodes), organic electroluminescence, or the like, and its light emission may be controlled by the control computer 101. The position, shape, and size of the first light-emitting unit 11 are not limited to those shown in the figure. The mobile robot 100 may also include the first light-emitting unit 11 even when the lifting mechanism 140 is not included. The prefixes "first" and "second" are added to the first light-emitting unit 11 and the second light-emitting unit 12 described below merely to distinguish them. The mobile robot 100 may not include either or both of the first light-emitting unit 11 and the second light-emitting unit 12.

[0055] The stand 120 is attached to the chassis 110. The stand 120 is a rod-shaped member extending upward from the chassis 110. Here, the stand 120 is formed in a cylindrical shape with the longitudinal direction in the Z direction, but of course, the shape is not important, and the mobile robot 100 may be configured without the stand 120. The longitudinal direction of the stand 120 is arranged parallel to the Z direction. The stand 120 is arranged outside the lifting mechanism 140. In other words, the stand 120 is arranged so as not to interfere with the lifting operation of the lifting mechanism 140. The stand 120 is arranged at one end of the chassis 110 in the Y direction (left-right direction). The stand 120 is attached near the front right corner of the chassis 110. In the XY plane, the stand 120 is provided at the end of the chassis 110 on the +X side and the -Y side.

[0056] The stand 120 may also be provided, on its upper surface, with a stick portion 131 of a joystick device or an emergency stop button for emergency stopping the mobile robot 100. The joystick device is a device for operating the mobile robot 100 in a user-intended direction in user operation mode. The stick portion 131 can be called a grip portion because it can be held by the user's hand. The user can accept directional control by tilting the stick portion 131 in the desired direction. The joystick device can also be controlled so that a switching operation, which switches between the autonomous movement mode and the user operation mode, is performed by pressing the stick portion 131 downward. Alternatively, the joystick device can be controlled so that a confirmation operation is performed by pressing the stick portion 131 downward. The stick portion 131 can also be configured to function as an emergency stop button when pressed downward for a predetermined period of time. If the joystick device is configured to accept multiple operations among the switching operation, confirmation operation, and emergency stop operation, the predetermined period can be set to be different for each operation. The shape and size of the joystick device described above are not limited to those exemplified.

[0057] The stand 120 may also include a second light-emitting unit 12 at a position surrounding the stick unit 131. The second light-emitting unit 12 may be configured to emit light for notification purposes, and may be configured with, for example, one or more LEDs, organic electroluminescence, or the like, and the light emission may be controlled by the control computer 101. The position, shape, and size of the second light-emitting unit 12 are not limited to those shown in the drawings. Note that the mobile robot 100 may also be provided with the second light-emitting unit 12 even when the stand 120 is not provided, or when the mobile robot 100 is provided with the stand 120 but not the stick unit 131.

[0058] The stand 120 supports the operation unit 130. The operation unit 130 is attached near the upper end of the stand 120. This allows the operation unit 130 to be installed at a height that is easy for the user to operate. In other words, the stand 120 extends to a height that is easy for the user to operate while standing, and the stick unit 131 is also disposed at a height that is easy for the user to operate. The operation unit 130 extends from the stand 120 to the +Y side. From the perspective of ease of operation, the operation unit 130 can be disposed in the center of the chassis 110 in the left-right direction.

[0059] The operation unit 130 may include a touch panel monitor or the like that accepts user operations. In this case, the operation unit 130 can display a user interface image, such as a graphical user interface (GUI) image, on a screen and accept touch operations on the touch panel as user operations. Of course, the operation unit 130 can also include a microphone or the like for voice input. The monitor of the operation unit 130 faces away from the chassis 110. In other words, the display surface (operation surface) of the operation unit 130 is the +X side. The operation unit 130 may be detachable from the stand 120. In other words, the stand 120 may be equipped with a holder that holds the touch panel. By operating the operation unit 130, the user can input or change information such as the destination of the transported item and transport information related to the transported item. Furthermore, the operation unit 130 can be configured to display information such as the contents of the transported item, the destination, and the transport route of the transported item, and to accept changes to the destination and the transport route. The operation unit 130 can also be configured to accept movement operations for the mobile robot 100 in the same manner as a joystick device. Of course, the mobile robot 100 may not have the operation unit 130, but even in that case, it is configured to be operable by a user. For example, the mobile robot 100 can be equipped with an operation device that accepts user operations, as exemplified by a joystick device equipped with a stick unit 131. The mobile robot 100 can also be connected to an operation device that performs remote control.

[0060] Furthermore, as shown in the figure, the operation unit 130 and the stick unit 131 can be disposed at least at the same height so that the operation can be performed intuitively. This allows the user to perform operations in an intuitive flow, even when a pressing operation on the stick unit 131 is assigned to an operation for deciding on the operation content displayed on the operation unit 130.

[0061] Furthermore, an IC card reader may be provided on the stand 120 at a position approximately at the same height as the operation unit 130 or inside the operation unit 130 to allow the user to perform user authentication using an IC (Integrated Circuit) card or the like. By providing the mobile robot 100 with a user authentication function, it is possible to block operations by third parties, i.e., restrict operation by third parties. This operation restriction will be described later. The user authentication function is not limited to those using IC cards. It can also be implemented by providing a sensor for reading information, such as a reader for electronic identification (ID) authentication using a mobile terminal device such as a smartphone or biometric authentication. Here, electronic ID authentication using various short-range wireless communication technologies that enable contactless information transmission and reception can reduce user effort and prevent infection. Various biometric authentication methods can be used, such as fingerprint authentication, finger vein authentication, iris authentication, and voiceprint authentication. For biometric authentication, configuring the biometric information to be read contactlessly can also reduce user effort and prevent infection.

[0062] A user can place an object in the wagon 500 placed on the mobile robot 100 and request transportation of the object using the mobile robot 100. In the following description, the wagon 500 itself can also be referred to as the object, and for convenience, the object stored in the wagon 500 will be referred to as an article. The mobile robot 100 autonomously moves to a set destination and transports the wagon 500. In other words, the mobile robot 100 executes a transportation task for the wagon 500. In the following description, the location where the wagon 500 is loaded will be referred to as the origin or loading location, and the location where the wagon 500 is delivered will be referred to as the destination or destination.

[0063] For example, let us assume that the mobile robot 100 moves within a general hospital with multiple medical departments. The mobile robot 100 transports items such as supplies, consumables, and medical instruments between the multiple medical departments. For example, the mobile robot 100 delivers items from the nurse's station of one medical department to the nurse's station of another medical department. Alternatively, the mobile robot 100 delivers items from a storage room for supplies and medical instruments to the nurse's station of a medical department. Furthermore, the mobile robot 100 delivers medicine dispensed in a pharmacy department to the medical department or patient that will use the medicine.

[0064] Examples of items include consumables such as medicines and bandages, specimens, testing equipment, medical instruments, hospital food, stationery, and other supplies. Medical equipment includes blood pressure monitors, transfusion pumps, syringe pumps, foot pumps, nurse call buttons, bed exit sensors, foot pumps, low-pressure continuous inhalers, electrocardiogram monitors, drug infusion controllers, enteral nutrition pumps, ventilators, cuff pressure gauges, touch sensors, aspirators, nebulizers, pulse oximeters, blood pressure monitors, artificial resuscitators, sterilization devices, and ultrasound machines. Food such as hospital food and test meals may also be delivered. Furthermore, the mobile robot 100 may deliver used equipment, used tableware, and other items. If the delivery destination is on a different floor, the mobile robot 100 may travel using an elevator or other means.

[0065] 2 and 3, details of the wagon 500 and an example of how the mobile robot 100 holds the wagon 500 will be described. Fig. 3 is a perspective view showing how the mobile robot 100 is transporting the wagon 500.

[0066] The wagon 500 includes a storage section for storing items, and a support section for supporting the storage section with a space formed below the storage section for allowing at least a portion of the chassis 110 to enter. As shown in FIG. 2, the storage section can be configured to include side panels 504 on both sides of the wagon 500 and an openable / closable cover 501. A user can open the cover 501 to load and unload items stored inside the wagon 500. As shown in FIG. 2, the support section can be configured to include a support frame 505 for supporting the storage section, and wheels 502 attached to the underside of the support frame 505. The wheels 502 can also be provided with a cover (not shown).

[0067] As described above, the wagon 500 can be held by the lifting mechanism 140 in the mobile robot 100. The lifting mechanism 140 is a mechanism for loading and unloading the wagon 500 as an object to be transported onto at least a portion of the upper surface of the chassis 110. By providing the lifting mechanism 140, the mobile robot 100 can easily transport the wagon 500 automatically.

[0068] As shown in FIG. 3, the mobile robot 100 can hold the wagon 500 using the lifting mechanism 140. The space into which at least a portion of the chassis 110 described above can enter is the space S formed below the wagon 500 shown in FIG. 2, and this space S is the space into which the chassis 110 can enter. In other words, the chassis 110 can enter the space S directly below the wagon 500. When the chassis 110 is to mount the wagon 500, the mobile robot 100 moves in the −X direction and enters directly below the wagon 500. The chassis 110 enters directly below the wagon 500 from the side in the front-rear direction where the stand 120 is not provided. In this way, the wagon 500 can be mounted without the stand 120 interfering with the wagon 500. In other words, the stand 120 can be attached near a corner of the chassis 110 so as not to interfere with the wagon 500.

[0069] 1, the mounting surface of the lifting mechanism 140 may be provided with a recess 141. On the other hand, a protrusion (not shown) may be provided on the underside of the storage section of the wagon 500. Then, by fitting the protrusion into the recess 141, the wagon 500 can be fixed to the mobile robot 100.

[0070] Although the wagon 500 is shown as a dolly with wheels 502, the shape and configuration of the wagon 500 are not particularly limited. The specific wagon exemplified by the wagon 500 may have any shape, size, and weight that can be transported by the mobile robot 100.

[0071] The operation of the mobile robot 100 to load the wagon 500, transport it to the destination, and unload it will be described. First, with regard to loading the wagon 500, the mobile robot 100 may be a mobile robot that is preset as a target to transport the wagon 500 and searches for the wagon 500 or moves to a known location. For example, the mobile robot 100 may be designated as a target or search target for the wagon 500 whose location has been specified by the user, and may autonomously move to transport the wagon 500. Alternatively, the mobile robot 100 may automatically transport the wagon 500 to the destination when it finds the wagon 500 on a return route after completing a transport task to transport another wagon or an item. Note that these examples are not limiting and various methods can be applied as a method for transporting the wagon 500 by the mobile robot 100.

[0072] The mobile robot 100 moves to the position of the wagon 500, and the control computer 101 recognizes the wagon 500 based on information acquired by the camera 104 or other sensors, and controls the lifting mechanism 140 to stack the wagon 500. This stacking control can also be called pickup control.

[0073] In the pickup control, first, the chassis 110 is caused to enter the space S directly below the wagon 500, and once the entry is complete, the lifting mechanism 140 is raised. As a result, the lifting stage, which is the upper surface of the lifting mechanism 140, comes into contact with the wagon 500, and the lifting mechanism 140 can lift the wagon 500. In other words, when the lifting mechanism 140 is raised, the wheels 502 lift off, and the wagon 500 is loaded onto the chassis 110. As a result, the mobile robot 100 docks with the wagon 500 and is ready to head to the destination. Next, the control computer 101 controls the drive of the wheels 111, etc. so that the wagon 500 moves autonomously along a set route, thereby transporting the wagon 500 to the destination.

[0074] The mobile robot 100 moves to the destination of the wagon 500, and the control computer 101 controls the lifting mechanism 140 to lower the wagon 500. In this control, the lifting mechanism 140 is lowered to lower the wagon 500 from the chassis 110. The wheels 502 come into contact with the floor surface, and the upper surface of the lifting mechanism 140 moves away from the wagon 500. The wagon 500 is placed on the floor surface. The wagon 500 can then be lowered from the chassis 110.

[0075] In the above examples, the mobile robot 100 has been described assuming that it transports a wagon such as the wagon 500 as the transported object. However, even if the mobile robot 100 is configured to transport a wagon, it may transport individual items (baggage) as the transported object during operation. In this case, it is recommended to attach a storage box or shelf to the mobile robot 100 to prevent the items from falling while moving.

[0076] Furthermore, in operation, there may be a situation where the mobile robot 100 needs to transport multiple items to multiple destinations. In this case, the user can unload the items at the destination regardless of whether the transport is using the wagon 500. The mobile robot 100 can transport the wagon or individual items by moving autonomously to a set destination or by moving according to user operation.

[0077] Next, an example of the main features of this embodiment will be described with reference to Fig. 4 and Fig. 5. Fig. 4 is a block diagram showing an example of the configuration of the control computer 101 in the mobile robot 100. Fig. 5 is a diagram showing an example of classification of non-staff and staff determined by the control computer 101 in Fig. 4.

[0078] In this embodiment, the mobile robot 100 includes an operation unit for operating the mobile robot 100, such as a joystick device or the operation unit 130, that is, an operation interface for operating the mobile robot 100.

[0079] The control computer 101 then executes the following modification control as at least a part of the above-described system control. This modification control recognizes the characteristics of a moving object present in the vicinity of the operation interface that operates the mobile robot 100, and modifies the operation restrictions on the operation interface if the recognized characteristics satisfy a predetermined condition. For example, if the moving object recognized as approaching the operation interface is a person who is clearly expected to operate erroneously, the control computer 101 may perform control to impose an operation restriction on the operation interface.

[0080] Here, the operation interface may include at least one of an interface such as a GUI image displayed on a display device installed in the mobile robot 100, a joystick device installed in the mobile robot 100, and a remote control device for remotely controlling the mobile robot 100. The display device may be exemplified by a display unit provided in the operation unit 130, and the joystick device may be exemplified by a joystick unit provided with a stick unit 131.

[0081] In the following, an example will be described in which the operation interface includes a joystick device and an operation unit 130. However, as described above, the operation interface may be any interface that allows user operations related to control of the mobile robot 100, and may be, for example, an interface that accepts operations to move the mobile robot 100 in a user operation mode. However, the operation interface may also be configured to accept an operation to switch between the autonomous movement mode and the user operation mode, and as described above, some user operations, such as this switching operation, may be accepted even in the autonomous movement mode.

[0082] Furthermore, the moving object includes a person, and may also include other mobile robots of the same type or other mobile robots of different types. Whether or not the moving object is present in the vicinity of the joystick device and operation unit 130 can be determined by the control computer 101, for example, based on image data captured by the camera 104 or a camera (not shown) mounted on the operation unit 130 or joystick device (hereinafter referred to as the operation unit camera). For example, the control computer 101 analyzes the image data, and if it can recognize a moving object with a face and that the face is a predetermined size or larger, it can determine that a moving object is present in the vicinity.

[0083] Alternatively, the control computer 101 may adopt a configuration in which the mobile robot 100 can be connected to an environmental camera (not shown) installed in the facility, and in such a configuration, it can determine whether or not a moving object is present in the vicinity based on image data captured by the environmental camera. For example, the control computer 101 can analyze the image data and determine that a moving object with a face and the mobile robot 100 are both recognized, and that the distance between the moving object and the mobile robot 100 is within a certain distance, and then determine that a moving object is present in the vicinity.

[0084] Alternatively, the control computer 101 can provide the mobile robot 100 with a user authentication function, such as an IC card reader, and determine whether or not a moving object is present in the vicinity based on user information obtained by the user authentication function. That is, the control computer 101 can determine that a moving object is present in the vicinity when the user authentication function is used, and can determine that no moving object is present in the vicinity when the user authentication function is not used. Furthermore, the control computer 101 can determine whether or not the user is a pre-registered staff member (i.e., a non-staff member) by obtaining the result of user authentication.

[0085] Furthermore, when determining the presence of a moving object using image data, the characteristics of the moving object may refer to the size of the moving object, the attributes of the moving object, etc. The attributes may be, for example, an attribute indicating whether the object is a child or not, or an attribute indicating whether the object is a staff member or not. When determining the presence of a moving object using a user authentication function, the characteristics of the moving object may be, for example, an attribute indicating whether the object is a registered staff member or not.

[0086] With this configuration, the mobile robot 100 can change the operation limits according to the moving objects recognized around the joystick device and the operation unit 130, which prevents the safety of the mobile robot 100 to be operated from varying depending on the operator, thereby improving safety. Note that this effect is obtained for the operation interface mounted on the mobile robot 100.

[0087] For the above-mentioned change control, the control computer 101 can include, for example, an information acquisition unit 101a, a feature recognition unit 101b, a determination unit 101c, and an operation restriction change unit 101d, as shown in FIG.

[0088] The information acquisition unit 101a acquires processing target information, which is information to be processed. As described above, the information acquisition unit can acquire image data as the processing target information, and can be referred to as an image data acquisition unit. The following description will be given on the assumption that image data is acquired as the processing target information, but the information acquisition unit 101a can also be configured to acquire processing target information other than image data, such as user information obtained by a user authentication function, detected by a sensor.

[0089] The information acquisition unit 101a acquires image data of an image captured by a camera. Here, various cameras as described above can be used as the camera. The image data may be the image data itself captured by the camera, or may be data obtained by processing the image data. For example, the image data may be data of features extracted from the image data. Information such as the time and location of capture may also be added to the image data. The information acquisition unit 101a may acquire image data from multiple cameras.

[0090] The feature recognition unit 101b recognizes the features of moving objects present around the joystick device and the operation unit 130 based on the image data acquired by the information acquisition unit 101a. The feature recognition unit 101b can recognize the features of the moving object by extracting them from the image data and comparing the extracted features with predetermined features. Therefore, the feature recognition unit 101b can also be called a feature extraction unit.

[0091] More specifically, the feature recognition unit 101b detects a moving object included in the image data by performing image processing on the image data. Then, the feature recognition unit 101b extracts features of a moving object, such as a person, included in the image data. Alternatively, a calculation processing unit (not shown) provided in a camera that acquires the image data may perform at least a part of the processing for extracting features. Note that various techniques are known to those skilled in the art as a means for detecting that a moving object, such as a person, is included in image data, such as HOG (Histograms of Oriented Gradients) features and machine learning including convolution processing. Therefore, detailed description thereof will be omitted here.

[0092] An example will be described in which the predetermined condition is that the nearby moving object is a non-staff member. In this case, the feature recognition unit 101b detects, for example, the color of clothing worn by the detected moving object so as to distinguish between staff and non-staff members. More specifically, for example, the feature recognition unit 101b can calculate the proportion of an area of ​​a specific color in the clothing worn by the detected moving object, or in the clothing worn by a person actually detected due to the presence of clothing. Alternatively, the feature recognition unit 101b detects the color of clothing worn in a specific portion of the clothing worn by the detected person.

[0093] In this way, the feature recognition unit 101b can extract characteristic features from the staff member's clothing, and if the extraction is successful, it can recognize that the feature refers to a staff member. On the other hand, if the feature recognition unit 101b cannot extract characteristic features from the staff member's clothing, it can recognize that the feature refers to a non-staff member.

[0094] Whether clothing belongs to staff or non-staff can be determined by detecting the colors of the clothing of "pharmacist" and "nurse" as shown in Figure 5, and by detecting no clothing of those colors, determining that the clothing belongs to non-staff. In fact, in hospitals, staff wear uniforms of different colors and shapes according to their category, such as pharmacist or nurse, making it possible to recognize whether or not someone is staff. Of course, features extracted are not limited to clothing color; they may also include the shape of the clothing, hats, or other distinctive shapes of staff clothing, as well as distinctive accessories and other personal belongings. Of course, staff can also include other mobile robots moving within the facility, in which case shape features can be extracted from clothing features.

[0095] The feature recognition unit 101b may also extract features of facial images of moving objects present in the vicinity. That is, the feature recognition unit 101b may extract features for facial recognition and compare them with facial features registered in advance to recognize a person.

[0096] The feature recognition unit 101b may also include a learning model 101ba, which is a trained model. Specifically, the feature recognition unit 101b may be configured to recognize features using a learning model 101ba that receives images captured around the joystick device and the operation unit 130 and outputs features. In this case, for example, machine learning can be performed using images captured for each staff category as training data. That is, by performing supervised learning using image data to which the staff category is labeled as the correct answer as training data, a machine learning model with high recognition accuracy can be constructed. That is, images captured of staff wearing a specific uniform can also be used as training data. Note that the algorithm of the learning model 101ba is not critical. In this way, the feature recognition unit 101b can accurately recognize features by constructing a highly accurate learning model 101ba.

[0097] Regardless of the method used to recognize the features, the feature recognition unit 101b supplies the recognized features to the determination unit 101c. Note that even when acquiring processing target information other than image data, it is possible to obtain features according to the acquired processing target information (for example, staff / non-staff information in the case of a user authentication function).

[0098] The determination unit 101c determines whether the recognized feature satisfies a predetermined condition. As in the above example, if the predetermined condition is that the moving object present in the vicinity is not a staff member, the recognized feature is determined to satisfy the predetermined condition.

[0099] The determination unit 101c can also determine whether a feature satisfies a predetermined condition by using a learning model (not shown) that receives a feature and outputs whether the feature satisfies a predetermined condition. The algorithm of this learning model is not important. By constructing a highly accurate learning model, the determination unit 101c can accurately determine whether the feature satisfies a predetermined condition.

[0100] Regardless of the determination method, and regardless of the method used to perform the determination, if a predetermined condition is satisfied, the determination unit 101c supplies information indicating that the condition is satisfied to the operation restriction change unit 101d. On the other hand, if the predetermined condition is not satisfied, the determination unit 101c does not perform such a supply, or supplies information indicating that the condition is not satisfied to the operation restriction change unit 101d.

[0101] In the configuration example of FIG. 4, the feature recognition unit 101b and the determination unit 101c are distinguished from each other. However, the function of the feature recognition unit 101b can be included in the determination unit 101c. In this case, a learning model (not shown) can be used to obtain a result of whether or not a predetermined condition is satisfied. This learning model can be any learning model that inputs an image obtained by capturing an image of the joystick device and the surroundings of the operation unit 130 and outputs whether or not the image satisfies the predetermined condition, and the algorithm, etc., of the learning model are not important. The determination unit 101c can also use such a learning model to determine whether or not a feature satisfies the predetermined condition. In this way, the determination unit 101c can accurately determine whether or not a predetermined condition is satisfied by constructing a highly accurate learning model.

[0102] In addition, whether the information acquisition unit 101a acquires image data or acquires information to be processed other than image data, the judgment unit 101c can regard the area from which the information to be processed was acquired as a characteristic of the moving body and judge whether or not the area satisfies predetermined conditions based on the characteristic.

[0103] Specifically, in a facility where the mobile robot 100 is moved, area divisions are set in the map data of the facility, and the feature recognition unit 101b can recognize the features of the mobile body depending on which of the set divisions the area in which the information to be processed is acquired corresponds to.

[0104] Examples of methods for dividing areas include whether they are staff-only areas, areas reserved for special staff authorized to handle drugs, and children's areas where children are likely to be present. Areas that are not staff-only areas include waiting rooms, areas where patients are present, such as the hallways in front of examination rooms, and child care rooms. Children's areas include child care rooms. The feature recognition unit 101b can consider the identification of the area classification of the area where the mobile robot 100 is located as having identified, or recognized, the characteristics of the surrounding moving objects. For example, if the area where the mobile robot 100 is located is a staff-only area, a special staff-only area, or a children's area, the feature recognition unit 101b can recognize the characteristics of the moving object as staff characteristics, special staff characteristics, and children's characteristics, respectively. The determination unit 101c can then determine whether a predetermined condition is met based on the recognized characteristics.

[0105] In particular, in facilities such as hospitals where there may be many unspecified persons, the user information obtained by the user authentication function only provides minimal information about the characteristics of moving objects, such as staff / non-staff. However, by combining this recognition of moving object characteristics based on area divisions, particularly by dividing areas into various divisions, it is possible to recognize more features from processing target information in which many features cannot be recognized. Of course, the accuracy of recognizing moving object characteristics can also be improved when image data is the processing target information. Furthermore, when image data is the processing target information, even if area divisions are not set in the map data, it is possible to identify the divisions by placing marks such as signs or stickers indicating the area divisions in various locations within the facility and recognizing the marks from the image data, i.e., to recognize the characteristics of moving objects.

[0106] When the determination unit 101c determines that the predetermined condition is satisfied, the operation limit change unit 101d changes the operation limit on the joystick device and the operation unit 130. The operation limit can be changed, for example, by the operation limit change unit 101d sending a change instruction to the joystick device and the operation unit 130.

[0107] Next, examples of operation restrictions will be described with reference to Fig. 6 to Fig. 8. Fig. 6 is a diagram showing an example of a GUI image displayed on the operation unit 130 under the control of the control computer 101 in Fig. 4, Fig. 7 is a diagram showing an example of a GUI image when operation restrictions are imposed on the GUI image in Fig. 6, and Fig. 8 is a diagram showing another example of a GUI image when operation restrictions are imposed on the GUI image in Fig. 6.

[0108] Here, the above-mentioned predetermined condition will be described by taking as an example a case where the recognized feature is a feature indicating a non-staff member of the facility that operates the mobile robot 100. However, for example, the same processing can be applied when the predetermined condition is a feature indicating a child, and the same processing can be applied when the predetermined condition is a feature indicating a child and a feature indicating a person using crutches. Furthermore, the same processing can be applied when the predetermined condition is a feature indicating a non-staff member and a feature indicating a child.

[0109] As a modification control, the control computer 101 can perform control to impose operational restrictions on the joystick device and the operation unit 130 when the characteristics satisfy a predetermined condition.

[0110] By this control, when an operator who would lower safety if allowed to operate the joystick device and the operation unit 130 is present near the joystick device and the operation unit 130, it is possible to impose operation restrictions compared to when an operator who can maintain safety even if allowed to operate the joystick device and the operation unit 130 is present near the joystick device and the operation unit 130. As in this example, by restricting operation by anyone other than the staff of the facility that operates the mobile robot 100, it is possible to ensure the safety of people around the mobile robot 100. Therefore, this control makes it possible to prevent safety from varying depending on the operator.

[0111] More specifically, the operation restriction can be a restriction that prohibits some or all of the operations of the joystick device and the operation unit 130. This makes it possible to prohibit operations of the joystick device and the operation unit 130 when a predetermined condition is met, thereby further preventing safety from varying depending on the operator.

[0112] Here, the operation restriction on the joystick device can be implemented, for example, by mechanically fixing the stick portion 131 so that it does not move.

[0113] Furthermore, the operation restriction on the operation unit 130 can be, for example, a restriction that makes it impossible to accept some or all of the operations on the operation screen. Regarding such a restriction, as shown in Fig. 6, an example will be described in which the display unit 80 provided in the operation unit 130 displays an operation image 81 and a camera image 82 that is being captured by the camera 104 as GUI images before the operation restriction is imposed. In this example, the operation image 81 displays an emergency stop button 83, a nurse call button 84, a mode switch button 85, a change destination button 86, a change route button 87, a button 88 to unload the cart, and a button 89 to move to a charging location in a selectable manner.

[0114] The emergency stop button 83 is a button for making an emergency stop of the mobile robot 100, and when selected, the control computer 101 controls the mobile robot 100 to make an emergency stop. The nurse call button 84 is a button for calling a nurse, and when selected, the control computer 101 controls the control computer 101 to send a call notification via wireless communication to a pre-registered notification destination. The mode switch button 85 is a button for switching between an autonomous movement mode and a user operation mode, and when selected, the control computer 101 switches from the current mode to another mode.

[0115] The change destination button 86 is a button for changing the destination; when selected, the control computer 101 switches to an image that allows the user to select a new destination, and then changes to the selected destination when the destination is selected. The change route button 87 is a button for changing the transport route; when selected, the control computer 101 switches to an image that allows the user to select a transport route, and then changes to the selected transport route when the transport route is selected. The unload wagon button 88 is a button for unloading the wagon 500; when selected, the control computer 101 controls the lifting mechanism 140, etc., to unload the wagon 500. The move to charging location button 89 is a button for moving the mobile robot 100 to a charging location; when selected, the control computer 101 changes the destination or stopover to, for example, the nearest charging location.

[0116] When an operation restriction is imposed on such an operation image 81, the control computer 101 can make all buttons other than the nurse call button 84 unselectable, as shown in Fig. 7. Of course, the control computer 101 may leave some of the other buttons selectable, or may make all of the buttons unselectable.

[0117] Furthermore, the operation restriction is not limited to prohibition of operation. For example, the operation restriction can be a restriction that makes it difficult to operate some or all of the joystick device and operation unit 130. More specifically, an alarm message can be displayed as a pop-up image on the display unit 80 to make the operation difficult, but the restriction that makes the operation difficult is not limited to this. In this way, when a predetermined condition is met, it is possible to make the operation of the joystick device and operation unit 130 difficult, thereby further preventing safety from varying depending on the operator.

[0118] Furthermore, as part of the system control, the control computer 101 can also execute notification change control, which changes at least one of the content and method of notification by the joystick device and the operation unit 130 when the characteristics satisfy predetermined conditions. Notification regarding the joystick device can be performed, for example, by the second light-emitting unit 12. When the characteristics satisfy predetermined conditions, the light-emitting pattern of the second light-emitting unit 12 can be changed to make the light-emitting brightness or color of the second light-emitting unit 12 more noticeable. In this case, not only the light-emitting unit 12 but also the light-emitting pattern of the first light-emitting unit 11 can be changed. Notification regarding the operation unit 130 can be performed by displaying a pop-up image, such as "Do not operate," when operation is restricted. The notification change control can also include control to output an alarm sound or an alarm message aloud.

[0119] Furthermore, as part of the system control, the control computer 101 can also execute display control to display an image of a moving object on the operation unit 130, such as the camera image 82a in FIG. 8, when the characteristics satisfy a predetermined condition. The camera image 82a can be an image captured by a camera provided in the operation unit 130. By displaying this as a pop-up image, for example, as shown in the figure, the person attempting to operate the device will appear on the display unit 80 as the camera image 82a, which can cause the person to hesitate to operate the device. In this case, as illustrated in FIG. 8, it is even more effective to make some or all of the buttons inoperable or difficult to operate, as in the example of FIG. 7. The camera image 82a can also be displayed in place of the camera image 80. The display control exemplified by the display of the camera image 82a can also be executed together with the notification change control described above.

[0120] In this way, with this configuration, when a predetermined condition is satisfied, the mobile robot 100 can at least one of notifying the user through the joystick device and the operation unit 130 and displaying an image of the moving object. With this control, when an operator who would lower safety if allowed to operate the joystick device and the operation unit 130 is present near the joystick device and the operation unit 130, the operator can be discouraged from trying to operate the joystick device and the operation unit 130, thereby further preventing the safety from varying depending on the operator.

[0121] In addition, although the predetermined condition has been given as an example of a case where a nearby moving object is a non-staff member, by determining whether or not the non-staff member is a child, different operation restriction methods can be adopted for children and non-staff members. Children can be distinguished from adults by determining their age or height, etc.

[0122] It is to be noted that, assuming that the specified conditions will no longer be met after a certain period of time has passed, the operation restriction may be implemented, for example, for only a certain period of time, but it is also possible to continue making judgments and continuing to implement operation restrictions until the specified conditions are no longer met.

[0123] Furthermore, as part of the system control, the control computer 101 can also execute control to stop the mobile robot 100 when a predetermined condition is met. In this way, when a predetermined condition is met, not only control to change the operation limit but also control to stop the mobile robot 100 can be executed, thereby further improving safety.

[0124] Next, an example of the flow of such an operation restriction process will be described with reference to Fig. 9. Fig. 9 is a flow chart for explaining an example of the operation restriction process executed by the control computer 101 of Fig. 4.

[0125] First, the control computer 101 acquires information to be processed, such as an image captured by the camera 104 or a camera (not shown) provided on the operation unit 130, in order to recognize the characteristics of a moving object present around the operation unit 130 (step S11). Next, the control computer 101 recognizes the characteristics based on the acquired information to be processed (step S12).

[0126] Next, the control computer 101 determines whether the recognized feature corresponds to a non-staff feature (step S13), and if the result is YES, it transmits an instruction to impose an operation restriction on the joystick device and the operation unit 130 (step S14), and ends the process. On the other hand, if the result is NO in step S13, the control computer 101 ends the process as it is.

[0127] Next, another example of the operation restriction process will be described with reference to Fig. 10. Fig. 10 is a flow diagram for explaining another example of the operation restriction process executed by the control computer 101 of Fig. 4. Note that, in the example of Fig. 10 as well, the various examples described in Figs. 1 to 8 can be applied, except for the differences from the example of Fig. 9.

[0128] First, the control computer 101 acquires processing target information (step S21) in the same manner as in steps S11 and S12, and recognizes features based on the acquired processing target information (step S22).

[0129] Next, the control computer 101 determines whether the recognized characteristics correspond to the characteristics of a staff member of the facility that operates the mobile robot 100 (step S23). If the result of step S23 is YES, the control computer 101 sends an instruction to the joystick device and the operation unit 130 to relax the operation restrictions (step S24), and ends the process. On the other hand, if the result of step S23 is NO, the control computer 101 ends the process as is, that is, with the operation restrictions still imposed.

[0130] As shown in the example of Figure 10, the predetermined condition can be that the characteristic is a characteristic that indicates a staff. In this case, the change control is a control that relaxes the operation restrictions on the joystick device and operation unit 130 when the characteristic satisfies the predetermined condition. In other words, such operation restrictions are imposed before the change control. For example, the control computer 101 can hide the display unit 80 on the operation unit 130 when the predetermined condition is not met, and display the display as shown in Figure 6 when the condition is met.

[0131] In this way, the control computer 101 can impose operation restrictions when an operator who would lower safety if allowed to operate the joystick device and operation unit 130 is present near the joystick device and operation unit 130, and can relax operation restrictions when an operator who can maintain safety even if allowed to operate the joystick device and operation unit 130 is present near the joystick device and operation unit 130. Even with this type of control, it is possible to prevent safety from varying depending on the operator.

[0132] Other application examples can be applied, for example, to the examples described with reference to Figures 6 to 8. However, in the notification change control, for example, when the feature satisfies a predetermined condition, it is preferable to perform control as if no operation restriction is imposed when the operation restriction is relaxed, such as changing the light emission pattern so that the light emission brightness or light emission color of second light-emitting unit 12 is not noticeable.

[0133] It is to be noted that, assuming that the specified conditions will no longer be met after a certain period of time has passed, the relaxation of operation restrictions can be carried out, for example, for only a certain period of time, but it is also possible to continue making judgments and continuing to relax operation restrictions until the specified conditions are no longer met.

[0134] In the above explanation, an example was given in which the transport system is mainly composed of the mobile robot 100, but the control system according to this embodiment may be any system that executes system control for controlling a system including a mobile robot, such as the transport system, as described above. This system may also include a server that can be connected to the mobile robot 100 via wireless communication. This server provides the mobile robot 100 with information for autonomous movement. Because this server manages the mobile robot 100, it may also be called a host management device.

[0135] An example of this transport system including the mobile robot 100 and a host management device will be described below with reference to Fig. 11. Fig. 11 is a schematic diagram showing an example of the overall configuration of a transport system including the mobile robot 100.

[0136] 11, the transport system 1 includes a mobile robot 100, a host management device 2, a network 3, a communication unit 4, an environmental camera 5, and a user terminal device 300. The transport system 1 is a system that transports an object using the mobile robot 100, and includes the control system in this configuration example. In this example, the control system can refer to the mobile robot 100 and the host management device 2, or can refer to the components of the control system provided in the mobile robot 100 and the host management device 2.

[0137] The mobile robot 100 and the user terminal device 300 are connected to a host management device 2 via a communication unit 4 and a network 3. The network 3 is a wired or wireless LAN (Local Area Network) or WAN (Wide Area Network). Furthermore, the host management device 2 and the environmental camera 5 are connected to the network 3 by wire or wireless. As can be seen from this configuration, the mobile robot 100, the host management device 2, and the environmental camera 5 all have communication units. The communication unit 4 is, for example, a wireless LAN unit installed in each environment. The communication unit 4 may also be, for example, a general-purpose communication device such as a WiFi (registered trademark) router.

[0138] The host management device 2 is a device that can be connected to the mobile robots 100 via wireless communication, is a management system that manages multiple mobile robots 100, and can be equipped with a control unit 2a that controls it. The control unit 2a can be realized, for example, by an integrated circuit, and can be realized, for example, by a processor such as an MPU or CPU, a working memory, and a non-volatile storage device. A control program to be executed by the processor is stored in this storage device, and the processor reads the program into the working memory and executes it, thereby fulfilling the functions of the control unit 2a. The control unit 2a can be called a control computer.

[0139] The transport system 1 can efficiently control multiple mobile robots 100 within a specific facility, either by moving the mobile robots 100 autonomously in an autonomous movement mode or by moving the mobile robots 100 based on user operation in a user operation mode. Note that the facility can refer to various types of facilities, such as medical and welfare facilities such as hospitals, rehabilitation centers, nursing homes, and elderly care facilities, hotels, restaurants, office buildings, event venues, commercial facilities such as shopping malls, and other complexes.

[0140] To achieve such efficient control, multiple environmental cameras 5 can be installed within a facility. The environmental cameras 5 capture images of the area in which people and the mobile robot 100 move and output image data representing the images. This image data can be still image data or moving image data. If still image data is used, still image data is obtained at each capture interval. In the transport system 1, the images captured by the environmental cameras 5 and information based thereon are collected by the host management device 2. Images used to control the mobile robot 100 can be transmitted directly to the mobile robot 100, or in the user operation mode, can be transmitted to the user terminal device 300 via the host management device 2 or directly. The environmental cameras 5 can be installed as surveillance cameras in the corridors and entrances of the facility.

[0141] For each transport request, the upper management device 2 can determine the mobile robot 100 that will execute the transport task and send an operation command to the determined mobile robot 100 to execute the transport task. The mobile robot 100 can autonomously move from the transport source to the transport destination according to the operation command. Note that the method of determining the transport route at this time is not limited.

[0142] For example, the host management device 2 assigns a transport task to a mobile robot 100 at or near the transport source. Alternatively, the host management device 2 assigns a transport task to a mobile robot 100 heading toward or near the transport source. The mobile robot 100 assigned the task will go to the transport source to pick up the transported item.

[0143] The user terminal device 300 is a device that remotely controls the mobile robot 100 in the user operation mode, either directly or via the host management device 2, and may be equipped with a communication function for this purpose, and may also be equipped with a display unit 304. Various types of terminal devices, such as a tablet computer or a smartphone, can be used as the user terminal device 300. The user terminal device 300 can also accept a switching operation between the user operation mode and the autonomous movement mode, and when this switching operation is performed, the mode of the mobile robot 100 can be switched via the host management device 2. The user terminal device 300 can also be used for authentication using the electronic ID described above.

[0144] Here, an example is given in which the user terminal device 300 includes a joystick device. In addition to the main body 31, the user terminal device 300 may include a stick unit 302 and a button 303 as parts of the joystick device. In user operation mode, this joystick device is a device for performing operations to move the mobile robot 100 in a direction intended by the user. Directional operations can be accepted by tilting the stick unit 302 in the desired direction of movement. The joystick device can also be controlled so that a switching operation, which switches between the autonomous movement mode and the user operation mode, is performed by pressing the button 303 downward. Alternatively, the joystick device can be controlled so that a confirmation operation is performed by pressing the button 303 downward. The button 303 can also be configured to function as an emergency stop button when pressed downward for a predetermined period of time. When the button 303 is configured to be able to accept multiple operations from the switching operation, confirmation operation, and emergency stop operation, i.e., when multiple operations are assigned to the button 303, it is sufficient to set a predetermined period corresponding to each operation.

[0145] Furthermore, if the user terminal device 300 is equipped with a joystick device, the user can perform the same operation as a joystick device even if the mobile robot 100 does not have a joystick device. The mobile robot 100 may also be equipped with a button such as button 303 on the top surface of the stick 131. Furthermore, in a configuration in which the transport system 1 manages multiple mobile robots 100, in the user operation mode, the mobile robot 100 to be remotely controlled can be selected from the user terminal device 300.

[0146] The display unit 304 can display an image represented by image data received from the camera 104 in the mobile robot 100 and an image represented by image data received from the environmental camera 5 located around the mobile robot 100. This allows the user to operate the mobile robot 100 using the stick unit 302 and the button 303.

[0147] The user terminal device 300 can also function as a device for issuing a transport request to the higher-level management device 2. This transport request can also include information indicating the item to be transported.

[0148] The change control in the transport system 1 can be executed by the control computer 101 of the mobile robot 100, by the control unit 2a of the host management device 2 recognizing the characteristics of a moving object present in the vicinity of the operation interface, and transmitting an instruction to the mobile robot 100 to change the operation restrictions on the operation interface if the characteristics satisfy a predetermined condition. The various examples described above for the case where the mobile robot 100 executes the change control can be similarly applied to examples of the recognition of characteristics, the determination of predetermined conditions, and the operation restrictions.

[0149] The present embodiment has been described above, but the control method according to the present embodiment may be performed by the host management device 2 or by an edge device. The edge device may include, for example, one or more of the environmental camera 5, the mobile robot 100, the communication unit 4, and the user terminal device 300. The environmental camera 5, the mobile robot 100, and the host management device 2 may cooperate to execute the control method. That is, the control system according to the present embodiment may be mounted in the environmental camera 5. Alternatively, at least a part or all of the control system may be mounted in a device other than the mobile robot 100, for example, the host management device 2. The host management device 2 is not limited to being a single physical device, but may be distributed across multiple devices. That is, the host management device 2 may include multiple memories and multiple processors.

[0150] Furthermore, the transport system 1, the host management device 2, the mobile robot 100, the user terminal device 300, the environmental camera 5, and the communication unit 4 according to the above-described embodiment are not limited to those having the shapes shown in the examples or those performing the controls shown in the examples, as long as they can perform the functions of each device. Furthermore, in the above-described embodiment, an example was given in which the control system was incorporated into the transport system, but it is not necessary that it be incorporated into the transport system.

[0151] Furthermore, the control computer 101, the host management device 2, the environmental camera 5, and the user terminal device 300 of the mobile robot 100 according to the above-described embodiment may each have, for example, the following hardware configuration. Fig. 12 is a diagram showing an example of the hardware configuration of the devices.

[0152] 12 may include a processor 1001, a memory 1002, and an interface 1003. The interface 1003 may include, for example, a communication interface and interfaces with a drive unit, a sensor, an input / output device, etc., as required depending on the device.

[0153] The processor 1001 may be, for example, an MPU, a CPU, or a GPU (Graphics Processing Unit). The processor 1001 may include multiple processors. The memory 1002 is configured, for example, by a combination of volatile memory and non-volatile memory. The functions of each device are realized by the processor 1001 reading a program stored in the memory 1002 and executing it while exchanging necessary information via the interface 1003.

[0154] Furthermore, the above-described program includes a set of instructions (or software code) that, when loaded into a computer, causes the computer to perform one or more functions described in the embodiments. The program may be stored in a non-transitory computer-readable medium or a tangible storage medium. By way of example and not limitation, computer-readable media or tangible storage media include random-access memory (RAM), read-only memory (ROM), flash memory, solid-state drive (SSD) or other memory technology, CD-ROM, digital versatile disc (DVD), Blu-ray (registered trademark) disc or other optical disk storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage device. The program may also be transmitted on a transitory computer-readable medium or communication medium. By way of example and not limitation, transitory computer-readable media or communication media include electrical, optical, acoustic, or other forms of propagated signals.

[0155] The present invention is not limited to the above-described embodiment, and can be modified as appropriate within the scope of the invention. [Explanation of symbols]

[0156] 1. Transport system 2 Upper management device 3 Network 4 Communication Unit 5. Environmental Camera 11 First light-emitting part 12 Second light-emitting part 80 Display section 81 Operation Images 82 camera images 82a Camera image 100 Mobile Robots 101 Control Computer 101a Information acquisition section 101b Feature recognition unit 101ba Learning Model 101c Judgment section 101d Operation restriction change section 104 Camera 110 chassis 111 Wheels 120 Stand 130 Operation section 131 Stick part 140 Lifting mechanism 141 recess 300 User terminal device 500 Wagon 501 Cover 502 wheels

Claims

1. executes system control to control a system including a mobile robot in accordance with characteristics of moving objects present in the vicinity of an operation interface for operating a mobile robot that is capable of autonomous movement and is user-operable; the system control includes a change control that changes an operation restriction on the operation interface when the feature satisfies a predetermined condition; the predetermined condition includes a first condition in which the characteristic is one of a characteristic indicative of a non-staff member of a facility that operates the mobile robot and a characteristic indicative of a child; the change control includes control of imposing the operation restriction on the operation interface when the feature satisfies the first condition; the system control includes control of displaying an image of the moving object on the operation interface when the feature satisfies the first condition. Control system.

2. the predetermined condition includes a second condition in which the characteristic is indicative of a staff member of a facility that operates the mobile robot; the change control includes control of relaxing the operation restriction on the operation interface when the feature satisfies the second condition. The control system of claim 1 .

3. the operation interface includes at least one of an interface to be displayed on a display device installed on the mobile robot, a joystick device installed on the mobile robot, and an operation device for remotely operating the mobile robot; 3. A control system according to claim 1 or 2.

4. the system control includes control of stopping the mobile robot when the first condition is satisfied.

3. A control system according to claim 1 or 2.

5. recognizing the feature based on an image obtained by capturing an image of the periphery of the operation interface; 3. A control system according to claim 1 or 2.

6. an image obtained by capturing an image of the periphery of the operation interface is input, and a learning model is used to output whether the image satisfies the predetermined condition, thereby determining whether the feature satisfies the predetermined condition; 3. A control system according to claim 1 or 2.

7. The operation restriction is a restriction that prohibits some or all of the operations in the operation interface.

3. A control system according to claim 1 or 2.

8. The operation restriction is a restriction that makes it difficult to perform some or all of the operations in the operation interface.

3. A control system according to claim 1 or 2.

9. the system control includes control of changing at least one of a content of a notification and a method of a notification by the operation interface when the feature satisfies the predetermined condition.

3. A control system according to claim 1 or 2.

10. The facility is a facility of a medical institution, the operation interface is capable of accepting an operation to call the staff member, The operation restriction on the operation interface is an operation restriction that does not restrict at least an operation of calling the staff member.

10. The control system of claim 9.

11. executes system control to control a system including a mobile robot in accordance with characteristics of moving objects present in the vicinity of an operation interface for operating a mobile robot that is capable of autonomous movement and is user-operable; the system control includes a change control that changes an operation restriction on the operation interface when the feature satisfies a predetermined condition; the predetermined condition includes a first condition in which the characteristic is one of a characteristic indicative of a non-staff member of a facility that operates the mobile robot and a characteristic indicative of a child; the change control includes control of imposing the operation restriction on the operation interface when the feature satisfies the first condition; the system control includes control of displaying an image of the moving object on the operation interface when the feature satisfies the first condition. Control method.

12. the predetermined condition includes a second condition in which the characteristic is indicative of a staff member of a facility that operates the mobile robot; the change control includes control of relaxing the operation restriction on the operation interface when the feature satisfies the second condition. The control method according to claim 11.

13. the operation interface includes at least one of an interface to be displayed on a display device installed on the mobile robot, a joystick device installed on the mobile robot, and an operation device for remotely operating the mobile robot; 13. The control method according to claim 11 or 12.

14. the system control includes control of stopping the mobile robot when the first condition is satisfied.

13. The control method according to claim 11 or 12.

15. recognizing the feature based on an image obtained by capturing an image of the periphery of the operation interface; 13. The control method according to claim 11 or 12.

16. The determination of whether the feature satisfies the predetermined condition is performed using a learning model that inputs an image obtained by capturing an image of the periphery of the operation interface and outputs whether the image satisfies the predetermined condition.

13. The control method according to claim 11 or 12.

17. The operation restriction is a restriction that prohibits some or all of the operations in the operation interface.

13. The control method according to claim 11 or 12.

18. The operation restriction is a restriction that makes it difficult to perform some or all of the operations in the operation interface.

13. The control method according to claim 11 or 12.

19. the system control includes control of changing at least one of a content of a notification and a method of a notification by the operation interface when the feature satisfies the predetermined condition.

13. The control method according to claim 11 or 12.

20. The facility is a facility of a medical institution, the operation interface is capable of accepting an operation to call the staff member, The operation restriction on the operation interface is an operation restriction that does not restrict at least an operation of calling the staff member.

20. The control method of claim 19.

21. On the computer, A program for executing a process of executing system control for controlling a system including a mobile robot in accordance with characteristics of moving objects present in the vicinity of an operation interface for operating a mobile robot that is capable of autonomous movement and that can be operated by a user, the program comprising: the system control includes a change control that changes an operation restriction on the operation interface when the feature satisfies a predetermined condition; the predetermined condition includes a first condition in which the characteristic is one of a characteristic indicative of a non-staff member of a facility that operates the mobile robot and a characteristic indicative of a child; the change control includes control of imposing the operation restriction on the operation interface when the feature satisfies the first condition; the system control includes control of displaying an image of the moving object on the operation interface when the feature satisfies the first condition. program.

22. the predetermined condition includes a second condition in which the characteristic is indicative of a staff member of a facility that operates the mobile robot; the change control includes control of relaxing the operation restriction on the operation interface when the feature satisfies the second condition.

22. The program of claim 21.

23. the operation interface includes at least one of an interface to be displayed on a display device installed on the mobile robot, a joystick device installed on the mobile robot, and an operation device for remotely operating the mobile robot; 23. The program according to claim 21 or 22.

24. the system control includes control of stopping the mobile robot when the first condition is satisfied.

23. The program according to claim 21 or 22.

25. recognizing the feature based on an image obtained by capturing an image of the periphery of the operation interface; 23. The program according to claim 21 or 22.

26. The determination of whether the feature satisfies the predetermined condition is performed using a learning model that inputs an image obtained by capturing an image of the periphery of the operation interface and outputs whether the image satisfies the predetermined condition.

23. The program according to claim 21 or 22.

27. The operation restriction is a restriction that prohibits some or all of the operations in the operation interface.

23. The program according to claim 21 or 22.

28. The operation restriction is a restriction that makes it difficult to perform some or all of the operations in the operation interface.

23. The program according to claim 21 or 22.

29. the system control includes control of changing at least one of a content of a notification and a method of a notification by the operation interface when the feature satisfies the predetermined condition.

23. The program according to claim 21 or 22.

30. The facility is a facility of a medical institution, the operation interface is capable of accepting an operation to call the staff member, The operation restriction on the operation interface is an operation restriction that does not restrict at least an operation of calling the staff member.

30. The program of claim 29.

Images