Mobile body control device, mobile body control method, and program

Abstract

This mobile body control device according to an embodiment comprises: a recognition unit that recognizes a situation around a mobile body and a user on the basis of an image in which the situation around the mobile body is captured; and a control unit that executes following control in which at least the mobile body moves following the user, on the basis of the surrounding situation. The control unit sets an entry prohibition region for the mobile body based on a line formed by a plurality of persons recognized by the recognition unit, and limits the following control when the user enters the set entry prohibition region.

Description

MOBILE BODY CONTROL DEVICE, MOBILE BODY CONTROL METHOD, AND PROGRAM

[0001] The present invention relates to a control device for a mobile object, a control method for a mobile object, and a program.

[0002] In recent years, various technologies related to autonomously moving vehicles have been researched, including a technology related to an autonomous vehicle that transports objects loaded by a worker along a predetermined travel route and a technology that uses an elevator to move a self-propelled robot from a waiting point on a current floor to a destination point on a destination floor (see, for example, Patent Documents 1 and 2).

[0003] JP 2023-81161 A JP 2023-88139 A

[0004] However, when moving a moving object so as to follow the movement of a user, there are cases where the operation of the moving object cannot be appropriately controlled depending on the user's situation and surrounding circumstances.

[0005] The present invention has been made in consideration of these circumstances, and one of its objects is to provide a control device for a moving body, a control method for a moving body, and a program that can more appropriately control the movement of a moving body that follows a user.

[0006] The present invention provides a control device for a moving body, a control method for a moving body, and a program, each having the following configuration: (1): A control device for a moving body according to one aspect of the present invention includes a recognition unit that recognizes a situation around the moving body and a user based on an image of the situation around the moving body, and a control unit that executes tracking control to move at least the moving body to follow the user based on the situation around the moving body, wherein the control unit sets a no-entry area for the moving body based on a matrix formed by a plurality of people recognized by the recognition unit, and suppresses the tracking control when the user enters the set no-entry area.

[0007] (2): In the above aspect (1), the control unit moves the moving body to an evacuation area when a predetermined time has elapsed since the user entered the no-entry area, or when the user moves away from the queue by a predetermined distance or more.

[0008] (3): In the above aspect (1), the control unit moves the moving body to an evacuation area when it is determined that a queue including the plurality of persons and the user is formed and that other persons are lined up behind the user.

[0009] (4): In the above aspect (1), a notification unit is further provided for notifying the user of predetermined information, and the control unit notifies the user, via the notification unit, that when the user enters a no-entry area of ​​the moving body, that operation control of the moving body will be executed in accordance with the type of the no-entry area.

[0010] (5): In the above aspect (1), the control unit resumes the tracking control when the user exits the no-entry area or when a call is received from the user.

[0011] (6): Another aspect of the present invention provides a method for controlling a moving body, in which a computer recognizes the situation around the moving body and a user based on an image capturing the situation around the moving body, performs tracking control for at least the moving body to move in a manner that follows the user based on the situation around the moving body, sets a no-entry area for the moving body based on a matrix formed by multiple people recognized from the situation around the moving body, and inhibits the tracking control when the user enters the set no-entry area.

[0012] (7): Another aspect of the present invention provides a program that causes a computer to recognize the situation around a moving body and a user based on an image capturing the situation around the moving body, executes tracking control for at least the moving body to move in a manner that follows the user based on the situation around the moving body, sets a no-entry area for the moving body based on a matrix formed by multiple people recognized from the situation around the moving body, and inhibits the tracking control when the user enters the set no-entry area.

[0013] According to the above aspects (1) to (7), the movement of the moving object following the user can be more appropriately controlled.

[0014] 1 is a diagram illustrating an example of the configuration of a mobile body system 1 including a mobile body M in a first embodiment. A diagram for explaining an example of a usage mode in which the mobile body M is made to follow a user U. A perspective view showing an example of the external configuration of the mobile body M. A diagram for explaining an example of the functional configuration of the mobile body M. A diagram for explaining the following control of the mobile body M in the first embodiment. A diagram for explaining the retreat control of the mobile body M in the first embodiment. A diagram for explaining operation control for resuming the following control in the first embodiment. A flowchart showing an example of processing executed by the control device 200 according to the first embodiment. A diagram for explaining operation control of the mobile body M in a second embodiment. A diagram for explaining the retreat control of the mobile body M in the second embodiment. A diagram for explaining operation control for resuming the following control in the second embodiment. A flowchart showing an example of processing executed by the control device 200 according to the second embodiment. A diagram for explaining operation control of the mobile body M in a third embodiment. A diagram for explaining the retreat control of the mobile body M in the third embodiment. A diagram for explaining operation control for resuming the following control in the third embodiment. A flowchart showing an example of processing executed by the control device 200 according to the third embodiment.

[0015] Hereinafter, embodiments of a mobile object control device, a mobile object control method, and a program according to the present invention will be described with reference to the drawings.

[0016] 1 is a diagram showing an example of the configuration of a mobile body system 1 including a mobile body M in a first embodiment. The mobile body system 1 includes, for example, a terminal device 2, a management device 10, an information providing device 20, and the mobile body M. These communicate with each other via, for example, a network NW. The network NW is, for example, any network such as a LAN (Local Area Network), a WAN (Wide Area Network), or an internet line.

[0017] The terminal device 2 is a computer device such as a smartphone, a tablet terminal, etc. The terminal device 2 is used by a user of the mobile system 1, and requests the management device 10 to use the mobile device M based on the user's operation, and obtains information from the management device 10 indicating that the use of the mobile device M is permitted.

[0018] [Management Device] The management device 10 manages the usage status and usage reservations of the mobile bodies M in the mobile body system 1. In response to a request obtained from the terminal device 2, the management device 10 sets usage authorization for the mobile bodies M that the user can use, and transmits information indicating that use of the set mobile body M is permitted to the terminal device 2 to provide the user with the information. In addition, the management device 10 generates and manages schedule information that associates, for example, pre-registered user identification information with the date and time of reservation for use of the mobile body M.

[0019] [Information Providing Device] The information providing device 20 provides the mobile body M with the location of the mobile body M, the area in which the mobile body M is moving, map information around the area, etc. The information providing device 20 may generate a route from the current location of the mobile body M to a destination in response to a request from the mobile body M, and provide the generated route to the mobile body M. The management device 10 and the information providing device 20 may be, for example, a server device, or may be a device configured by cloud computing including one or more information processing devices.

[0020] [Mobile Body] The mobile body M is, for example, a mobile body capable of autonomous movement. Autonomous movement means moving the mobile body M by executing one or both of speed control and turning control of the mobile body M without a driving operation by a user. Turning control includes, for example, changing the direction of the mobile body M by spinning or turning, and steering control in the case where the mobile body M has a steering wheel. The mobile body M is, for example, a vehicle, but may also include other mobile bodies capable of autonomous movement (for example, a walking robot, etc.). Vehicles include not only four-wheeled vehicles but also general vehicles that can move on three wheels, two wheels, etc. The mobile body M has a structure that can carry and transport an object, for example, luggage. The object may include a person, such as a user.

[0021] The mobile object M is used by a user based on, for example, a usage authority set by the management device 10. For example, the user loads luggage or the like onto the mobile object M and has it follow the user's movement or lead the user to a destination. The location information, usage status, usage reservation status, etc. of the mobile object M are managed by the management device 10. Furthermore, the mobile object M acquires information from the management device 10 and the information providing device 20 and controls its movement based on the usage authority and the provided information.

[0022] Here, the mobile body system 1 of the first embodiment may have a plurality of at least one of the terminal device 2, the management device 10, the information providing device 20, and the mobile body M. Furthermore, the mobile body system 1 of the embodiment may have the management device 10 and the information providing device 20 integrated together. Furthermore, the mobile body system 1 may be configured without the management device 10. In this case, at least some of the functions of the management device 10 are provided in the mobile body M, and the terminal device 2 communicates with the mobile body M via the network NW to manage usage rights and the like.

[0023] The mobile object M will be described in detail below. Note that the following description will mainly focus on a case where the mobile object M is used in a manner in which it moves by following the movement of a user. The same applies to the second and third embodiments described below. FIG. 2 is a diagram for explaining an example of a usage manner in which the mobile object M follows the user U. The mobile object M is located, for example, at a predetermined location in a facility or town. When the user U wishes to use the mobile object M, the user U operates the operation unit of the mobile object M to set a usage authority and start using the mobile object M. Alternatively, the user U may operate the terminal device 2 to send a usage request to the management device 10 so that the mobile object M can be used, and start using the mobile object M based on the usage authority set by the management device 10.

[0024] For example, when user U goes shopping and wants to reduce the burden of carrying purchased luggage, etc., he / she considers leaving the luggage in mobile body M, and operates terminal device 2 or the operation unit of mobile body M to set the usage authority of mobile body M and start usage. When starting usage, user authentication by mobile body M (for example, vein authentication using palm information of user U) is performed. If, as a result of the authentication, it is determined that the user is authorized to use the mobile body M (a user permitted to use the mobile body M), the mobile body M stores user U's luggage and autonomously follows the movement of user U. This allows user U to continue shopping or head to his / her next destination in a comfortable state without having to carry luggage.

[0025] The moving object M may not only move on roads (roadways or sidewalks) but also in indoor or outdoor facilities such as shopping centers, airports, parks, and theme parks, as well as on private property. In addition to the tracking control for following the user U as described above, the moving object M may also be capable of other operational controls, such as guidance control, evacuation control, and emergency control. Guidance control is, for example, control for guiding the user U to a destination specified by the user U, such as autonomously moving in front of the user U at a speed that matches the user U's movement to lead the user U. Furthermore, guidance control does not necessarily require the user U to set the destination. For example, the moving object M may predict the direction of movement of the user U based on the user U's orientation or a movement trajectory from a predetermined time ago, and autonomously move in front of the user U at a speed that matches the user U's movement. Evacuation control is, for example, control for moving to a predetermined evacuation area and waiting when the user U moves into a no-entry area that the moving object M cannot enter during the execution of tracking control or guidance control. Emergency control is, for example, control that moves autonomously to seek help from nearby people or facilities in order to help user U if something unusual happens to user U (for example, if user U collapses) while traveling with user U.

[0026] 3 is a perspective view showing an example of the external configuration of the moving body M. In the description of FIG. 3, the forward direction of the moving body M is the positive x direction, the backward direction of the moving body M is the negative x direction, the width direction of the moving body M is the left direction based on the positive x direction, the right direction is the negative y direction, and the height direction of the moving body M, which is a direction perpendicular to the x direction and y direction, is the positive z direction.

[0027] In the example of FIG. 3 , the mobile body M includes, for example, a base body 110, a door unit 112 provided on the base body 110, and wheels (first wheel 120, second wheel 130, and third wheel 140) attached to the base body 110. For example, a user U can open the openable door unit 112 to put luggage into a storage compartment provided on the base body 110 or take luggage out of the storage compartment. The first wheel 120 and the second wheel 130 are drive wheels that rotate by power from a motor or the like. The third wheel 140 is an auxiliary wheel (driven wheel). Note that the mobile body M may be movable using a configuration other than wheels, such as caterpillars.

[0028] A cylindrical support body 150 extending in the positive z direction is provided on the surface of the base body 110 facing the positive z direction. A camera 180 that captures images of the surroundings of the moving body M at a wide angle (e.g., 360 degrees) is provided on the end of the support body 150 facing the positive z direction. The position at which the camera 180 is provided may be any position different from the above. The camera 180 may include multiple cameras that capture images of a predetermined area. The moving body M shown in FIG. 3 is an example, and other components may be added, or some components (e.g., the door portion 112) may be absent.

[0029] [Functional Configuration of Mobile Object] Fig. 4 is a diagram showing an example of the functional configuration of the mobile object M. In addition to the configuration shown in Fig. 3, the mobile object M includes, for example, a first motor 122, a second motor 132, a battery 134, a braking device 136, a steering device 138, an operation unit 160, an information output unit 170, a communication unit 190, and a control device 200. The first motor 122 and the second motor 132 are operated by power supplied to the battery 134. The first motor 122 drives the first wheel 120, and the second motor 132 drives the second wheel 130. The first motor 122 may be an in-wheel motor provided in the wheel of the first wheel 120, and the second motor 132 may be an in-wheel motor provided in the wheel of the second wheel 130.

[0030] The brake device 136 outputs a brake torque to each wheel based on an instruction from the control device 200. The steering device 138 includes an electric motor. The electric motor applies a force to a rack and pinion mechanism based on an instruction from the control device 200, for example, to change the direction of the first wheel 120 or the second wheel 130, thereby changing the course of the moving object M.

[0031] The operation unit 160 accepts input operations by the user U. The operation unit 160 includes, for example, at least one of a touch panel, a switch, a key, etc. The operation unit 160 may also have a voice input unit (such as a microphone) that accepts input operations by the user U through voice input.

[0032] The information output unit 170 outputs predetermined information to the surroundings of the user U or the moving body M. The information output unit 170 includes, for example, a display unit and an audio output unit. The display unit is, for example, an LCD (Liquid Crystal Display) or an organic EL (Electro Luminescence) display. The display unit may also be configured as a touch panel integrated with the operation unit 160. The display unit displays images corresponding to various pieces of information in the embodiment. The display unit may also include a light-emitting unit that emits light of a predetermined color. The audio output unit outputs sounds corresponding to the movement of the moving body M or sounds corresponding to images output by the display unit. The information output unit 170 is, for example, an example of an alarm unit that notifies the surroundings of the user U or the moving body M of predetermined information.

[0033] The communication unit 190 is a communication interface for communicating with the terminal device 2, the management device 10, or the information providing device 20 via the network NW. The communication unit 190 may also communicate with other mobile objects.

[0034] The control device 200 is housed within the base 110. The control device 200 includes, for example, a position identification unit 202, an information processing unit 204, a recognition unit 206, a trajectory generation unit 210, a first control unit 212, a second control unit 214, and a storage unit 220. The position identification unit 202, the information processing unit 204, the recognition unit 206, the trajectory generation unit 210, the first control unit 212, and the second control unit 214 are realized by a hardware processor such as a CPU (Central Processing Unit) executing a program (software). Some or all of these components may be realized by hardware (including circuitry) such as an LSI (Large Scale Integration), an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or a GPU (Graphics Processing Unit), or may be realized by a combination of software and hardware. The program may be stored in advance in a storage device (a storage device with a non-transitory storage medium) such as a hard disk drive (HDD) or flash memory, or may be stored in a removable storage medium (a non-transitory storage medium) such as a DVD or CD-ROM, and installed by inserting the storage medium into a drive device. Some or all of the information processing unit 204, the trajectory generation unit 210, the first control unit 212, and the second control unit 214 are examples of a "control unit." Some or all of the functional configuration included in the control device 200 may be included in another device. For example, the other device and the moving body M may communicate with each other and cooperate to control the moving body M.

[0035] The storage unit 220 is realized by a storage device such as a HDD, flash memory, or RAM (Random Access Memory). The storage unit 220 stores control information 222, including a control program for controlling the behavior of the moving object M referenced by the first control unit 212 and the second control unit 214, and map information 224. The map information 224 is, for example, map information provided by the information providing device 20, such as the location of the moving object M, the area in which the moving object M moves, and the surrounding area. The map information 224 may include information such as the location of products, the location of stores, and floor maps of facilities such as shopping centers, art galleries, and museums, associated with location information on the map (e.g., latitude and longitude). The map information 224 may also include information regarding preset no-entry areas into which the moving object M cannot enter, information regarding evacuation areas for the moving object M, and location information of specific objects. A specific object is, for example, an object that is predicted to remain (stop) at a predetermined position or area for a predetermined period of time or longer in order for the user U to visually recognize it. The map information 224 may also include detailed map information such as the width, gradient, curvature, etc. of roads and passages. The storage unit 220 may also store characteristic information of the user U and characteristic information of a specific object (e.g., characteristic information based on shape, size, color, etc.), and information indicating a correspondence between the actions (gestures) of the user U and the behavior (motion control) of the moving object M. At least a portion of the information stored in the storage unit 220 may be updated as needed by the communication unit 190 communicating with another device such as the management device 10 or the information providing device 20.

[0036] The position identification unit 202 identifies the position of the mobile object M. The position identification unit 202 acquires position information of the mobile object M using a GPS (Global Positioning System) device (not shown) built into the mobile object M. The position information may be, for example, two-dimensional map coordinates or latitude and longitude information. The position identification unit 202 may estimate the position information of the mobile object M by communicating with a communication device located within a predetermined distance via the communication unit 190 using a short-range wireless communication method such as Bluetooth (registered trademark) and acquiring the position information of the communication device. The position identification unit 202 may also estimate the position of the mobile object M based on specific structures (landmarks, signs, signs, etc.) included in camera images captured by the camera 180. The position identification unit 202 may also estimate the position of the mobile object M simultaneously with creating an environmental map using a method such as so-called SLAM (Simultaneous Localization and Mapping) that uses camera images captured by the camera 180 or a sensor such as LIDAR.

[0037] The information processing unit 204 manages information acquired from, for example, the terminal device 2, the management device 10, or the information providing device 20. Furthermore, the information processing unit 204 transmits information to the terminal device 2, the management device 10, or the information providing device 20 based on information received by the operation unit 160, outputs information received from each device (e.g., control information 222, map information 224, characteristic information of the user U or a specific object) to each component of the control device 200, or stores it in the storage unit 220. Furthermore, the information processing unit 204 causes the information output unit 170 to output predetermined information. The predetermined information includes, for example, information such as an image, sound, or alarm according to the type of operation control being performed by the mobile object M, information indicating that the user U has been recognized, information indicating forward movement, reverse movement, or turning direction, etc.

[0038] The information processing unit 204 also performs processing to register characteristic information of the user U who uses the mobile object M and processing to authenticate the user U. For example, when registering characteristic information, the information processing unit 204 generates characteristic information regarding the face, body shape (height), hair color, skin color, clothing color, etc. from a facial image or a full-body image of the user U captured by the camera 180 before the user U starts using the mobile object M. The information processing unit 204 may also generate characteristic information regarding the posture and movement (walking movement) of the user U. The information processing unit 204 may also acquire voice data or an image of the palm of the user U to generate characteristic information regarding fingerprints, veins, and voice. The information processing unit 204 may also generate information regarding the movement of the mobile object M corresponding to the gesture of the user U and register the generated information in the storage unit 220.

[0039] Furthermore, when performing user authentication, the information processing unit 204 generates feature information from the voice acquired from the image microphone capturing an image of the user, references feature information previously stored in the storage unit 220 based on the generated feature information, and if matching feature information (feature information with a similarity equal to or greater than a threshold) is found, permits the user to use the mobile body M. Furthermore, if matching feature information is not found in the storage unit 220, the information processing unit 204 does not permit use and causes the information output unit 170 to output an error message or a message prompting the user to register feature information.

[0040] The recognition unit 206 recognizes the situation around the moving body M, for example, based on an image captured by the camera 180. For example, the recognition unit 206 recognizes the positions (distance from the moving body M and direction relative to the moving body M) and states such as speed and acceleration of objects around the moving body M. The objects include people such as pedestrians and traffic participants such as cyclists, obstacles present in facilities or on roads, specific objects, etc. The specific objects may include, for example, exhibits or objects displayed on display shelves, or may include a matrix formed by multiple objects (e.g., people). The recognition unit 206 may also recognize the characteristics of people or specific objects in the vicinity, or the direction of a person's face, etc.

[0041] The recognition unit 206 also recognizes, for example, a user U using the moving object M and tracks the recognized user U. For example, the recognition unit 206 performs a matching process between the user's characteristic information recognized from the camera image captured by the camera 180 and the characteristic information of the user U registered in advance when the user U uses the moving object M or the characteristic information of the user U provided by the terminal device 2 or the management device 10, thereby identifying the user U using the moving object M from people in the vicinity. The recognition unit 206 may also identify the user U using AI (artificial intelligence) technology such as machine learning. When tracking the moving user U, the recognition unit 206 sets a future predicted trajectory based on the user U's past movement trajectory, and further compares the predicted trajectories with those of other people in the vicinity to identify the person moving along a smoother trajectory as the user U.

[0042] The recognition unit 206 may also recognize a gesture made by the user U. Gestures include, for example, a gesture of pointing to a specific position in the vicinity, a gesture of calling the moving object M, and a gesture of switching the operation control (following control, guidance control, evacuation control, emergency control) performed by the moving object M. The recognition unit 206 may also recognize a specific movement of the user U (for example, the movement of the user U falling down) based on a camera image captured by the camera 180, and may recognize that something unusual has happened to the user U when the recognition unit 206 recognizes the specific movement.

[0043] The moving body M may be provided with a detection unit, such as a radar device or a LIDAR (Light Detection and Ranging) device, that is different from the camera 180. In this case, the recognition unit 206 recognizes the situation around the moving body M using the detection results of the radar device or LIDAR instead of (or in addition to) the image.

[0044] The trajectory generation unit 210 generates a target trajectory along which the moving body M should travel in the future, based on, for example, the position of the user U, surrounding objects, etc. For example, the trajectory generation unit 210 generates a target trajectory that allows the moving body M to move smoothly to a target point, in accordance with an action based on a preset operation control (for example, following control, guidance control, evacuation control, emergency control).

[0045] In the case of tracking control, the trajectory generation unit 210 generates a target trajectory with a target point that is within a predetermined distance range from the user U and behind the user U (not limited to directly behind the user U, but may be diagonally behind within a predetermined angle from the directly behind direction). In the case of guidance control, the trajectory generation unit 210 generates a target trajectory with a target point that is within a predetermined distance range from the user U and in front of the user U (not limited to directly in front of the user U, but may be diagonally forward within a predetermined angle from the front direction). In the case of evacuation control, the trajectory generation unit 210 generates a target trajectory with a predetermined evacuation area as the target point. In the case of emergency control, the trajectory generation unit 210 generates a trajectory for autonomous movement to seek help from nearby people or facilities.

[0046] Furthermore, the trajectory generation unit 210 generates a target trajectory corresponding to the behavior (or motion control) of the moving body M in response to the gesture, based on, for example, a predetermined correspondence between the gesture and the behavior of the user U, or generates a target trajectory (or corrects the generated target trajectory) for moving toward a destination while avoiding surrounding objects. For example, when the recognition unit 206 recognizes a gesture by the user U such that the user U beckons toward the moving body M, the trajectory generation unit 210 generates a target trajectory for the moving body M so that the distance between the moving body M and the user U becomes closer than a reference distance (for example, approximately 1 to 3 m) (for example, so that the distance between the moving body M and the user U becomes within 30 cm). Furthermore, when the motion control is switched in response to the gesture of the user U, the trajectory generation unit 210 generates a future trajectory corresponding to the switched motion control.

[0047] Furthermore, the trajectory generation unit 210 may generate a target trajectory along which the moving body should travel in the future, based on a destination set by the user U or the next destination of the user U estimated by the second control unit 214 described later. Note that the trajectory generation unit 210 may, for example, generate multiple trajectories according to the behavior of the moving body M, calculate the risk for each trajectory, and, if the total value of the calculated risks or the risk at each trajectory point meets a predetermined criterion (for example, if the total value is equal to or less than a first threshold value Th1 and the risk at each trajectory point is equal to or less than a second threshold value Th2), adopt the trajectory that satisfies the criterion as the target trajectory along which the moving body M should travel. For example, the risk tends to be higher the closer the distance between the trajectory (trajectory point on the trajectory) and an obstacle, and lower the closer the distance between the trajectory and the obstacle.

[0048] The first control unit 212 controls the motors (first motor 122, second motor 132), the braking device 136, and the steering device 138 so that the moving body M travels along the trajectory generated by the trajectory generating unit 210.

[0049] Based on the recognition result of the recognition unit 206, the second control unit 214 controls the moving object M so that the moving object M does not come into contact with surrounding objects and so that the distance between the user U and the moving object M is within a predetermined distance range. The predetermined distance range is a distance range between a predetermined shortest distance Dmin and a predetermined longest distance Dmax. The shortest distance Dmin and the maximum distance Dmax may be variable distances depending on, for example, the type of operation control (following control, guidance control, etc.) and surrounding conditions (shape of the moving path, degree of crowding), or may be fixed distances.

[0050] In addition, in the tracking control, the second control unit 214 determines whether the user U has moved (entered) a predetermined area where entry of the moving body M is prohibited (hereinafter referred to as the prohibited entry area), and if the user U has entered the prohibited entry area, performs operation control of the moving body according to the type of prohibited entry area.

[0051] [Movement Control of Moving Body] Next, a specific description will be given of the movement control of the moving body M according to the first embodiment. Note that, below, the movement control will be described in a situation where tracking control is executed so that the moving body M follows the user U.

[0052] FIG. 5 is a diagram illustrating tracking control of a moving object M in the first embodiment. The example in FIG. 5 illustrates a no-entry area AR11 for the moving object M, an evacuation area AR12 for the moving object M, and objects OB1 to OB3, such as walls. The no-entry area AR11 in the first embodiment is an area of ​​a fixed position. Examples of the no-entry area AR11 include areas where the movement path is narrow or where congestion of people or objects is expected, such as restrooms, reception areas, checkout areas (where cash registers are located), food court areas, and restaurants, as well as areas where movement of the moving object M is expected to be difficult, such as farms, fields, stairs, and uneven roads. The evacuation area AR12 may be an area that is expected not to interfere with the passage of nearby passersby (traffic participants such as pedestrians) (e.g., an area where the degree of traffic flow at a predetermined time is below a threshold), or may be an area set based on the positions of nearby objects (e.g., an area near a wall or a corner of a floor). The evacuation area AR12 may also be set at a position within a predetermined distance from the no-entry area AR11. The evacuation area AR12 may also be set at a position depending on the type of no-entry area AR11. For example, if the no-entry area AR11 is a restroom or a checkout area, the evacuation area AR12 is set near the exit (within a predetermined distance from the exit), and if the no-entry area AR11 is a food court, the evacuation area AR12 is set at a position outside the no-entry area AR11 but within a predetermined distance from the location of a table in the food court used by the user U.

[0053] In addition, the moving body M may refer to the map information 224 based on the position information of the moving body M identified by the position identification unit 202, and obtain information on nearby no-entry areas AR11 and evacuation areas AR12.

[0054] In the scene shown in FIG. 5 , the recognition unit 206 of the moving body M recognizes the surrounding situation and also recognizes the user U. The trajectory generation unit 210 generates a target trajectory for moving so as to follow the recognized user U from behind. The first control unit 212 moves the moving body M along the generated target trajectory. At this time, the first control unit 212 moves the moving body M at a speed MV corresponding to the speed UV of the user U so that the distance D1 between the moving body M and the user U is maintained within a predetermined distance range. When the recognition unit 206 recognizes the presence of other traffic participants such as pedestrians or objects such as obstacles in the vicinity, the second control unit 214 moves the moving body M while performing deceleration control, turning control, etc. so that the moving body M can move without coming into contact with the traffic participants or objects. Since the recognition unit 206 continuously recognizes the position of the user U (at a predetermined period), even if, for example, another pedestrian passes between the user U and the moving body M, or if the distance D1 between the user U and the moving body M falls outside the predetermined distance range due to movement control to avoid an obstacle, speed control and turning control are continuously performed so that the distance falls within the predetermined distance range again.

[0055] FIG. 6 is a diagram illustrating the evacuation control of the moving object M in the first embodiment. When the user U enters a no-entry area AR11 during the tracking control (which may be rephrased as the user U being present within the no-entry area AR11), the second control unit 214 suppresses the tracking control of the moving object M and executes the movement control of the moving object M according to the type of no-entry area. Suppressing the tracking control includes stopping the tracking control or reducing the degree of tracking control below a standard. Reducing the degree of tracking control below a standard includes, for example, slowing the speed MV of the moving object M below the speed UV of the user U (including stopping the moving object M) or increasing the distance D1 beyond the maximum distance Dmax within a predetermined distance range. Suppressing the tracking control may also include preventing the moving object M from entering the no-entry area AR11.

[0056] For example, the second control unit 214 may refer to the map information 224 using the position information of the moving body M identified by the position identification unit 202, and determine whether the user U has entered the prohibited area AR11 based on whether the position of the user U as seen from the moving body M is within the prohibited area AR11. Alternatively, the second control unit 214 may estimate the position of the user U based on the distance D1 between the moving body M and the user U and the position of the moving body M, and may use the estimated position of the user U to refer to the map information 224 to determine whether the user U has entered the prohibited area AR11. Alternatively, when the user U is moving while carrying the terminal device 2, the second control unit 214 may obtain position information of the terminal device 2 from the communication unit 190, and use the obtained position information of the terminal device 2 as the position information of the user U to determine whether the user U has entered the prohibited area AR11. In addition, the second control unit 214 may determine whether or not the user U has entered the no-entry area AR11 by inputting an image of the user U and the no-entry area AR11 captured by the camera 180 into a learned model previously stored in the memory unit 220 or the like.

[0057] Furthermore, the second control unit 214 may determine whether the user U has entered the no-entry area AR11 based on the surrounding conditions of the moving object M recognized by the recognition unit 206. For example, if the no-entry area AR11 is a toilet, the second control unit 214 recognizes a mark, character, or pictogram (graphic symbol) indicating that the area is a toilet from a camera image captured by the camera 180, and further recognizes from the camera image that the user U has entered the toilet through an entrance / exit. In addition, the second control unit 214 may acquire a final determination result by integrating the determination using the map information 224 described above and the determination using the recognition result of the surrounding conditions by the recognition unit 206.

[0058] The second control unit 214 also counts the time since the user U entered the no-entry area AR11 and determines whether the time since entry is equal to or longer than a predetermined time. The predetermined time may be a fixed time or a variable time depending on the type of no-entry area AR11. If the second control unit 214 determines that the time since entry is equal to or longer than the predetermined time, the second control unit 214 causes the trajectory generation unit 210 to generate a target trajectory for moving the moving object M to the evacuation area AR12 and causes the first control unit 212 to move the moving object M along the generated target trajectory. Furthermore, if the recognition unit 206 is no longer able to recognize the user U who entered the no-entry area AR11, the second control unit 214 may control the moving object M to move to the evacuation area AR12 without waiting for the predetermined time to elapse since entry. In the example of FIG. 6 , the user U is hidden by the object OB1 and cannot be recognized from the moving object M. Therefore, the second control unit 214 executes operation control (evacuation control) to evacuate the moving body M to the evacuate area AR12.

[0059] In the evacuation control, the trajectory generation unit 210 generates a target trajectory including a turning trajectory T11 that turns the front of the moving body M toward the evacuation area AR12, a movement trajectory K1 that moves the position of the moving body M into the evacuation area AR12, and a turning trajectory T12 that turns the front of the moving body M toward the entrance / exit side of the no-entry area AR11 (e.g., the position where the user U entered the no-entry area AR11, or the exit side if the entrance and exit are separate), as shown in FIG. 6 . Note that the trajectory generation unit 210 may generate a target trajectory that turns while moving along the movement trajectory K1. Then, the first control unit 212 can evacuate the moving body M to the evacuation area AR12 by moving the moving body M along the target trajectory generated by the trajectory generation unit 210.

[0060] Furthermore, after moving the moving object M to the evacuation area AR12, the second control unit 214 temporarily stops the moving object M and waits until the user U exits the no-entry area AR11. The recognition unit 206 may continue to recognize the situation around the moving object M and the user U while waiting. Furthermore, when the moving object M is in the evacuation area AR12, the information processing unit 204 may perform locking control to prevent the door 112 from opening. This can prevent theft of luggage stored in the storage unit when the user U is not nearby. Furthermore, while the moving object M is waiting in the evacuation area AR12, the information processing unit 204 may cause the information output unit 170 to output information indicating that the moving object M is waiting. This can notify those around the moving object M that the moving object M is being used by the user U (in other words, that it cannot be used by other users).

[0061] Furthermore, the second control unit 214 determines whether the user U has left the restricted area AR11 based on the recognition result by the recognition unit 206. For example, the second control unit 214 determines that the user U has left the restricted area AR11 when the recognition unit 206 recognizes the user U and recognizes that the user U's position is outside the restricted area AR11. Furthermore, while the user U is recognized by the recognition unit 206, the second control unit 214 may refer to the map information 224 using position information acquired from the terminal device 2 as the position information of the user U to determine whether the user U has left the restricted area AR11. Furthermore, the second control unit 214 may estimate the position information of the user U from the distance between the position of the moving object M and the recognized user U, and use the estimated position information of the user U to refer to the map information 224 to determine whether the user U has left the restricted area.

[0062] When it is determined that the user U has left the no-entry area AR11, the second control unit 214 resumes the tracking control for tracking the user U. FIG. 7 is a diagram for explaining the operation control for resuming the tracking control in the first embodiment. When resuming the tracking control, the second control unit 214 causes the trajectory generation unit 210 to generate a target trajectory K2 for approaching the user U until the distance to the user U is within a predetermined distance range, and controls the first control unit 212 to move the moving object M at a speed MV along the generated target trajectory K2. This eliminates the need for the user U to again instruct the moving object M to follow, and allows the tracking control to be resumed at an appropriate timing.

[0063] [Processing flow according to the first embodiment] Fig. 8 is a flowchart showing an example of processing executed by the control device 200 according to the first embodiment. Note that in the example of Fig. 8, the explanation will mainly focus on the following control processing and the escape control processing among the processing executed by the control device 200. In other words, it is assumed that the user U has set up operation control in advance for the moving body M so that the moving body M performs following control. The same applies to the processing flows according to the second and third embodiments described below.

[0064] 8 , the recognition unit 206 recognizes the situation around the moving object M and the user U (step S100). Next, the first control unit 212 executes tracking control to track the user U (step S110). Next, the second control unit 214 determines whether the user U has entered the no-entry area AR11 of the moving object M (step S120). If it is determined that the user U has entered the no-entry area AR11 of the moving object M, the second control unit 214 suppresses the ongoing tracking control (step S130).

[0065] Next, the second control unit 214 determines whether the time since the user U entered the no-entry area AR11 is equal to or longer than a predetermined time (step S140). Note that in the process of step S140, instead of determining whether the time since the user U entered the no-entry area AR11 is equal to or longer than a predetermined time, it may also determine whether the user U who entered the no-entry area AR11 can no longer be recognized by the recognition unit 206.

[0066] If it is determined in step S140 that the time since the user U entered the no-entry area AR11 is not equal to or longer than the predetermined time, the process returns to step S130, and the suppression of tracking control continues. If it is determined that the time since the user U entered the no-entry area AR11 is equal to or longer than the predetermined time, the second control unit 214 moves the moving object M to the evacuation area AR12 (step S150). After moving the moving object M to the evacuation area AR12, the second control unit 214 temporarily stops the moving object M and makes it wait (step S160).

[0067] Next, the second control unit 214 determines whether the user U has exited the no-entry area AR11 (step S170). If it is determined that the user U has not exited the no-entry area AR11, the process returns to step S160 and continues in a standby state. If it is determined that the user U has exited the no-entry area, the second control unit 214 resumes tracking control of the moving object M (step S180). This ends the process of this flowchart. If it is determined in the process of step S120 that the user U has not entered the no-entry area AR11 of the moving object M, the process of this flowchart ends.

[0068] In addition, in the processing of step S140 described above, if it is determined that user U has exited the no-entry area AR11 before a predetermined time has passed since user U entered the no-entry area AR11, the second control unit 214 may resume tracking control from the current position of the moving body M.

[0069] According to the first embodiment described above, the control device 200 for the moving body M is equipped with a recognition unit 206 that recognizes the situation around the moving body M and the user U based on an image of the situation around the moving body M, and a control unit (trajectory generation unit 210, first control unit 212, second control unit 214) that performs tracking control so that at least the moving body M moves to follow the user U based on the surrounding situation.When the user U enters a no-entry area for the moving body M, the control unit performs operation control of the moving body M according to the type of no-entry area, thereby more appropriately controlling the operation of the moving body M following the user U.

[0070] For example, according to the first embodiment, the tracking control of the moving object M is suppressed when the user U enters the no-entry area AR11, and further, when a predetermined condition is satisfied, the moving object M is moved to the escape area AR12, thereby making it possible to move the moving object M without interfering with the passage of surrounding passersby. Also, according to the first embodiment, the tracking control is resumed from the position of the escape area AR12 when the user U exits the no-entry area AR11, making it possible to resume the tracking control at a more appropriate timing. Furthermore, since the user U does not need to instruct the moving object M to follow again, the convenience of the moving object M can be further improved.

[0071] Second Embodiment Next, a second embodiment of the control device 200 for the moving body M will be described. In the second embodiment, the no-entry area for the moving body M is not a specific location or area as in the first embodiment, but is variably set based on the surrounding circumstances. More specifically, in the second embodiment, when a queue of multiple people in the vicinity is formed, a no-entry area for the moving body M is set based on the queue, and when a user U enters the set no-entry area, tracking control is suppressed, etc. Note that the second embodiment can apply the same configurations as the moving body system 1, the moving body M, etc. as in the first embodiment. Therefore, the following description will use the same configuration as the moving body M described in the first embodiment, and will mainly focus on the differences from the first embodiment.

[0072] 9 is a diagram illustrating the operation control of a moving object M in the second embodiment. The example of FIG. 9 shows a scene in which multiple people P1 to P3 are present near a store entrance SH1. In the example of FIG. 9, the moving object M moves behind the user U, following the user U so as to maintain a distance D1 between the moving object M and the user U.

[0073] 9 , the recognition unit 206 of the moving body M recognizes the surrounding situation and also recognizes the user U. The trajectory generation unit 210 generates a target trajectory for moving from behind the recognized user U so as to follow the user U. The first control unit 212 moves the moving body M along the generated target trajectory. At this time, the second control unit 214 moves the moving body M at a speed MV corresponding to the speed UV of the user U so that the distance D1 from the user U is maintained within a predetermined distance range.

[0074] In the second embodiment, the second control unit 214 estimates that a queue is formed by multiple people when a predetermined number (e.g., three people) or more of people facing the same direction (including a predetermined tolerance range) are present within a predetermined range based on the surrounding situation recognized by the recognition unit 206. Note that the second control unit 214 considers the same direction to be, for example, the direction toward a specific location such as the store entrance / exit SH1, a vending machine, or a reception desk. The predetermined range may be the distance when multiple people are lined up vertically, the distance when they are lined up horizontally, or an area including both vertical and horizontal directions.

[0075] Furthermore, in addition to (or instead of) the above-mentioned conditions, the second control unit 214 may estimate that a queue is formed when the moving speed of multiple people in a predetermined time period is less than a predetermined distance (including when they stop). Note that the above-mentioned queue estimation process is performed only when the recognition unit 206 recognizes multiple people in the traveling direction of the user U, and the estimation process does not need to be performed when multiple people are recognized in a direction other than the traveling direction of the user U.

[0076] When the second control unit 214 estimates that a queue has been formed, it sets a no-entry area for the moving body M based on the area of ​​the queue, and when the user U moves into the set no-entry area, it suppresses the tracking control of the moving body M.

[0077] 9, the moving body M performs tracking control so that the distance D1 between the moving body M and the user U is maintained within a predetermined distance range. In this case, the recognition unit 206 recognizes the position and orientation of each of multiple people P1 to P3 present in the traveling direction of the user U, for example, from a camera image captured by the camera 180. The second control unit 214 determines whether or not a queue is formed by the people P1 to P3, based on the positions and orientations of the people P1 to P3 recognized by the recognition unit 206.

[0078] For example, when the persons P1 to P3 recognized by the recognition unit 206 are facing the same direction and that direction is a predetermined direction (e.g., a store entrance / exit SH1), the second control unit 214 determines whether the range including all of the persons P1 to P3 (e.g., the vertical distance D2 (line-up direction) and / or the horizontal distance W2) is within a predetermined range. The range including all of the persons P1 to P3 may be acquired from a camera image, or instead of (or in addition to) a camera image, it may be acquired from detection results by another detection unit such as a radar device or LIDAR. When it is determined that the range including all of the persons P1 to P3 is within the predetermined range, the second control unit 214 sets the range including all of the persons P1 to P3 (which may be a range including a predetermined margin) as a matrix area MA, and further sets a no-entry area AR21 including the matrix area MA.

[0079] For example, since it is expected that other people will line up behind the queue formed by people P1 to P3, the second control unit 214 sets, based on the orientations of people P1 to P3, an area including a predetermined area behind person P3, who is estimated to be at the end of the queue, as the no-entry area AR21. Therefore, in the second embodiment, the queue area MA is variable depending on the number and positions of people forming the queue, and therefore the no-entry area AR21 is also variable as the queue area MA changes. The above-mentioned predetermined area may be a fixed area or a variable area depending on the size of the queue area MA, the shape of the surrounding passageway, etc.

[0080] When the user U enters the set no-entry area AR21, the second control unit 214 suppresses the tracking control of the moving object M. This makes it possible to prevent the moving object M from entering an area near a queue where a crowd is expected to form, thereby preventing further congestion.

[0081] Furthermore, when the time that the user U has been in the no-entry area AR21 is equal to or longer than a predetermined time, the second control unit 214 moves the moving object M to the evacuation area AR22. Fig. 10 is a diagram for explaining evacuation control of the moving object M in the second embodiment. In the example of Fig. 10, the second control unit 214 sets the evacuation area AR22 from the surrounding situation based on the position of the user U recognized by the recognition unit 206, and executes movement control to move the moving object M to the set evacuation area AR22 when the time that the user U has been in the no-entry area AR21 is equal to or longer than a predetermined time.

[0082] If the evacuation area AR22 is registered in advance in the map information 224, that area is used. If the evacuation area AR22 is not registered in the map information 224, a location with fewer pedestrians (e.g., a space near a wall) or a location where the mobile unit M can easily remain stable in a stopped state is set as the evacuation area AR22 based on the surrounding conditions recognized by the recognition unit 206. The evacuation area AR22 may also be set based on the position and size of the set no-entry area AR21. In this case, the evacuation area AR22 is set at a location where the waiting mobile unit M is predicted to be visible to users U waiting in line within the no-entry area AR21. The evacuation area AR22 may also be set at a location within a predetermined distance from the store entrance / exit SH1.

[0083] After setting the evacuation area AR22, the second control unit 214 causes the trajectory generation unit 210 to generate a target trajectory for moving the moving object M into the evacuation area AR22, and causes the first control unit 212 to execute evacuation control so that the moving object M moves along the generated trajectory. For example, as shown in FIG. 10 , the trajectory generation unit 210 generates a target trajectory including a turning trajectory T21 for directing the moving object M toward the evacuation area AR22, a movement trajectory K3 for moving the position of the moving object M into the evacuation area AR22, and a turning trajectory T22 for directing the front of the moving object M toward the no-entry area AR21, the queue area MA, or the store entrance / exit SH1 after moving into the evacuation area AR22. Note that the trajectory generation unit 210 may generate a target trajectory for turning while moving along the movement trajectory K1, and the first control unit 212 can evacuate the moving object M to the evacuation area AR22 by moving the moving object M along the target trajectory generated by the trajectory generation unit 210. This allows the moving body M to wait in a position (safer position) that does not obstruct the passage of other pedestrians until the user U leaves the no-entry area AR21.

[0084] In the second embodiment, instead of (or in addition to) when the user U remains in the no-entry area AR21 for the moving body M for a predetermined period of time or longer, the second control unit 214 may perform evacuation control to move the moving body M to the evacuation area AR22 when no movement of the user U deviating from the matrix (matrix area MA) including the user U is recognized for a predetermined period of time or longer.

[0085] Furthermore, the second control unit 214 may move the moving body M to the evacuation area AR22 when a queue including multiple people and the user U is formed and it is estimated that another person is waiting behind the user U. In the example of FIG. 10 , the second control unit 214 estimates that a person P4 is waiting behind the user U and executes evacuation control to the evacuation area AR22. Note that in the example of FIG. 10 , the second control unit 214 estimates that a queue including the people P1 to P3, the user U, and the person P4 is formed, and sets the queue area MA and the no-entry area AR21 based on the estimated queue. In this way, when another person is waiting behind the user U, it can be determined that the user U is more likely to be waiting in a queue. Therefore, by moving the moving body M to the evacuation area AR22 in this case, it is possible to evacuate the moving body M to the evacuation area AR22 at a more appropriate time.

[0086] In addition, after moving to the evacuation area AR22, the moving body M temporarily stops and waits, and while waiting, the recognition unit 206 continues to recognize the surrounding situation and the user U.

[0087] In the second embodiment, when the user U leaves the no-entry area AR21, the second control unit 214 resumes the tracking control that causes the moving body M to follow the user U. FIG. 11 is a diagram for explaining the operation control that resumes the tracking control in the second embodiment. In the example of FIG. 11, a matrix is ​​formed by people P4 to P6, and a no-entry area AR21 that includes a matrix area MA based on the formed matrix is ​​set. In this case, when the second control unit 214 determines, based on the position of the user U, that the user U has left the no-entry area AR21, the second control unit 214 resumes the tracking control that follows the user U.

[0088] When resuming the following control, the second control unit 214 causes the trajectory generation unit 210 to generate a target trajectory K4 for bringing the distance to the user U within a predetermined distance range (a turning trajectory T23 for turning the front of the moving body M toward the user U may be generated). Furthermore, the second control unit 214 controls the first control unit 212 to move the moving body M at a speed MV along the target trajectory generated by the trajectory generation unit 210. This eliminates the need for the user U to again instruct the moving body M to follow, and allows the following control to be resumed at an appropriate timing.

[0089] Furthermore, instead of (or in addition to) determining that the user U has left the no-entry area AR21, the second control unit 214 may determine that the user U has left the queue and resume the tracking control when the user U moves away from the queue area MA by a predetermined distance or more. The second control unit 214 may also resume the tracking control when a call is received from the user U. For example, the second control unit 214 may resume the tracking control when the user U receives an instruction to call the moving object M via the terminal device 2 or when a specific voice (e.g., "Come on!" or "Let's go!") from the user U. This allows the tracking control to be resumed at an appropriate time in response to an instruction from the user, even if no queue has formed or no no-entry area has been set when the user U exits the store entrance / exit SH1.

[0090] [Processing Flow According to the Second Embodiment] FIG. 12 is a flowchart illustrating an example of processing executed by the control device 200 according to the second embodiment. In the example of FIG. 12, the recognition unit 206 recognizes the surrounding situation of the moving object M and the user U (step S200). Next, the first control unit 212 executes tracking control to track the user U (step S210). Next, the second control unit 214 estimates a matrix formed by multiple people based on the surrounding situation recognized by the recognition unit 206 (step S220). Next, the second control unit 214 determines whether the user U has entered a no-entry area AR21 of the moving object M, which is set based on the estimated matrix (matrix area) (step S230). If it is determined that the user U has entered the no-entry area AR21, the second control unit 214 suppresses tracking control (step S240).

[0091] Next, the second control unit 214 determines whether the time since the user U entered the no-entry area AR21 is equal to or longer than a predetermined time (step S250). In the process of step S250, the determination condition may be whether the user U has moved a predetermined distance or more from the queue area MA, or whether another person has lined up behind the user U. If it is determined that the time since entering the no-entry area AR21 is not equal to or longer than the predetermined time, the second control unit 214 returns to the process of step S240 and continues to suppress the tracking control. Furthermore, if it is determined that the time since entering the no-entry area AR21 is equal to or longer than the predetermined time, the second control unit 214 sets an evacuation area AR22 for the moving object M to escape, and moves the moving object M to the set evacuation area AR22 (step S260). After moving the moving object M to the evacuation area AR22, the second control unit 214 temporarily stops the moving object M and causes it to wait (step S270).

[0092] Next, the second control unit 214 determines whether a call for the moving body M from the user U has been received (step S280). Note that in the processing of step S280, instead of determining whether a call for the moving body M from the user U has been received, it may also determine whether the user U has left the no-entry area AR21. If it is determined in the processing of step S280 that a call for the moving body M from the user U has not been received, the process returns to the processing of step S270 and continues in the standby state. Furthermore, if it is determined that a call from the user U has been received, the second control unit 214 resumes tracking control of the moving body M (step S290). This ends the processing of this flowchart. If it is determined in the processing of step S230 that the user U has not entered the no-entry area for the moving body M, the processing of this flowchart ends.

[0093] In addition, in the processing of step S250 described above, if it is determined that user U has exited the no-entry area AR21 before a predetermined time has passed since user U entered the no-entry area AR21 of moving body M, the second control unit 214 may resume tracking control from the current position of moving body M.

[0094] According to the second embodiment described above, the control device 200 for the moving body M is equipped with a recognition unit 206 that recognizes the situation around the moving body M and the user U based on an image capturing the situation around the moving body M, and a control unit that performs tracking control so that at least the moving body M moves to follow the user U based on the surrounding situation.The control unit sets a no-entry area for the moving body M based on a matrix formed by multiple people recognized by the recognition unit 206, and suppresses tracking control when a user enters the set no-entry area, thereby more appropriately controlling the operation of the moving body M following the user U.

[0095] For example, according to the second embodiment, by recognizing a queue formed by multiple people and suppressing tracking control or executing evacuation control depending on whether a user U has entered a no-entry area AR21 of a moving body M that includes the queue, it is possible to set a variable no-entry area that is not a fixed position area, thereby realizing appropriate operation control of the moving body M according to the surrounding situation (e.g., a crowd).

[0096] Third Embodiment Next, a third embodiment of the control device 200 for the moving body M will be described. In the third embodiment, when a user U to be tracked is heading toward a specific object, a no-entry area for the moving body M is set near the specific object, and when the user U enters the set no-entry area, tracking control of the moving body M is suppressed. In the third embodiment, the specific object is, for example, an object that the user U or another person is expected to stay (stop) in for a predetermined period of time or more in order to visually recognize. Visual recognition may include concepts such as appreciating, viewing, watching, observing, or checking to perform a predetermined operation or procedure. In the third embodiment, the specific object includes at least one of exhibits, display shelves displaying merchandise, display devices, cash registers, vending machines, and reception desks. Exhibits include specific items such as sculptures, paintings, photographs, and products. Display devices include, for example, display devices used for digital signage and large screens used for public viewing.

[0097] Fig. 13 is a diagram for explaining the operation control of the moving object M in the third embodiment. In the example of Fig. 13, the moving object M moves by following the user U from behind so as to maintain a distance D1 from the user U. In addition, in the example of Fig. 13, objects OB4 and OB5 are assumed to be display shelves (an example of a specific object).

[0098] In the third embodiment, the second control unit 214 determines whether a specific object exists among the surrounding objects recognized by the recognition unit 206. For example, the second control unit 214 compares feature information obtained from an object included in a camera image captured by the camera 180 with feature information of a specific object previously stored in the storage unit 220, and determines that the object included in the camera image is a specific object if they match (if the similarity is equal to or greater than a threshold value). If they do not match, the second control unit 214 determines that the object is not a specific object. The above-described comparison process may be performed by the recognition unit 206. The determination of whether the specific object exists may also be performed when an object is recognized in the traveling direction of the user U (within a predetermined angle range based on the forward direction). In the example of FIG. 13 , objects OB4 and OB5 are determined to be specific objects by the above-described determination process.

[0099] If it is determined that a specific object is present, the second control unit 214 sets a no-entry area according to the position of the specific object. For example, the second control unit 214 sets an area near the specific object (within a predetermined distance from the specific object) where the user U is expected to remain for a predetermined period of time or more as the no-entry area.

[0100] In the example of FIG. 13 , the user U is moving toward the specific object OB4, so the second control unit 214 sets the area in front of the object OB4 as the no-entry area AR31, as viewed from the moving body M. The second control unit 214 sets the width W31 of the no-entry area AR31 based on the width WOB of the object OB4. For example, the second control unit 214 sets the width W31 to be equal to or greater than the width WOB of the object OB4. Furthermore, the second control unit 214 may set the width (distance) W32 in the front direction of the object OB4 to a fixed distance or a variable distance depending on the type of object OB4. For example, if the specific object is a display shelf, the user U is likely to stay near the specific object to pick up and look at (browse) the displayed products, so the width W32 is set small. However, if the specific object is a sculpture, painting, or the like, the user U is likely to view the specific object from a slightly greater distance, so the width W32 is set larger than when the specific object is a display shelf.

[0101] When the user U moves into the set no-entry area AR31, the second control unit 214 suppresses tracking control of the moving object M. Furthermore, when the user U continues to view the object OB4 for a predetermined period of time or longer, the second control unit 214 moves the moving object M to the escape area AR32. For example, the second control unit 214 determines that the user U is viewing the object OB4 when the user U is within a predetermined distance from the object OB4 and the orientation of the user U's face recognized by the recognition unit 206 (a direction within a predetermined angle range based on the forward orientation of the face) is facing the direction in which the object OB4 is located. Furthermore, the second control unit 214 may include the presence of the user U within the no-entry area AR31 as a condition for moving to the escape area AR32.

[0102] The evacuation area AR32 may be determined by referring to map information 224 based on the location information of the moving body M, and the evacuation area closest to the moving body M may be obtained, or the evacuation area AR32 may be set in a location (e.g., near a wall) that does not obstruct the passage of nearby passersby based on the surrounding conditions.

[0103] 14 is a diagram illustrating the evacuation control of the moving object M in the third embodiment. When moving the moving object M to the evacuation area AR32, the trajectory generation unit 210 generates a target trajectory including a turning trajectory T31 that turns the front of the moving object M toward the evacuation area AR32, a movement trajectory K5 that moves the position of the moving object M into the evacuation area AR32, and a turning trajectory T32 that turns the front of the moving object M toward the no-entry area AR31 after moving to the evacuation area AR32. Note that the trajectory generation unit 210 may generate a target trajectory that turns while moving along the movement trajectory K5. Then, the first control unit 212 can evacuate the moving object M to the evacuation area AR12 by moving the moving object M along the target trajectory generated by the trajectory generation unit 210.

[0104] After moving the moving object M to the evacuation area AR32, the second control unit 214 causes the moving object M to temporarily stop and wait. Note that the recognition unit 206 continues to perform recognition processing of the situation around the moving object M and the user U even during standby.

[0105] In the third embodiment, the second control unit 214 estimates the location or direction of the user U next while suppressing tracking control. For example, the second control unit 214 estimates object OB5, which is a specific object closest to object OB4 among the surrounding objects recognized by the recognition unit 206, as the location of the user U next. Furthermore, the second control unit 214 may estimate, for example, the direction of object OB5 as seen from the position of object OB4 as the direction of the user U next. Note that, if information indicating the next route (e.g., a sign or a notice) is recognized based on the surrounding conditions recognized by the recognition unit 206, the second control unit 214 may estimate the location and direction of the user U next from the information.

[0106] Furthermore, the second control unit 214 may refer to the map information 224 based on the position information of the moving body M, and estimate the location and direction to which the user U will move next from the positional relationship between the object OB4 and other specific objects. Note that, for example, if there are multiple exhibits (one example of specific objects) on a floor that the user U has not viewed (when it is determined that the user U has viewed all the exhibits on the floor), the location of a floor exit or a means of transportation (e.g., an elevator or escalator) for moving to another floor, a checkout area (where a cash register is located), or the like may be estimated as the next location to which the user U will move.

[0107] Then, when the user U moves to the estimated next moving location or moving direction, the second control unit 214 resumes the tracking control of the moving object M. Furthermore, the second control unit 214 may include in the above-mentioned conditions for resuming the tracking control that the user U has left the no-entry area. Furthermore, the conditions for resuming the tracking control may include a case where the face of the user U is facing toward the next object OB5.

[0108] 15 is a diagram illustrating operation control for resuming tracking control in the third embodiment. The example in FIG. 15 also shows a movement trajectory TR of the user U after the user U enters the no-entry area. The second control unit 214 recognizes the behavior of the user U within the evacuation area AR32 (movement trajectory, facial direction, etc.), and determines whether the condition for resuming tracking control is met based on the recognition result.

[0109] In the third embodiment, the second control unit 214 controls so that tracking control is not resumed, for example, when the user U leaves the no-entry area AR31, if the location or direction of the user U's departure differs from the estimated next location or direction of movement. For example, if the resumption condition is that the user U's movement direction is the direction in which the next object OB5 is located, tracking control is not resumed even if the user U leaves in a direction other than the direction in which the object OB5 is located (for example, if the user U leaves to the right of the no-entry area AR31 (when moving through area AR# in FIG. 15)), as shown by the movement trajectory TR in FIG. 15 .

[0110] This allows for more appropriate movement control of the moving body M, since tracking control is not resumed until it is determined that the user U is moving to the next location, even when the user U moves to view the object OB4 from a different angle or position, or when the user U moves to view displayed products in order.

[0111] When resuming the tracking control, the second control unit 214 causes the trajectory generation unit 210 to generate a target trajectory K6 for bringing the distance to the user U within a predetermined distance range (the turning trajectories T33 and T34 may be generated based on the forward direction of the moving body M and the direction in which the user U is located). The second control unit 214 also controls the first control unit 212 to move the moving body M at a speed MV along the target trajectory generated by the trajectory generation unit 210. This eliminates the need for the user U to instruct the moving body M to follow again, and allows the tracking control to be resumed at an appropriate timing for moving toward the next destination. Furthermore, when the user U is moving toward the object OB5, the second control unit 214 sets a no-entry area for the object OB5 and performs the same processing as described above.

[0112] [Processing flow according to the third embodiment] Fig. 16 is a flowchart showing an example of processing executed by the control device 200 according to the third embodiment. Note that in the example of Fig. 16, similar to the first and second embodiments, the explanation will mainly focus on the follow-up control processing and the evacuation control processing.

[0113] In the example of FIG. 16 , the recognition unit 206 recognizes the situation around the moving object M and the user U (step S300). Next, the first control unit 212 executes tracking control to track the user U (step S310). Next, the second control unit 214 determines whether a specific object is present in the moving direction of the user U (step S320). If it is determined that a specific object is present, the second control unit 214 sets a no-entry area AR31 for the moving object M corresponding to the specific object (step S330). Next, the second control unit 214 determines whether the user U has entered the no-entry area AR31 for the moving object M (step S340). If it is determined that the user U has entered the no-entry area AR31 for the moving object M, the second control unit 214 suppresses the tracking control (step S350).

[0114] Next, the second control unit 214 determines whether the user U is viewing a specific object within the no-entry area AR31 (step S360). If it is determined that the user U is not viewing a specific object, the process returns to step S130 and continues to suppress tracking control. If it is determined that the user U is viewing a specific object, the second control unit 214 sets an evacuation area AR32 and moves the moving object M to the evacuation area AR32 (step S370). After moving the moving object M to the evacuation area AR32, the second control unit 214 estimates the next location (destination location) to which the user U will move based on the surrounding conditions (step S380). Note that the second control unit 214 may estimate the next location before or during the movement of the moving object M to the evacuation area AR32. Next, the second control unit 214 temporarily stops the moving object M and causes it to wait (step S390).

[0115] Next, the second control unit 214 determines whether the user U has left the no-entry area and moved to the next location (step S400). If it is determined that the user U has not left the no-entry area AR31 and moved to the next location, the process returns to step S390 and continues in the standby state. If it is determined that the user U has left the no-entry area AR31 and moved to the next location, the second control unit 214 resumes tracking control of the moving object M (step S410). This ends the process of this flowchart. If it is determined in step S320 that no specific object exists in the moving direction of the user U, or if it is determined in step S340 that the user U has not moved into the no-entry area of ​​the moving object M, the process of this flowchart ends.

[0116] According to the third embodiment described above, the control device 200 for the moving body M is equipped with a recognition unit 206 that recognizes the situation around the moving body M and the user U based on an image capturing the situation around the moving body M, and a control unit that performs tracking control so that at least the moving body M moves to follow the user U based on the surrounding situation.When a specific object is present around the moving body M, the control unit sets a no-entry area for the moving body according to the position of the specific object, and when the user U enters the set no-entry area and the user is viewing the specific object, the control unit suppresses the tracking control, thereby more appropriately controlling the operation of the moving body following the user.

[0117] Furthermore, according to the third embodiment, the location or direction to which the user U will move next is estimated, and tracking control is resumed when the user U leaves the no-entry area AR31 of the moving body M and moves to the estimated next location or direction, thereby suppressing unnecessary movement control and saving power. Furthermore, according to the third embodiment, even when the user U moves within the no-entry area, tracking control can be resumed at an appropriate timing.

[0118] Note that the no-entry areas in the first to third embodiments described above are of different types (different acquisition methods, setting methods, etc.). Therefore, as described above, in the first to third embodiments, when a user U enters a no-entry area for a moving object M, different operation control of the moving object M is executed according to the type of no-entry area. Furthermore, in the first to third embodiments, the second control unit 214 sets an evacuation area according to the type (setting mode) of each no-entry area.

[0119] Each of the first to third embodiments described above may be combined with at least a part of the other embodiments. For example, in the first or third embodiment, the moving object M waiting in the evacuation area may accept a call from the user U and resume tracking control, as shown in the second embodiment.

[0120] Furthermore, in the first to third embodiments, the tracking control is suppressed when the user U enters a prohibited area of ​​the moving body M. However, the moving body M may suppress the tracking control when it is predicted that the user U will enter the prohibited area within a predetermined time (before the entry), and may enter an evacuation preparation mode at this stage to set the evacuation area. Furthermore, when it is predicted that the user U will enter the prohibited area within a predetermined time, the moving body M may gradually move away from the user U and start moving toward the evacuation area. This allows the user U to easily understand that the moving body M is heading toward the evacuation area before entering the prohibited area, and allows the user U to enter the prohibited area with peace of mind.

[0121] Furthermore, in the first to third embodiments, when the user U enters a no-entry area for the moving body M, the information processing unit 204 may notify the user U via the information output unit 170 that operation control of the moving body M will be executed in accordance with the type of no-entry area. This can alleviate the anxiety of the user U that the moving body M will no longer follow (that evacuation control of the moving body M will be started).

[0122] Furthermore, in the first to third embodiments described above, the shapes of the no-entry areas and the evacuation areas are not limited to rectangular shapes such as a rectangle, and may be other shapes such as a circle, an ellipse, an L-shape, a T-shape, etc. The shapes of the no-entry areas and the evacuation areas may be changed as appropriate depending on, for example, nearby obstacles, the floor layout, etc.

[0123] In the first to third embodiments, the no-entry area or evacuation area for the moving body M may be set by the user U through an instruction from the terminal device 2 or the operation unit 160 or by a gesture operation. This allows the moving body M to evacuate to a timing or area desired by the user U.

[0124] In addition, in the first to third embodiments, when setting no-entry areas or evacuation areas, the second control unit 214 may use image recognition technology (e.g., a trained model using machine learning) to detect crowds or empty spaces.

[0125] Furthermore, the moving body M in the first to third embodiments may be one that can be ridden by a user U. In this case, the moving body M is provided with driving controls such as a steering wheel, an accelerator pedal, and a brake pedal so that the user U can drive it. Furthermore, at least a part of the speed control and turning control of the moving body M may be controlled by the user's operation (driving operation).

[0126] Furthermore, in the first to third embodiments, switching from tracking control to evacuation control and resumption of tracking control have been mainly described, but the present invention may also be applied to a case where the moving body M is performing guidance control in addition to (or instead of) tracking control. For example, in a situation where the moving body M is moving ahead of the user U under guidance control and the user U behind enters a no-entry area for the moving body M, such as a toilet, the moving body M moves to the evacuation area and waits there, and resumes guidance control when the user U exits the no-entry area. This allows for more appropriate control of the operation of the moving body guiding the user.

[0127] The above-described embodiment can be expressed as follows: A control device for a moving body, comprising: a storage medium for storing computer-readable instructions; and a processor connected to the storage medium, wherein the processor executes the computer-readable instructions to: recognize a surrounding situation of the moving body and a user based on an image of the surrounding situation of the moving body, perform tracking control to move at least the moving body to follow the user based on the surrounding situation, set a no-entry area for the moving body based on a matrix formed by a plurality of people recognized from the surrounding situation, and suppress the tracking control when the user enters the set no-entry area.

[0128] The above describes the form for carrying out the present invention using an embodiment, but the present invention is not limited to such an embodiment, and various modifications and substitutions can be made within the scope that does not deviate from the gist of the present invention.

[0129] REFERENCE SIGNS LIST 1 Mobile system 2 Terminal device 10 Management device 20 Information providing device 110 Base 160 Operation unit 170 Information output unit 180 Camera 190 Communication unit 200 Control device 202 Position identification unit 204 Information processing unit 206 Recognition unit 210 Trajectory generation unit 212 First control unit 214 Second control unit 220 Storage unit M Mobile body

Claims

1. A control device for a moving body, comprising: a recognition unit that recognizes the situation around a moving body and a user based on an image capturing the situation around the moving body; and a control unit that executes tracking control so that at least the moving body moves to follow the user based on the situation around the moving body, wherein the control unit sets a no-entry area for the moving body based on a matrix formed by a plurality of people recognized by the recognition unit, and suppresses the tracking control when the user enters the set no-entry area.

2. The control device for a moving body as described in claim 1, wherein the control unit moves the moving body to an evacuation area when a predetermined time has elapsed since the user entered the no-entry area, or when the user moves away from the queue by a predetermined distance or more.

3. The control device for a moving body as described in claim 1, wherein the control unit moves the moving body to an evacuation area when it is determined that a queue including the plurality of persons and the user is formed and that other persons are lined up behind the user.

4. A control device for a moving body as described in claim 1, further comprising an alarm unit for notifying the user of specified information, wherein the control unit, when the user enters a no-entry area of ​​the moving body, notifies the user via the alarm unit that operational control of the moving body will be executed in accordance with the type of the no-entry area.

5. The control device for a moving body according to claim 1, wherein the control unit resumes the tracking control when the user exits the no-entry area or when a call is received from the user.

6. A method for controlling a moving object, in which a computer recognizes the surrounding situation of the moving object and a user based on an image capturing the surrounding situation of the moving object, performs tracking control for at least the moving object to move in a manner to follow the user based on the surrounding situation, sets a no-entry area for the moving object based on a matrix formed by a plurality of people recognized from the surrounding situation, and suppresses the tracking control when the user enters the set no-entry area.

7. A program which causes a computer to recognize the situation around a moving object and a user based on an image capturing the situation around the moving object, execute a tracking control for moving at least the moving object to follow the user based on the situation around the moving object, set a no-entry area for the moving object based on a matrix formed by a plurality of people recognized from the situation around the moving object, and inhibit the tracking control when the user enters the set no-entry area.

Images