Mobile body control device, mobile body control system, mobile body control method, and program

Abstract

A mobile body control device according to an embodiment of the present invention comprises: an acquisition unit that acquires detection results of a plurality of sensors that detect conditions around a mobile body; a self-position estimation unit that estimates the self-position of the mobile body on the basis of a detection result obtained from at least one of the plurality of sensors and acquired by the acquisition unit; and a movement control unit that performs, on the basis of position information estimated by the self-position estimation unit, movement control on the mobile body so as to move at a position associated with a movement mode set by a user. The self-position estimation unit estimates the self-position of the mobile body on the basis of a detection result of a sensor associated with the movement mode among the plurality of sensors.

Description

Mobile body control device, mobile body control system, mobile body control method, and program

[0001] The present invention relates to a mobile body control device, a mobile body control system, a mobile body control method, and a program.

[0002] In recent years, efforts have been actively made to provide access to a sustainable transportation system that takes into account people in vulnerable positions among traffic participants. Toward this realization, research and development focused on further improving traffic safety and convenience through research and development related to autonomous driving technology have been carried out. In this regard, conventionally, there is a technique for estimating the self-position of a mobile body based on the result of comparing a plurality of captured images taken in a predetermined direction at different positions with a reference image captured in advance (see, for example, Patent Document 1). Further, conventionally, there is a technique for estimating the position of a mobile body using information acquired by a sensor unit and obtaining a plurality of position estimators that obtain the reliability regarding the estimation result, and obtaining the self-position of the mobile body according to the estimation result corresponding to the highest reliability among the plurality of reliabilities respectively obtained by the plurality of position estimators (see, for example, Patent Document 2).

[0003] International Publication No. 2019 / 073795 Japanese Patent Application Laid-Open No. 2021-18638

[0004] By the way, in autonomous driving technology, there has been a problem that the processing load increases when estimating the self-position using a wide range of images around the mobile body. Also, when the mobile body follows or leads (guides) a user, there is a possibility that the surrounding situation cannot be grasped depending on the position of the user. Therefore, there has been a problem that there is a possibility that appropriate self-position estimation cannot be performed according to the movement situation of the mobile body.

[0005] One of the objects of the present application is to provide a mobile body control device, a mobile body control system, a mobile body control method, and a program that can more appropriately estimate the self-position according to the movement situation of the mobile body in order to solve the above problems. And, by extension, it contributes to the development of a sustainable transportation system.

[0006] The control device for a mobile body, the mobile body control system, the method for controlling a mobile body, and the program according to this invention employ the following configuration: (1) A control device for a mobile body according to one aspect of this invention comprises: an acquisition unit that acquires detection results from a plurality of sensors that detect the surrounding conditions of a mobile body; a self-position estimation unit that estimates the self-position of the mobile body based on the detection result obtained from at least one of the plurality of sensors acquired by the acquisition unit; and a movement control unit that controls the movement of the mobile body to move at a position corresponding to a movement mode set by the user, based on the position information estimated by the self-position estimation unit, wherein the self-position estimation unit estimates the self-position of the mobile body based on the detection result of the sensor among the plurality of sensors that corresponds to the movement mode.

[0007] (2) In the embodiment of (1) above, the movement mode includes at least a follow mode that follows the user, a lead mode that leads the user, and a parallel mode that moves alongside the user.

[0008] (3) In the embodiment of (2) above, the plurality of sensors include a first sensor that detects the conditions of a region including the front of the moving body, a second sensor that detects the conditions of a region including the rear of the moving body, a third sensor that detects the conditions of a region including the left side of the moving body, and a fourth sensor that detects the conditions of a region including the right side of the moving body, wherein the self-position estimation unit estimates the self-position of the moving body based on the detection result of the second sensor when the movement mode is the follow mode, estimates the self-position of the moving body based on the detection result of the first sensor when the movement mode is the lead mode, and estimates the self-position of the moving body based on the detection result of at least one of the first sensor and the second sensor when the movement mode is the parallel movement mode.

[0009] (4) In the embodiment of (2) above, the plurality of sensors include a first sensor that detects the conditions of a region including the front of the moving body and a second sensor that detects the conditions of a region including the rear of the moving body, and the self-position estimation unit estimates the self-position of the moving body based on the detection result of the second sensor when the movement mode is the follow mode, estimates the self-position of the moving body based on the detection result of the first sensor when the movement mode is the lead mode, and estimates the self-position of the moving body based on the detection result of at least one of the first sensor and the second sensor when the movement mode is the parallel mode.

[0010] (5) In the embodiment of (2) above, the plurality of sensors include a third sensor that detects the conditions of a region including the left side of the moving body and a fourth sensor that detects the conditions of a region including the right side of the moving body, and the self-position estimation unit estimates the self-position of the moving body based on the detection result from the sensor of the third sensor and the fourth sensor that does not detect the presence of the user when the movement mode is parallel movement mode.

[0011] (6) In the embodiment of (1) above, the self-position estimation unit does not estimate the self-position of the moving body using the detection results of the sensors if there is a sensor among the plurality of sensors that detects a predetermined number or more of dynamic objects.

[0012] (7) In the embodiment of (1) above, the self-position estimation unit, when switching sensors used to estimate the self-position, retains the final position before the switch and associates the self-position before and after the switch with the retained final position as the initial position after the switch.

[0013] (8) A mobile body control system according to another aspect of the present invention comprises: an acquisition unit that acquires detection results from a plurality of sensors that detect the surrounding conditions of a mobile body; a self-position estimation unit that estimates the self-position of the mobile body based on detection results obtained from at least one of the plurality of sensors acquired by the acquisition unit; and a movement control unit that controls the movement of the mobile body to move at a position corresponding to a movement mode set by the user, based on the position information estimated by the self-position estimation unit, wherein the self-position estimation unit estimates the self-position of the mobile body based on the detection results of the sensor among the plurality of sensors that is corresponding to the movement mode.

[0014] (9) A method for controlling a moving body according to another aspect of the present invention is a method for controlling a moving body in which a computer acquires detection results from a plurality of sensors that detect the conditions around the moving body, estimates the self-position of the moving body based on the detection results obtained from at least one of the acquired plurality of sensors, controls the movement of the moving body to move to a position corresponding to a movement mode set by the user based on the estimated self-position information of the moving body, and estimates the self-position of the moving body based on the detection results of the sensor among the plurality of sensors that corresponds to the movement mode.

[0015] (10): A program according to another aspect of the present invention is a program which causes a computer to acquire detection results from a plurality of sensors that detect the surrounding conditions of a moving object, to estimate the self-position of the moving object based on the detection results obtained from at least one of the acquired plurality of sensors, to perform movement control of the moving object to move to a position corresponding to a movement mode set by the user based on the estimated self-position information of the moving object, and to estimate the self-position of the moving object based on the detection results of the sensor among the plurality of sensors that corresponds to the movement mode.

[0016] According to the embodiments described in (1) to (10) above, the movement of a mobile body that is followed by multiple people can be controlled more appropriately.

[0017] This figure shows an example of the configuration of a mobile body control system 1 including a mobile body M in an embodiment. This is a perspective view showing an example of the external configuration of the mobile body M. This figure shows an example of the functional configuration of the mobile body M. This figure illustrates the target trajectory of the mobile body M in follow mode. This figure illustrates the target trajectory of the mobile body M in lead mode. This figure illustrates the target trajectory of the mobile body M in parallel mode. This figure illustrates the function of the self-position estimation unit. This is a flowchart showing an example of processing performed by the control device 200 of the mobile body M in an embodiment.

[0018] Hereinafter, embodiments of the control device for a mobile body, a mobile body control system, a mobile body control method, and a program of the present invention will be described with reference to the drawings.

[0019] [System Configuration] Figure 1 shows an example of the configuration of a mobile control system 1 including a mobile body M in an embodiment. The mobile control system 1 includes, for example, a terminal device 2, a management device 10, an information providing device 20, and a mobile body M. These communicate via a network NW or the like. The network NW is any network such as a LAN (Local Area Network), WAN (Wide Area Network), or Internet connection.

[0020] [Terminal device] Terminal device 2 is a computer device such as a smartphone or a tablet. Terminal device 2 is used by a user of the mobile control system 1 and requests the use of the mobile device M from the management device 10 based on the user's operation, and obtains information from the management device 10 indicating that the use of the mobile device M has been permitted.

[0021] [Management Device] The management device 10 manages the usage status and usage reservations of mobile devices M within the mobile device control system 1. In response to requests received from the terminal device 2, the management device 10 sets usage rights for mobile devices M available to the user and provides the user with information indicating that the use of the set mobile device M has been permitted by sending it to the terminal device 2. The management device 10 also generates and manages schedule information, for example, which associates pre-registered user identification information with the date and time of the mobile device M usage reservation.

[0022] [Information Provisioning Device] The information provisioning device 20 provides the mobile object M and the terminal device 2 with information such as the location of the mobile object M, the area in which the mobile object M is moving, and map information of the area surrounding the area. The information provisioning device 20 may also generate a route from the current location of the mobile object M to its destination in response to a request from the mobile object M, and provide the generated route to the mobile object M. The management device 10 and the information provisioning device 20 may be, for example, a server device, or a device configured by cloud computing consisting of one or more information processing devices.

[0023] [Mobile Entity] The mobile entity M is, for example, a mobile entity capable of autonomous movement. Autonomous movement means moving the mobile entity M by performing either or both of the following: speed control or turning control, without driver operation by the user. Turning control includes, for example, changing the orientation of the mobile entity M by rotation or turning, or steering control if it has a steering wheel. The mobile entity M is, for example, a vehicle, but may also include other mobile entities capable of autonomous movement (e.g., walking robots). Vehicles include not only four-wheeled vehicles, but also all vehicles capable of movement with three wheels, two wheels, etc. The mobile entity M has a structure that can carry and transport objects such as luggage. The above objects may include people such as users.

[0024] The mobile vehicle M is used by users based on usage rights set by the management device 10, for example. For example, a user can load luggage or other items onto the mobile vehicle M and have it follow the user's movements, lead the user to a destination, or travel alongside the user, according to the movement mode instructed by the user. The location information, usage status, and usage reservation status of the mobile vehicle M are managed by the management device 10. The mobile vehicle M also acquires information from the management device 10 and the information providing device 20 and performs movement control based on usage rights and the provided information.

[0025] The mobile control system 1 of this embodiment may have multiple terminal devices 2, a management device 10, an information providing device 20, and a mobile body M, at least one of these. Alternatively, the mobile control system 1 of this embodiment may have the management device 10 and the information providing device 20 integrated into one unit. Furthermore, the mobile control system 1 may not have a management device 10. In this case, at least a portion of the functions of the management device 10 are provided on the mobile body M, and the terminal device 2 communicates with the mobile body M via a network NW to manage user rights and the like.

[0026] [External Configuration of the Mobile Body] Figure 2 is a perspective view showing an example of the external configuration of the mobile body M. In the explanation of Figure 2, the forward direction of the mobile body M is the positive x direction, the backward direction of the mobile body M is the negative x direction, the width direction of the mobile body M is the positive y direction to the left and the negative y direction to the right, and the height direction of the mobile body M, which is perpendicular to the x and y directions, is the positive z direction.

[0027] In the example shown in Figure 2, the mobile body M comprises, for example, a base 110, a door 112 provided on the base 110, and wheels (first wheel 120, second wheel 130, and third wheel 140) mounted on the base 110. For example, a user can open the openable and closable door 112 to put luggage into a storage compartment provided on the base 110, or to take luggage out of the storage compartment. The first wheel 120 and the second wheel 130 are drive wheels and rotate by power from a motor or the like. The third wheel 140 is an auxiliary wheel (driven wheel). The mobile body M may also be movable using a configuration other than wheels, such as an endless track.

[0028] A cylindrical support 150 extending in the positive z direction is provided on the surface of the base body 110 in the positive z direction. A camera 180 that captures a wide-angle image of the area around the mobile body M is provided at the end of the support 150 in the positive z direction. For example, the camera 180 may consist of multiple cameras that capture images of the front, rear, and sides of the mobile body M, respectively. The position where the camera 180 is provided may be any position other than that described above. In addition, an information output unit 170 that outputs predetermined information to the user or the area around the mobile body M is provided on the front surface of the base body 110. Note that the mobile body M shown in Figure 2 is just an example, and other configurations may be added, or some configurations (for example, the door portion 112) may be omitted, and furthermore, the size, shape, arrangement position, etc. of each configuration are not limited to the example in Figure 2.

[0029] [Functional Configuration of the Mobile Unit] Figure 3 is a diagram showing an example of the functional configuration of the mobile unit M. In addition to the configuration shown in Figure 2, the mobile unit M includes, for example, a first motor 122, a second motor 132, a battery 134, a braking device 136, a steering device 138, an operating unit 160, a communication unit 190, and a control device 200. The first motor 122 and the second motor 132 are powered by electricity supplied to the battery 134. The first motor 122 drives the first wheel 120, and the second motor 132 drives the second wheel 130. The first motor 122 is an in-wheel motor provided on the wheel of the first wheel 120, and the second motor 132 may be an in-wheel motor provided on the wheel of the second wheel 130.

[0030] The braking device 136 outputs brake torque to each wheel based on instructions from the control device 200. The steering device 138 includes an electric motor. The electric motor, for example, applies force to the rack and pinion mechanism based on instructions from the control device 200 to change the direction of the first wheel 120 or the second wheel 130, thereby changing the course of the moving body M.

[0031] The operation unit 160 accepts user input operations. The operation unit 160 includes, for example, at least one of a touch panel, a switch, a key, etc. The operation unit 160 may also have a voice input unit (microphone, etc.) that accepts user input operations via voice input or accepts the voices of people in the surrounding area.

[0032] The communication unit 190 is a communication interface for communicating with the terminal device 2, the management device 10, or the information providing device 20 via the network NW. The communication unit 190 may also communicate with other mobile devices.

[0033] Furthermore, the information output unit 170 shown in Figure 3 includes, for example, a display unit 172 and an audio output unit 174. The display unit 172 is, for example, an LCD (Liquid Crystal Display) or an organic EL (Electro Luminescence) display. The display unit 172 may also be integrated with the operation unit 160 as a touch panel. The display unit 172 displays images corresponding to various types of information in the embodiment. The display unit 172 may also include a light-emitting unit such as an LED (Light Emitting Diode) that emits a predetermined color. The audio output unit 174 outputs sounds corresponding to the operation of the mobile body M and sounds corresponding to the images output by the display unit 172. The information output unit 170 is, for example, an example of a notification unit for notifying predetermined information around the mobile body M.

[0034] Furthermore, the camera 180 shown in Figure 3 includes a first camera 182 that images an area including the front of the moving object M, a second camera 184 that images an area including the rear of the moving object M, a third camera 186 that images an area including the left side of the moving object M, and a fourth camera 188 that images an area including the right side of the moving object M. The camera 180 is an example of a sensor (detection unit) that detects the surrounding conditions of the moving object M. The first camera 182 is an example of a "first sensor", the second camera 184 is an example of a "second sensor", the third camera 186 is an example of a "third sensor", and the fourth camera 188 is an example of a "fourth sensor". Note that the moving object M may also be equipped with sensors other than the camera 180, such as a radar device or LIDAR (Light Detection and Ranging).

[0035] The control device 200 is housed within the base body 110. The control device 200 includes, for example, an acquisition unit 202, a self-position estimation unit 204, an information processing unit 206, a recognition unit 208, a trajectory generation unit 210, a first control unit 212, a second control unit 214, and a storage unit 220. The acquisition unit 202, the self-position estimation unit 204, the information processing unit 206, the recognition unit 208, the trajectory generation unit 210, the first control unit 212, and the second control unit 214 are realized, for example, by a hardware processor such as a CPU (Central Processing Unit) executing a program (software). Some or all of these components may be implemented by hardware (including circuitry) such as LSI (Large Scale Integration), ASIC (Application Specific Integrated Circuit), FPGA (Field Programmable Gate Array), GPU (Graphics Processing Unit), and SOC (System On Chip), or by the cooperation of software and hardware. The program may be stored in advance in a storage device such as an HDD (Hard Disk Drive) or flash memory (a storage device equipped with a non-transient storage medium), or it may be stored in a removable storage medium such as a DVD or CD-ROM (a non-transient storage medium) and installed when the storage medium is mounted on a drive device. Some or all of the first control unit 212 and the second control unit 214 are examples of a "driving control unit". Some or all of the functional configuration included in the control device 200 may be included in other devices. For example, the mobile body M and other devices may communicate and cooperate to control the mobile body M.

[0036] The storage unit 220 is implemented by a storage device such as an HDD, flash memory, or RAM (Random Access Memory). The storage unit 220 stores control information 222, which includes a control program for controlling the operation of the mobile body M (for example, operation in a movement mode) that is referenced by the first control unit 212 and the second control unit 214, and map information 224. The map information 224 is, for example, map information such as the location of the mobile body M provided by the information providing device 20, the area in which the mobile body M moves, and the area surrounding the area. The map information 224 may include information such as the location of a store, or floor maps of facilities such as shopping centers, art museums, and museums, associated with location information on a map (for example, latitude and longitude). The map information 224 may also include detailed map information such as the width, gradient, and curvature of roads and passages. Furthermore, the map information 224 may include information on feature points corresponding to the road shape or the edges of structures (static obstacles) installed in the surrounding area.

[0037] Furthermore, the memory unit 220 may store user characteristic information, characteristic information of specific objects (for example, characteristic information based on shape, size, color, etc.), and information indicating the correspondence between user actions (gestures) and the motion control of the mobile body M. At least a portion of the information stored in the memory unit 220 may be updated from time to time by the communication unit 190 communicating with other devices such as the management device 10 or the information providing device 20.

[0038] The acquisition unit 202 acquires detection results from multiple cameras 180 that capture images of the surrounding environment of the mobile body M. For example, the acquisition unit 202 acquires images (camera images) captured by the first camera 182, the second camera 184, the third camera 186, and the fourth camera 188. The acquisition unit 202 also acquires information obtained from the operation unit 160 and the communication unit 190. For example, the acquisition unit 202 acquires information regarding the movement mode specified by the user from the operation unit 160. The movement modes include at least a follow mode in which the mobile body moves following the user, a lead mode in which the mobile body moves ahead of the user toward a destination, and a parallel mode in which the mobile body moves alongside the user. The movement modes may also include a standby mode in which the mobile body takes refuge (stands) at a location specified by the user, and an emergency mode in which specific control is performed in the event of an emergency for the user.

[0039] The self-position estimation unit 204 estimates the position (self-position) of the moving object M. For example, the self-position estimation unit 204 estimates the self-position of the moving object M based on a camera image (an example of a detection result) obtained from at least one of the multiple cameras acquired by the acquisition unit 202. Details of the functions of the self-position estimation unit 204 will be described later.

[0040] The information processing unit 206 manages information acquired from, for example, the terminal device 2, the management device 10, or the information providing device 20. The information processing unit 206 also transmits information to the terminal device 2, the management device 10, and the information providing device 20 based on the information received by the operation unit 160, and outputs information received from each device (for example, control information 222, map information 224, user and specific object characteristic information) to each component of the control device 200 or stores it in the storage unit 220.

[0041] Furthermore, the information processing unit 206 causes the information output unit 170 to output predetermined information. This predetermined information includes, for example, images (including color images and patterns), sounds, and alarms corresponding to the type of motion control being performed by the mobile body M, information indicating that the user has been recognized, information regarding the mode of movement, and information indicating the direction of forward, reverse, and turning. The output images, sounds, and other information may be stored in the storage unit 220, or they may be transmitted from the management device 10 or the information providing device 20.

[0042] Furthermore, the information processing unit 206 performs processes such as registering characteristic information of users using the mobile device M and authenticating the user. For example, when registering user characteristic information, the information processing unit 206 generates characteristic information such as face and body shape (back shape), hair color, skin color, and clothing color from the user's face image and full-body image captured by the camera 180 before the user starts using the mobile device M. The information processing unit 206 may also generate characteristic information about the user's posture and movements (walking motion). The information processing unit 206 may also acquire the user's voice data and palm image to generate characteristic information about fingerprints, veins, and voice. The information processing unit 206 may also generate information about the operation of the mobile device M corresponding to the user's gestures and register the generated information in the storage unit 220.

[0043] Furthermore, when performing user authentication, the information processing unit 206 generates feature information from the audio acquired from the image microphone capturing the user, and based on the generated feature information, it refers to the feature information previously stored in the storage unit 220. If matching feature information (feature information with a similarity of a threshold or higher) exists, it permits the user to use the mobile device M. If no matching feature information exists in the storage unit 220, the information processing unit 206 does not permit use and instead outputs an error message or a message prompting the registration of feature information to the information output unit 170.

[0044] The recognition unit 208 recognizes the situation around the moving body M based on, for example, a camera image captured by the camera 180. For example, the recognition unit 208 recognizes the position of an object around the moving body M (distance from the moving body M and direction with respect to the moving body M), and states such as speed and acceleration. The object includes a person such as a pedestrian, a traffic participant such as a bicycle, an obstacle existing in a facility or on a road, and the like. Further, the recognition unit 208 may recognize feature information for each person in the vicinity, or recognize the face orientation, moving speed, moving direction, etc. of each person. The feature information is, for example, information that can be obtained from the analysis result of the camera image, and specifically includes information such as the body shape and height of a person, feature information of the face, color information, information related to clothing such as clothes and hats, and the like. Further, the feature information may include information related to feature amounts. The feature amount is, for example, an amount of information for distinguishing from other persons, and the larger the number of distinguishable features, the larger the value.

[0045] Further, the recognition unit 208 may distinguish and recognize a moving dynamic object (for example, an obstacle such as a traffic participant) and a static object without movement (for example, an obstacle such as a structure) existing around the moving body M based on the camera image captured by the camera 180.

[0046] Further, the recognition unit 208, for example, recognizes a user who uses the moving body M and tracks (analyzes) the recognized user. For example, the recognition unit 208 performs matching processing between the feature information of the user recognized from the camera image captured by the camera 180 and the feature information of the user registered in advance when the user uses the moving body M or the feature information of the user provided by the terminal device 2 or the management device 10 to identify the user who uses the moving body M from the persons existing in the vicinity. Further, the recognition unit 208 may identify the user using AI (Artificial Intelligence) technology such as machine learning. Further, when tracking a moving user, the recognition unit 208, for example, sets a future predicted trajectory from the past movement trajectory of the user, and further compares it with the predicted trajectories for other persons existing in the vicinity to identify the person moving on a smoother trajectory as the user.

[0047] Further, the recognition unit 208 may recognize a gesture made by the user. Examples of gestures include a gesture pointing to a specific position in the vicinity, a gesture for calling the moving body M, a gesture for switching the movement mode executed by the moving body M, and the like. Also, the recognition unit 208 may recognize a specific action of the user (for example, an action of the user falling) based on the camera image captured by the camera 180, and may recognize that something abnormal has occurred to the user when recognizing the specific action.

[0048] When the moving body M is provided with sensors such as a radar device or LIDAR, the recognition unit 208 recognizes the situation around the moving body M using the detection results of the radar device or LIDAR instead of (or in addition to) the image.

[0049] The trajectory generation unit 210 generates a target trajectory along which the moving body M should travel in the future based on, for example, the position of the user and surrounding objects. For example, the trajectory generation unit 210 generates a target trajectory so that the moving body M can move smoothly to the target point according to the operation control (for example, following control, leading control, parallel running control, avoidance control, emergency control) corresponding to the movement mode instructed by the user.

[0050] FIG. 4 is a diagram for explaining the target trajectory of the moving body M in the following mode. In the example of FIG. 4, it is assumed that the user U is moving in the X-axis direction in the figure at a speed VU. For example, in the case of the following mode of following the user U, the trajectory generation unit 210 generates a target trajectory K1 with the distance (offset distance) D1 from the user U within a predetermined distance range and the rear of the user U as the target point. Also, the trajectory generation unit 210 determines a speed VM corresponding to the speed VU of the user U so as to maintain the state where the distance D1 from the user U is within the predetermined distance range. In the case of the following mode, the trajectory generation unit 210 is not limited to moving the moving body M directly behind the user U, and may generate a target trajectory K1 with a diagonal rearward direction within a predetermined angle with respect to the directly rearward direction as the target point.

[0051] Figure 5 is a diagram illustrating the target trajectory of the mobile body M in lead mode. In lead mode, the trajectory generation unit 210 generates a target trajectory K2 with the target point being in front of the user U (not limited to directly in front of the user U, but also including an oblique forward area within a predetermined angle relative to the forward direction) where the distance D2 from the user U is within a predetermined distance range. The trajectory generation unit 210 also determines the speed VM according to the speed VU of the user U so as to maintain the state where the distance D2 from the user U is within a predetermined distance range.

[0052] Figure 6 is a diagram illustrating the target trajectory of the moving object M in parallel mode. In parallel mode, the trajectory generation unit 210 generates a target trajectory K3 with the target point being within a predetermined distance range from the user U and to the side of the user U (not limited to directly beside the user U, but may also be diagonally forward or backward within a predetermined angle relative to the lateral direction). The trajectory generation unit 210 also determines the speed VM according to the speed VU of the user U so as to maintain the state in which the distance D3 from the user U is within a predetermined distance range.

[0053] In addition, in evacuation mode, the trajectory generation unit 210 generates a target trajectory with a pre-set evacuation area as the target point. In emergency mode, the trajectory generation unit 210 generates a trajectory for autonomous movement to request assistance from nearby people or facilities. Operational control corresponding to each of these movement modes is performed based on information stored in, for example, control information 222.

[0054] Furthermore, the trajectory generation unit 210 generates a target trajectory corresponding to the motion control of the mobile body M in response to a gesture, or generates a target trajectory for moving towards a destination while avoiding surrounding objects, based on the correspondence between the gestures of the user U stored in the memory unit 220 and the motion control, for example. For example, if the recognition unit 208 recognizes a gesture by the user U beckoning to the mobile body M, the trajectory generation unit 210 generates a target trajectory for the mobile body M so that the distance between the mobile body M and the user U is closer than a reference distance (for example, about 1 to 3 [m]) (for example, so that the distance between the mobile body M and the user U is within 30 [cm]). Also, if the movement mode (motion control) is switched in response to the gestures or voice of the user U, the trajectory generation unit 210 generates a future trajectory corresponding to the switched movement mode.

[0055] Furthermore, the trajectory generation unit 210 may generate a target trajectory that the mobile body M should travel in the future, based on a destination set by the user U. The trajectory generation unit 210 may, for example, generate multiple trajectories according to the movement mode of the mobile body M, determine the risk for each trajectory, and if the sum of the calculated risks or the risk at each trajectory point meets a predetermined criterion (for example, if the sum is less than or equal to the first threshold Th1 and the risk at each trajectory point is less than or equal to the second threshold Th2), then the trajectory that meets the criterion may be adopted as the target trajectory for the mobile body M to travel. Risk tends to be higher the smaller the distance to the obstacle relative to the trajectory (trajectory point), and lower the larger the distance to the obstacle relative to the trajectory.

[0056] Returning to Figure 3, the first control unit 212 and the second control unit 214 control the movement of the mobile body M so that it moves to a position corresponding to the movement mode set by the user, based on the position information (self-position information) estimated by the self-position estimation unit 204. For example, the first control unit 212 controls the motors (first motor 122, second motor 132), the brake device 136, and the steering device 138 so that the mobile body M travels along the target trajectory generated by the trajectory generation unit 210.

[0057] The second control unit 214 controls the mobile body M when it travels along the generated target trajectory, based on the recognition result of the recognition unit 208, so that the mobile body M does not come into contact with surrounding objects and the distance between the user U and the mobile body M (for example, the distances D1 to D3 described above) according to the mode of movement is within a predetermined distance range. The predetermined distance range is the distance range between the preset shortest distance Dmin and longest distance Dmax. The shortest distance Dmin and longest distance Dmax may be variable distances or fixed distances, for example, depending on the type of mode of movement and surrounding conditions (shape of the travel path, degree of crowding).

[0058] [Self-Position Estimation Unit] Next, the function of the self-position estimation unit 204 will be explained. Figure 7 is a diagram illustrating the function of the self-position estimation unit. In the example of Figure 7, the behavior of the moving body M during operation control in follow mode for a user U moving at a speed VU is shown in a simplified manner. Also, in the example of Figure 7, the shooting ranges of the first camera 182, second camera 184, third camera 186, and fourth camera 188 provided on the moving body M are shown in a simplified manner.

[0059] For example, the self-position estimation unit 204 performs known image analysis processing (e.g., edge extraction, feature extraction, pattern matching, etc.) on the image captured by the camera 180, and extracts feature points around the moving object M based on the analysis results. Feature points are, for example, feature points of objects in real space included in the image (e.g., traffic signals, road signs, traffic participants, buildings, road structures, etc.). Objects may also include road markings and stop lines that are recognizable from the image and drawn on the road. For example, the self-position estimation unit 204 extracts a sequence of points on the edges of objects included in the image as feature points (a group of feature points).

[0060] Furthermore, the self-position estimation unit 204 may extract feature points using a pre-trained model that has been trained to output the edges of objects contained in an image as a point cloud when an image is input. This pre-trained model may be stored in the memory unit 220 in advance, or it may be acquired from an external device via the communication unit 190 mounted on the mobile body M. The self-position estimation unit 204 may also extract feature points using, for example, the Visual SLAM (Simultaneous Localization and Mapping) method, which is a technique for grasping the self-position in three dimensions from an image. The method for extracting feature points from an image is not limited to the above example, and other known methods may be used. The "×" marks in Figure 7 indicate feature points FP acquired from the camera image. Based on these feature points, the self-position estimation unit 204 estimates the road (travel path) on which the mobile body M is traveling and estimates the position of the mobile body M on the road.

[0061] Alternatively, the self-position estimation unit 204 may estimate its own position on the road by comparing feature points obtained from the camera image with feature points included in the map information 224. In this case, the self-position estimation unit 204 estimates the position of the mobile body M by acquiring the position information of the mobile body M using a GPS (Global Positioning System) device (not shown) built into the mobile body M, or by communicating with a communication device located within a predetermined distance via the communication unit 190 using a short-range wireless communication method such as Bluetooth® to acquire the position information of the communication device, and then extracts surrounding feature points by referring to the map information based on the estimated position information.

[0062] In this embodiment, using all camera images from multiple cameras to extract feature points and estimate self-localization would increase the processing load for image processing, etc., so feature point extraction and self-localization are performed using only some of the camera images. In this case, depending on the position of user U, as shown in feature point FP# in Figure 7, there is a possibility that feature points that could be acquired may be hidden by user U and cannot be detected. As a result, matching with the feature points of the map information 224 may not be performed correctly, and appropriate self-localization may not be possible.

[0063] Therefore, in this embodiment, the self-position estimation unit 204 estimates the self-position of the moving object M based on the camera image captured by the camera associated with the movement mode set by the user U, among a plurality of cameras.

[0064] For example, if the movement mode set by the user U is the follow mode, the self-position estimation unit 204 would use the image captured by the first camera 182, but there are feature points (feature point FP# shown in Figure 7) that cannot be recognized from the image. Therefore, the self-position estimation unit 204 uses the camera image of the second camera 184, which captures an area including the rear, to estimate the self-position of the moving object M. In the follow mode, the user U is always in front of the moving object M and never behind it. By using the camera image of the second camera 184, which captures the rear, the feature points are not hidden by the user U, allowing the feature points to be acquired and a more appropriate self-position estimation to be performed.

[0065] Furthermore, when the movement mode is the leading mode, the self-position estimation unit 204 estimates the self-position of the moving body M using the camera image of the first camera 182, which captures an area including the front of the moving body M, because the user U will be located behind the moving body M.

[0066] Furthermore, when the movement mode is parallel movement mode, the self-position estimation unit 204 estimates the self-position of the moving object M using the image from at least one of the first camera 182 and the second camera 184, since the user U will be within the imaging range of the third camera 186 or the fourth camera 188. In this case, the self-position estimation unit 204 may use the image from the camera with the higher pre-set priority among the first camera 182 and the second camera 184, or it may use the image from the camera that has acquired more feature points.

[0067] When the moving object M is moving in one of the following, leading, or parallel movement modes, the distance between the moving object M and the user U is closer than with other objects, resulting in a greater number of feature points FP# being obscured by the user U. Therefore, by not using images containing the user U for self-localization (in other words, by using images without the user for self-localization), the self-localization of the moving object M can be estimated more accurately.

[0068] The self-position estimation unit 204 switches the camera image used for self-position estimation when the movement mode specified by the user U is switched. Since the positional relationship between the user U and the moving object M is determined according to the movement mode, by switching the camera used for self-position estimation at the time of switching the movement mode, it is possible to select an image in which the user is not in the shooting range (or an image in which the user will be out of the range in the near future) without performing known image analysis processing, thereby reducing the processing load and enabling more appropriate self-position estimation.

[0069] Furthermore, the self-position estimation unit 204 may choose not to use camera images obtained from multiple cameras for self-position estimation of the moving body M if the number of dynamic objects (e.g., traffic participants) included in the camera images is greater than a predetermined number. For example, if there is a predetermined number or more pedestrians in the camera image, it becomes difficult to recognize the feature points of static obstacles (objects), making self-position estimation using feature points difficult. By not using such images, self-position estimation can be performed more appropriately. In addition, if there is a predetermined number or more dynamic objects in the camera image of a camera associated with a movement mode, the self-position estimation unit 204 may, for example, use the camera image of another camera predetermined regardless of the movement mode to estimate its own position. Furthermore, if there are more dynamic objects in the camera image of the second camera 184 than in the camera image of the first camera 182, the self-position estimation unit 204 may use the camera image of the first camera 182 to estimate its own position regardless of the movement mode. Conversely, if there are more moving objects in the camera images of the first camera 182 than in the camera images of the second camera 184, the self-position estimation unit 204 may use the camera images of the second camera 184 to estimate its own position, regardless of the mode of movement.

[0070] [Processing when switching camera images] When the self-position estimation unit 204 switches the camera 180 (or camera image) to perform self-position estimation, it may retain the position of the mobile body M before the switch (final position) when switching the camera used to estimate the self-position, and associate the self-position before and after the switch with the initial position of the mobile body M in the camera image after the switch. This prevents the position of the mobile body M from changing significantly when switching the camera (camera image) used for self-position estimation, allowing for continuous self-position estimation and suppressing large changes in behavior during driving control.

[0071] Furthermore, if the recognition unit 208 recognizes a specific structure in the surrounding area (for example, a landmark, sign or marker, vending machine, utility pole, traffic light), the self-position estimation unit 204 may estimate the self-position of the moving object M before and after camera switching based on that specific structure or the characters drawn on that structure. This makes it possible to estimate the self-position before and after camera switching more accurately.

[0072] [Embodiment: Processing Flow] Figure 8 is a flowchart showing an example of processing performed by the control device 200 of the mobile body M in the embodiment. In the example of Figure 8, the processing performed by the control device 200 will be mainly described in part, focusing on the process of switching the camera image used according to the movement mode to estimate the self-position. Note that the processing shown in Figure 8 may be executed repeatedly at a predetermined period or timing. Also, in the example of movement mode shown in Figure 8, if it is neither the follow mode nor the lead mode, it will be described as the parallel running mode.

[0073] In the example shown in Figure 8, the acquisition unit 202 acquires camera images captured by multiple cameras (first camera 182, second camera 184, third camera 186, and fourth camera 188) (step S100). Next, the self-position estimation unit 204 acquires the movement mode instructed by the user (step S110). Next, the self-position estimation unit 204 determines whether the acquired movement mode is a follow mode (step S120). If it is determined to be a follow mode, the self-position estimation unit 204 estimates the self-position of the moving object M using the camera image captured by the second camera 184 (step S130).

[0074] Furthermore, if it is determined in step S120 that the mode is not follow mode, the self-position estimation unit 204 determines whether the movement mode is lead mode or not (step S140). If it is determined that the mode is lead mode, the self-position estimation unit 204 estimates its own position using the camera image captured by the first camera 182 (step S150). Also, if it is determined in step S140 that the mode is not lead mode, the movement mode is assumed to be parallel mode, and the self-position is estimated using the camera image captured by at least one of the first camera 182 and the second camera 184 (step S160). After the processing in steps S130, S150, or S160, movement control corresponding to the movement mode is executed based on the estimated self-position information (step S170). This completes the processing of this flowchart.

[0075] [Modification] In the mobile body control system 1 of the embodiment, at least a part of the configuration of the mobile body M may be provided in the management device 10 or the information providing device 20. For example, at least a part of the configuration of the acquisition unit 202, self-position estimation unit 204, recognition unit 208, and trajectory generation unit 210, etc., may be provided in the management device 10 or the information providing device 20, and the mobile body M may acquire various information and perform control, etc., while communicating with the above devices via the communication unit 190. Alternatively, in the mobile body control system 1 of the embodiment, a part of the configuration of the terminal device 2, the management device 10, and the information providing device 20 may be provided in the mobile body M.

[0076] In addition, in the embodiment, when the movement mode is follow mode, the self-position estimation unit 204 may estimate the self-position of the moving body M by using a camera image captured by the third camera 186, which captures an area including the left side of the moving body M, or by the fourth camera 188, which captures an area including the right side of the moving body M, instead of (or in addition to) the camera image captured by the second camera 184. Also, when the movement mode is lead mode, the self-position estimation unit 204 may estimate the self-position of the moving body M by using a camera image captured by the third camera 186 or the fourth camera 188, instead of (or in addition to) the camera image captured by the first camera 182.

[0077] Furthermore, the self-position estimation unit 204 may store movement history information, including past position information and feature point information of the moving object M, in the storage unit 220. Before and after the camera switch, it may refer to the movement history information based on the position information of the moving object M to acquire feature point information, and then estimate the position of the moving object M based on the acquired feature point information and the feature point information after the switch. By using past movement history, the self-position can be estimated more appropriately.

[0078] In other embodiments, the recognition unit 208 may use camera images from multiple cameras 180 (first camera 182, second camera 184, third camera 186, fourth camera 188) to recognize the surrounding conditions of the moving object M, and the self-position estimation unit 204 may use the camera image from the first camera 182 or the second camera 184 to estimate its own position. In this case, the camera switching corresponding to the movement mode described above is also applied to the self-position estimation process.

[0079] In another embodiment, the camera 180 may be a wide-angle camera that captures images of the area around the moving object M in a wide angle (for example, 360 degrees). In this case, the self-position estimation unit 204 may estimate its own position using feature points of the remaining image region after excluding the region containing the user U from the captured image.

[0080] In addition, in this embodiment, the self-position estimation unit 204 may exclude images containing the user U from the images used for self-position estimation. In this case, the self-position estimation unit 204 may switch the images used for self-position estimation at the timing when the images containing the user U are switched.

[0081] Furthermore, in this embodiment, instead of mounting four cameras (first camera 182, second camera 184, third camera 186, and fourth camera 188) on the mobile body M, only the pair of the first camera 182 and the second camera 184 may be mounted, or only the pair of the third camera 186 and the fourth camera 188 may be mounted. For example, if only the pair of the first camera 182 and the second camera 184 is mounted on the mobile body M, the self-position estimation unit 204 estimates the self-position of the mobile body M based on the detection result of the second camera 184 when the movement mode is the follow mode. Also, when the movement mode is the lead mode, the self-position estimation unit 204 estimates the self-position of the mobile body M based on the detection result of the first camera 182, and when the movement mode is the parallel movement mode, it estimates the self-position of the mobile body M based on the detection result of at least one of the first camera 182 and the second camera 184.

[0082] Furthermore, if the mobile body M is equipped only with the third camera 186 and the fourth camera 188, the self-position estimation unit 204 estimates the mobile body M's position based on the detection result from the camera that does not detect the presence of user U, when the movement mode is parallel movement mode. Alternatively, the self-position estimation unit 204 may estimate the mobile body M's position based on the detection result from at least one of the third camera 186 and the fourth camera 188 when the movement mode is follow mode or lead mode.

[0083] Furthermore, in the embodiments described above, the switching of cameras corresponding to the movement mode may be applied instead of (or in addition to) self-position estimation when creating a map from images taken while the moving object M is moving. For example, in so-called SLAM, not only self-position estimation but also environmental map creation is performed, so by applying the switching of cameras corresponding to the movement mode described above when creating the environmental map, it is possible to extract feature points necessary for creating the environmental map while reducing the processing load.

[0084] Furthermore, in the embodiments described above, the mobile vehicle M may be one on which a user U can ride. In this case, the mobile vehicle M is equipped with driving controls such as a steering wheel, accelerator pedal, and brake pedal so that the user U can drive it. In addition, at least a portion of the speed control and turning control of the mobile vehicle M may be controlled by user operation (driving operation).

[0085] According to the embodiments described above, the control device 200 for the mobile body M includes: an acquisition unit 202 that acquires detection results from a plurality of cameras 180 (an example of sensors) that detect the surrounding conditions of the mobile body M; a self-position estimation unit 204 that estimates the self-position of the mobile body M based on a camera image (an example of detection result) obtained from at least one of the plurality of cameras 180 acquired by the acquisition unit 202; and a movement control unit (first control unit 212, second control unit 214) that controls the movement of the mobile body M to move to a position corresponding to a movement mode set by the user based on the position information estimated by the self-position estimation unit 204. The self-position estimation unit 204 can estimate the self-position of the mobile body M more appropriately according to the movement status of the mobile body by estimating the self-position of the mobile body M based on the detection results of the sensor corresponding to the movement mode among the plurality of sensors.

[0086] Specifically, according to the embodiment, by setting the range of images used for self-position estimation for each movement mode (switching the camera used), the processing load can be reduced compared to using all images. Furthermore, according to the embodiment, when switching modes, the final position before the switch is retained and used as the initial position after the switch, allowing for continuous self-position estimation while switching the sensors used for each movement mode.

[0087] The embodiment described above can be expressed as follows: A control device for a moving body, comprising: a storage medium for storing computer-readable instructions; a processor connected to the storage medium, wherein the processor executing the computer-readable instructions to: acquire detection results from a plurality of sensors for detecting the surrounding conditions of a moving body; estimate the self-position of the moving body based on the detection results obtained from at least one of the acquired plurality of sensors; control the movement of the moving body to move to a position corresponding to a movement mode set by the user based on the estimated self-position information of the moving body; and estimate the self-position of the moving body based on the detection results of the sensor among the plurality of sensors that corresponds to the movement mode.

[0088] Although embodiments for carrying out the present invention have been described above using examples, the present invention is not limited in any way to these embodiments, and various modifications and substitutions can be made without departing from the spirit of the present invention.

[0089] 1...Mobile object control system, 2...Terminal device, 10...Management device, 20...Information provision device, 110...Base unit, 160...Operation unit, 170...Information output unit, 180...Camera, 190...Communication unit, 200...Control device, 202...Acquisition unit, 204...Self-position estimation unit, 206...Information processing unit, 208...Recognition unit, 210...Trajectory generation unit, 212...First control unit, 214...Second control unit, 220...Storage unit, M...Mobile object

Claims

1. A control device for a moving object, comprising: an acquisition unit that acquires detection results from a plurality of sensors that detect the surrounding conditions of a moving object; a self-position estimation unit that estimates the self-position of the moving object based on the detection result obtained from at least one of the plurality of sensors acquired by the acquisition unit; and a movement control unit that controls the movement of the moving object to move to a position corresponding to a movement mode set by the user, based on the position information estimated by the self-position estimation unit, wherein the self-position estimation unit estimates the self-position of the moving object based on the detection result of the sensor among the plurality of sensors that is corresponding to the movement mode.

2. The control device for a mobile body according to claim 1, wherein the movement mode includes at least a follow mode in which the mobile body follows the user, a lead mode in which the mobile body leads the user, and a parallel mode in which the mobile body moves alongside the user.

3. The control device for a mobile body according to claim 2, wherein the plurality of sensors include a first sensor for detecting the conditions of a region including the front of the mobile body, a second sensor for detecting the conditions of a region including the rear of the mobile body, a third sensor for detecting the conditions of a region including the left side of the mobile body, and a fourth sensor for detecting the conditions of a region including the right side of the mobile body, and the self-position estimation unit estimates the self-position of the mobile body based on the detection result of the second sensor when the movement mode is the follow mode, estimates the self-position of the mobile body based on the detection result of the first sensor when the movement mode is the lead mode, and estimates the self-position of the mobile body based on the detection result of at least one of the first sensor and the second sensor when the movement mode is the parallel movement mode.

4. The control device for a mobile body according to claim 2, wherein the plurality of sensors include a first sensor that detects the conditions of a region including the front of the mobile body and a second sensor that detects the conditions of a region including the rear of the mobile body, and the self-position estimation unit estimates the self-position of the mobile body based on the detection result of the second sensor when the movement mode is the follow mode, estimates the self-position of the mobile body based on the detection result of the first sensor when the movement mode is the lead mode, and estimates the self-position of the mobile body based on the detection result of at least one of the first sensor and the second sensor when the movement mode is the parallel movement mode.

5. The control device for a mobile body according to claim 2, wherein the plurality of sensors include a third sensor that detects the conditions of a region including the left side of the mobile body and a fourth sensor that detects the conditions of a region including the right side of the mobile body, and the self-position estimation unit estimates the self-position of the mobile body based on the detection result from the sensor of the third sensor and the fourth sensor that does not detect the presence of the user when the movement mode is parallel movement mode.

6. The control device for a moving body according to claim 1, wherein the self-position estimation unit does not estimate the self-position of the moving body using the detection results of the sensors when there is a sensor among the plurality of sensors that detects a predetermined number or more of dynamic objects.

7. The control device for a moving body according to claim 1, wherein the self-position estimation unit, when switching sensors used to estimate the self-position, retains the final position before the switch and associates the self-position before and after the switch as the initial position after the switch.

8. A mobile body control system comprising: an acquisition unit that acquires detection results from a plurality of sensors that detect the surrounding conditions of a mobile body; a self-position estimation unit that estimates the self-position of the mobile body based on the detection result obtained from at least one of the plurality of sensors acquired by the acquisition unit; and a movement control unit that controls the movement of the mobile body to move to a position corresponding to a movement mode set by the user, based on the position information estimated by the self-position estimation unit, wherein the self-position estimation unit estimates the self-position of the mobile body based on the detection result of the sensor among the plurality of sensors that is corresponding to the movement mode.

9. A method for controlling a moving object, comprising: a computer acquiring detection results from a plurality of sensors that detect the surrounding conditions of the moving object; estimating the self-position of the moving object based on the detection results obtained from at least one of the acquired plurality of sensors; controlling the movement of the moving object to move to a position corresponding to a movement mode set by the user based on the estimated self-position information of the moving object; and estimating the self-position of the moving object based on the detection results of the sensor among the plurality of sensors that corresponds to the movement mode.

10. A program that causes a computer to acquire detection results from multiple sensors that detect the surrounding conditions of a moving object, to estimate the self-position of the moving object based on the detection results obtained from at least one of the acquired multiple sensors, to control the movement of the moving object to move to a position corresponding to a movement mode set by the user based on the estimated self-position information of the moving object, and to estimate the self-position of the moving object based on the detection results of the sensor among the multiple sensors that corresponds to the movement mode.

Images