Mobile units and programs

Abstract

To provide a moving body capable of moving while appropriately following a target to be followed. [Solution] A moving body 1 that moves by tracking a target is equipped with a ranging sensor 11 that measures distances to surrounding objects in multiple directions, an image sensor 12 that acquires captured images of the area around the moving body 1, an identification unit 13 that, when the moving body 1 starts moving, identifies the position of the target in a local coordinate system of the moving body 1 using the measurement results of the ranging sensor 11 corresponding to the area of ​​the target identified in the captured image acquired by the image sensor 12, and, while the moving body 1 is moving, identifies the position of the target in the local coordinate system using the measurement results of the ranging sensor 11, a moving mechanism 16 that moves the moving body 1, and a movement control unit 17 that controls the moving mechanism 16 to follow the target identified by the identification unit 13.

Description

[Technical field]

[0001] The present invention relates to a moving object or the like that moves while following a target to be followed. [Background technology]

[0002] 2. Description of the Related Art Conventionally, a moving object that moves by following a human using the results of measurements by a distance sensor is known (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2014-178789 A Summary of the Invention [Problem to be solved by the invention]

[0004] Although the distance to the object around the moving object can be accurately known by using the distance sensor, there is a problem that it is difficult to properly judge whether it is a tracking target such as a human or not. If the moving object moves by tracking an object other than the appropriate tracking target, there is a problem that the movement will be unexpected.

[0005] The present invention has been made to solve the above-mentioned problems, and has an object to provide a moving body or the like that can move while appropriately tracking a target to be tracked. [Means for solving the problem]

[0006] In order to achieve the above-mentioned object, a moving body according to one aspect of the present invention is a moving body that moves by following a target to be tracked, and is equipped with a ranging sensor that measures distances to surrounding objects in multiple directions, an image sensor that acquires captured images of the area around the moving body, an identification unit that, when the moving body starts to move, identifies the position of the target to be tracked in a local coordinate system of the moving body using the measurement results of the ranging sensor corresponding to the area of ​​the target to be tracked identified in the captured image acquired by the image sensor, and, while the moving body is moving, identifies the position of the target to be tracked in the local coordinate system using the measurement results of the ranging sensor, a moving mechanism that moves the moving body, and a moving control unit that controls the moving mechanism to follow the target to be tracked identified by the identification unit.

[0007] With this configuration, the target object is identified using the captured image, and the position of the identified target object is identified using the measurement results of the distance measuring sensor, so that the position of the target object can be identified more accurately. As a result, the target object can be moved while being appropriately tracked.

[0008] In addition, in a moving body according to one aspect of the present invention, an output unit is further provided which outputs an action instruction regarding a visually recognizable action to a tracking target when the moving body starts moving, and the identification unit may identify a tracking target that has performed an action in accordance with the action instruction in a captured image acquired by the image sensor.

[0009] With this configuration, it is possible to more accurately identify the person to be tracked. For example, even if a captured image contains multiple people, it is possible to more accurately identify the person to be tracked.

[0010] In addition, in a moving body according to one aspect of the present invention, the movement control unit may control the speed and acceleration of the moving body within ranges below the upper limit of the speed and acceleration of the moving body so that the distance between the moving body and the target to be followed is constant, and may further include an output unit that outputs a deceleration instruction to the target to decelerate when the moving body is unable to follow the target.

[0011] With this configuration, the target to be followed moves more slowly in response to the deceleration command, thereby enabling the moving object to appropriately follow the target to be followed.

[0012] In addition, in the moving body according to one aspect of the present invention, the output unit may output a reason for the deceleration instruction together with the deceleration instruction.

[0013] With this configuration, for example, the person to be followed can be informed of the reason for the instruction to decelerate, so that in the event of a similar situation later, the person to be followed can move at a speed that allows the moving object to follow, for example.

[0014] Furthermore, in the moving body according to one aspect of the present invention, the output unit does not need to repeatedly output the same reason.

[0015] With this configuration, it is possible to avoid, for example, repeatedly outputting the same reason, which may be annoying to the person being followed.

[0016] In addition, in a moving body according to one aspect of the present invention, when the target to be tracked temporarily stops, the identification unit repeatedly identifies the target to be tracked in the captured image acquired by the image sensor and identifies the position of the target to be tracked in a local coordinate system using the measurement results of the ranging sensor corresponding to the area of ​​the target to be tracked identified in the captured image, and the movement control unit may stop the moving body when the target to be tracked temporarily stops, and may resume the moving body's tracking of the target to be tracked when the target to be tracked resumes movement.

[0017] With this configuration, when the target to be tracked is temporarily stopped, it is possible to prevent the moving body from making small movements in response to small movements of the target to be tracked.

[0018] In addition, a moving body according to one aspect of the present invention may further include a memory unit and a storage unit that stores the latest position of the target to be tracked identified by the identification unit in the memory unit, and when the position of the target to be tracked cannot be identified by the identification unit, the moving control unit may control the moving mechanism to move along a route corresponding to the latest position of the target to be tracked stored by the storage unit.

[0019] With this configuration, when it becomes impossible to move while following the target to be followed, it is possible to switch to movement along the route.

[0020] In addition, in the moving body according to one aspect of the present invention, the route may be a route corresponding to the latest position of the target to be followed, from among a plurality of routes previously set in a global coordinate system.

[0021] With this configuration, the vehicle moves along a route that has been set in advance, making it possible to achieve more stable movement.

[0022] In addition, in a moving body according to one aspect of the present invention, the memory unit stores map information, and the moving control unit may use the map information to generate a route from the position of the moving body when the identification unit is no longer able to identify the position of the target to be tracked to a predicted position based on the latest position of the target to be tracked.

[0023] With this configuration, even if the route has not been set in advance, it is possible to travel along a route generated using map information.

[0024] In addition, in the moving body according to one aspect of the present invention, the route may be a route corresponding to the latest position of the target to be followed, among a plurality of routes previously set in a local coordinate system of the moving body.

[0025] With this configuration, it becomes possible to move along a route without using map information.

[0026] In addition, in a moving body according to one aspect of the present invention, the identification unit may store in a memory unit characteristic information of the target to be tracked identified in the captured image acquired by the image sensor, and after the moving body moves along a route corresponding to the latest position of the target to be tracked, use the characteristic information to identify the target to be tracked in the captured image acquired by the image sensor.

[0027] With this configuration, after moving along the route, the stored feature information can be used to identify the target to be tracked in the captured image, making it possible to move again while tracking the target.

[0028] In addition, in a moving body according to one aspect of the present invention, the moving body may further include an output unit that outputs an action instruction regarding a visually recognizable action to the target to be tracked after the moving body has moved along a route corresponding to the latest position of the target to be tracked, and the identification unit may identify the target to be tracked that has performed an action corresponding to the action instruction in the captured image acquired by the image sensor.

[0029] With this configuration, the target to be tracked can be identified more accurately by identifying in the captured image the target that has moved along the route and then performed an action in response to the action instruction, and it becomes possible to move again by tracking the target to be tracked. Effect of the Invention

[0030] According to the moving body or the like according to one aspect of the present invention, it becomes possible to move while appropriately following up a target to be followed up. [Brief description of the drawings]

[0031] [Figure 1] FIG. 1 is a schematic diagram showing a moving body and a transport vehicle according to an embodiment of the present invention; [Diagram 2] FIG. 2 is a bottom view of the moving body according to the embodiment. [Diagram 3] FIG. 2 is a functional block diagram showing the configuration of a moving body according to the embodiment; [Figure 4A] A flowchart showing the operation of a moving body according to the embodiment. [Figure 4B] A flowchart showing the operation of a moving body according to the embodiment. [Figure 5A] FIG. 2 is a diagram for explaining a distance measuring sensor in the embodiment; [Figure 5B] FIG. 2 is a diagram for explaining an image sensor in the embodiment; [Figure 6] FIG. 13 shows an example of a captured image in the embodiment. [Figure 7] FIG. 13 is a diagram showing an example of a measurement result of a distance measuring sensor in the embodiment; [Figure 8A] FIG. 2 is a diagram showing an example of the position of a tracking target stored in the embodiment; [Figure 8B] FIG. 2 is a diagram showing an example of the position of a tracking target stored in the embodiment; [Figure 9A] FIG. 2 is a diagram for explaining an example of movement of a moving object in the embodiment; [Figure 9B] FIG. 2 is a diagram for explaining an example of movement of a moving object in the embodiment; [Figure 9C] FIG. 2 is a diagram for explaining an example of movement of a moving object in the embodiment; [Figure 9D] FIG. 2 is a diagram for explaining an example of movement of a moving object in the embodiment; [Figure 9E] FIG. 2 is a diagram for explaining an example of movement of a moving object in the embodiment; [Figure 10] FIG. 2 shows an example of the configuration of a computer system according to the embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0032] The moving body according to the present invention will be described below using an embodiment. In the following embodiments, components and steps with the same reference numerals are the same or equivalent, and repeated description may be omitted. The moving body according to this embodiment identifies a target to be tracked using a captured image, identifies the position of the identified target to be tracked using the measurement results of a distance measuring sensor, and moves so as to track the target to be tracked whose position has been identified.

[0033] FIG. 1 is a schematic diagram showing a moving body 1 and a transporting platform vehicle 2 according to this embodiment, FIG. 2 is a bottom view of the moving body 1, and FIG.

[0034] The moving body 1 according to the present embodiment may be, for example, a moving body that transports a transport target 3. The moving body 1 moves by following a follow-up target, and may include a main body 10 and a moving mechanism 16 attached to the main body 10, as shown in FIG. 1 and FIG. 2. The moving body 1 transporting the transport target 3 may be, for example, the moving body 1 transporting the transport target 3 loaded on itself, or the moving body 1 towing the transport cart 2 loaded with the transport target 3. In the present embodiment, the latter case will be mainly described. In the present embodiment, the moving body 1 transports the transport target 3, but the moving body 1 may be used for other purposes. The moving body 1 may be, for example, a moving body for guidance, guard, surveillance, entertainment, etc. That is, the moving body 1 may be a transport robot, a guide robot, a guard robot, a surveillance robot, an entertainment robot, etc. The moving body 1 may be, for example, a moving body that moves indoors, a moving body that moves outdoors, or both. In this embodiment, the case where the moving object 1 is a running object will be mainly described, and other cases will be described later. In this embodiment, the case where the tracking target is a person 5 as shown in FIG.

[0035] The main body 10 of the moving body 1 may be, for example, a loading platform, or may be a member to which the moving mechanism 16 and the like are attached. In this embodiment, a case where the main body 10 is a rectangular parallelepiped member to which the moving mechanism 16 is attached will be mainly described. The moving mechanism 16 will be described later.

[0036] Referring to FIG. 3, the moving body 1 according to this embodiment includes a distance measurement sensor 11, an image sensor 12, an identification unit 13, a memory unit 14, an accumulation unit 15, a moving mechanism 16, a moving control unit 17, and an output unit 18.

[0037] The distance measuring sensor 11 measures the distance to the surrounding object in a plurality of directions. The distance measuring sensor 11 may be, for example, a laser sensor, an ultrasonic sensor, a distance sensor using microwaves, or the like. The laser sensor may be, for example, a laser range sensor (laser range scanner) or LiDAR (lidar). Note that these distance measuring sensors are already known, and their description will be omitted. In this embodiment, the case where the distance measuring sensor 11 is a laser range sensor will be mainly described. The distance measuring sensor 11 may have one laser range sensor, or may have two or more laser range sensors. In the latter case, all directions may be covered by two or more laser range sensors. In addition, when the distance measuring sensor 11 is an ultrasonic sensor or a distance sensor using microwaves, the distance measuring direction of the distance sensor may be rotated to measure the distance in a plurality of directions, or the distance in a plurality of directions may be measured using a plurality of distance sensors arranged in each of the plurality of directions. Measuring the distance in a plurality of directions may mean, for example, measuring the distance in a plurality of directions at a predetermined angular interval for a predetermined angular range or the entire surroundings (i.e., 360 degrees). The angular intervals may be constant, for example, 1 degree intervals, 2 degree intervals, 5 degree intervals, etc. Information obtained from the distance measuring sensor 11 may be, for example, the distance to a surrounding object for each of a plurality of azimuth angles based on a certain orientation of the moving body 1. By using the distance, it becomes possible to know what objects exist around the moving body 1 in the local coordinate system of the moving body 1. The distance measuring sensor 11 may measure, for example, a distance in a two-dimensional planar direction, or may measure distances in multiple directions in three dimensions. In this embodiment, a case where the distance measuring sensor 11 measures a distance in a two-dimensional horizontal direction will be mainly described.

[0038] The image sensor 12 acquires a captured image of the surroundings of the moving object 1. The image sensor 12 may be, for example, a CCD image sensor or a CMOS image sensor. The image sensor 12 may include, for example, an optical system such as a lens for forming an image on the image sensor. The image sensor 12 may be monocular or binocular (stereo camera). In this embodiment, the case where the image sensor 12 is monocular will be mainly described. The image sensor 12 may be one that captures a moving image, that is, one that captures continuous image frames, or one that captures a still image. As will be described later, in the case where a tracking target that has performed an operation according to an operation instruction is identified, it is preferable that the image sensor 12 captures a moving image. The captured image may be, for example, color or grayscale, but from the viewpoint of identifying the tracking target, it is preferable that the image sensor 12 is color.

[0039] It is preferable that the distance measuring sensor 11 and the image sensor 12 each obtain at least information on the traveling direction (i.e., forward) of the moving object 1. FIG. 5A and FIG. 5B are top views of the moving object 1, and respectively show distance measurement by the distance measuring sensor 11 and image capture by the image sensor 12. Note that in FIG. 5A and FIG. 5B, the distance measuring sensor 11 and the image sensor 12 are shown in separate drawings for convenience of explanation, but in reality, the distance measuring sensor 11 and the image sensor 12 are mounted on one moving object 1. The distance measuring direction of the distance measuring sensor 11 and the image capture direction of the image sensor 12 may be the same, for example, as shown in FIG. 5A and FIG. 5B. In addition, the sensing ranges of both sensors are assumed to have at least an overlapping range. The sensing ranges of both sensors may be the same, for example, or one may be wider than the other. It is also preferable that the measurement results obtained by the distance measuring sensor 11 and the captured image obtained by the image sensor 12 are able to show the correspondence relationship between the angle of the distance measurement direction and the position in the captured image. For example, it is preferable that it is able to show which horizontal position in the captured image each angle of the distance measurement direction corresponds to.

[0040] Furthermore, the frequency at which the distance measuring sensor 11 obtains measurement results and the frequency at which the image sensor 12 obtains captured images (i.e., the number of images obtained per unit time) may be approximately the same, for example, or the frequency at which the distance measuring sensor 11 obtains measurement results may be higher than the frequency at which the image sensor 12 obtains captured images. There are no particular limitations on the frequency at which the two obtain information, but, for example, the distance measuring sensor 11 may perform measurements about 1 to 100 times per second, and the image sensor 12 may perform imaging about once every 1 to 10 seconds.

[0041] The identification unit 13 identifies the position of the target to be tracked in the local coordinate system of the moving object 1 using the measurement result of the distance measuring sensor 11 and the captured image acquired by the image sensor 12. When the moving object 1 starts moving, the identification unit 13 may identify the target to be tracked in the captured image acquired by the image sensor 12. If the target to be tracked is a person, this identification may be performed, for example, by recognizing the area of ​​the person in the captured image. The recognition of the area of ​​the person in the captured image may be performed, for example, by pattern matching of the person, by performing segmentation and identifying the area of ​​the person in the result of the segmentation, or by using a detection model of a person (for example, Yolov5, etc.).

[0042] Thereafter, the identification unit 13 may identify the position of the target to be tracked in the local coordinate system of the moving body 1 using the measurement result of the distance measuring sensor 11 corresponding to the region of the target to be tracked identified in the captured image. More specifically, the identification unit 13 may identify the position of the target to be tracked in the local coordinate system of the moving body 1 using the azimuth of the target to be tracked identified in the captured image or the distance measured by the distance measuring sensor 11 in the range of the azimuth angle. Note that the position of the target to be tracked identified by the identification unit 13 may be, for example, a coordinate value in a two-dimensional orthogonal coordinate system which is the local coordinate system of the moving body 1, or may be a distance and a direction from an origin in a two-dimensional polar coordinate system which is the local coordinate system of the moving body 1. In this embodiment, the former case will be mainly described.

[0043] The measurement result of the distance measuring sensor 11 is a set of distances for each angle, that is, a set of distances to each measurement point. Therefore, the identification unit 13 may perform clustering on the measurement results, for example. This clustering allows classification into clusters of distances to walls, clusters of distances to people, clusters of distances to obstacles other than people, and the like. For an example of this clustering, see, for example, Patent Document 1. The identification unit 13 may then identify a cluster corresponding to the area of ​​the target to be tracked identified in the captured image, and may set the representative position of the measurement points included in the cluster as the position of the target to be tracked. The cluster corresponding to the area of ​​the target to be tracked identified in the captured image may be, for example, a cluster present in the azimuth direction of the target to be tracked identified in the captured image. The representative position of the multiple measurement points may be, for example, the position of the center of gravity of the multiple measurement points, or the position of the measurement point closest to the moving body 1 among the multiple measurement points.

[0044] The identification unit 13 may identify the position of the target to be tracked in the local coordinate system by using the measurement result of the distance measurement sensor 11 while the moving body 1 is moving. For example, the identification unit 13 may identify the position of the target to be tracked in the local coordinate system of the moving body 1 without using the captured image acquired by the image sensor 12 while the moving body 1 is moving. For example, the identification unit 13 may identify the latest position of the target to be tracked by tracking the position of the target in the measurement result of the distance measurement sensor 11 while the moving body 1 is moving. More specifically, the identification unit 13 may perform clustering on the distance to each measurement point included in the measurement result of the distance measurement sensor 11 at each time point. In addition, the identification unit 13 may identify the correspondence relationship between the clusters by using the shape of the measurement point included in each cluster according to the measurement result at a certain time and the shape of the measurement point included in each cluster according to the measurement result at the next time. In identifying the correspondence relationship, clusters at adjacent times where the shapes of multiple measurement points are similar and are located close to each other may be identified as corresponding clusters. The shape of the plurality of measurement points may be similar, for example, when the similarity of the shape of the plurality of measurement points exceeds a threshold value. Also, the close position may be, for example, when it is closer than the threshold value. In this manner, the identification unit 13 may track the position of the cluster of the tracking target. Note that, for example, when the identification unit 13 is unable to identify the cluster corresponding to the tracking target using the distance measurement result, the identification unit 13 may determine that the tracking target cannot be identified. For example, when the tracking target becomes invisible to the moving body 1 by turning a corner, the identification unit 13 may determine that the tracking target cannot be identified.

[0045] Moreover, the moving body 1 moves by following the target to be followed. Therefore, in a situation where appropriate movement control is being performed, the position of the target to be followed in the local coordinate system will be almost the same. Therefore, the identification unit 13 may use the measurement result of the distance measuring sensor 11 to determine the position of an object whose position change in the local coordinate system during the movement of the moving body 1 is smaller than that of other objects as the position of the target to be followed. In this way, the identification unit 13 can prevent, for example, a person crossing in front of the moving body 1 from being erroneously recognized as the target to be followed. More specifically, the identification unit 13 may perform clustering on the distance to each measurement point included in the measurement result of the distance measuring sensor 11 at each time point. Furthermore, the identification unit 13 may identify the change in the position of each cluster in the local coordinate system by identifying the correspondence relationship of each cluster in the time series. Then, the identification unit 13 may determine the cluster whose position change is the smallest as the cluster to be followed.

[0046] When the target to be tracked pauses, the identification unit 13 may repeatedly identify the target to be tracked in the captured image acquired by the image sensor 12 and identify the position of the target to be tracked in the local coordinate system using the measurement result of the distance measuring sensor 11, which corresponds to the area of ​​the target to be tracked identified in the captured image. The case where the target to be tracked pauses may be, for example, when the target to be tracked does not move beyond a predetermined range (e.g., 1 meter, 2 meters, etc.) for a predetermined time (e.g., 3 seconds, 5 seconds, etc.) or more, or when the speed of movement of the target to be tracked becomes less than a threshold (e.g., 0.1 meters per second, 0.2 meters per second, etc.).

[0047] When the target to be tracked pauses during movement, the identification unit 13 may identify the target to be tracked in an area in the captured image corresponding to the latest position of the target to be tracked in the local coordinate system, which is identified using the measurement result of the distance measuring sensor 11. For example, when the target to be tracked is a person, the identification of the target to be tracked may be performed by recognizing the area of ​​the person in the captured image and identifying, as the target to be tracked, the person present in the area of ​​the captured image corresponding to the latest position of the target to be tracked in the local coordinate system.

[0048] During the temporary stop of the target to be followed, the identification unit 13 may identify the target by tracking the area of ​​the target in the captured image, and may also identify the position of the target in the local coordinate system using the measurement result of the distance measuring sensor 11, which corresponds to the area of ​​the target to be followed identified in the captured image. In order to reduce the processing load, the area of ​​the target to be followed in the captured image may be tracked only in the vicinity of the area of ​​the target to be followed. In this way, when the target to be followed is temporarily stopped, the position of the target to be followed can be identified more accurately by identifying the position of the target to be followed using not only the distance measurement result but also the captured image. When the target to be followed is moving, it is generally considered that there are not many people or the like that may be mistaken for the target to be followed around the target to be followed. On the other hand, when the target to be followed is temporarily stopped, there may be people or the like that may be mistaken for the target to be followed, but by tracking the area of ​​the target to be followed using the captured image, the possibility of identifying a person or the like that is not the target to be followed as the target to be followed can be reduced, and more accurate tracking of the target to be followed can be realized. The identification unit 13 may also determine whether the target to be followed has resumed movement using the position of the target in the local coordinate system. For example, the identification unit 13 may determine that the target to be followed has resumed movement when the distance between the moving body 1 and the target to be followed exceeds a threshold, or may determine that the target to be followed has resumed movement when at least one of the speed and acceleration of the target to be followed exceeds a threshold. When the target to be followed has resumed movement, the identification unit 13 may identify the position of the target to be followed in the local coordinate system of the moving body 1 using the measurement result of the distance measuring sensor 11 corresponding to the area of ​​the target to be followed identified in the captured image, as when the moving body 1 starts moving, and may identify the position of the target to be followed in the local coordinate system during the movement of the moving body 1 thereafter using the measurement result of the distance measuring sensor 11.

[0049] In the storage unit 14, at least the latest position of the target to be tracked identified by the identification unit 13 is stored. In addition, in the storage unit 14, the position of the target to be tracked in the latest predetermined range including the latest position may be stored. The predetermined range may be, for example, a predetermined time range (for example, a range of the latest 5 seconds or 10 seconds) or a predetermined distance range (for example, a range of the latest 5 meters or 10 meters). In the former case, for example, a position acquired within a predetermined time from the latest position is stored in the storage unit 14, and in the latter case, for example, an acquired position included within a predetermined distance from the latest position is stored in the storage unit 14. In this embodiment, a case where the latest predetermined range is a time range will be mainly described.

[0050] In addition, the storage unit 14 may store information other than the position of the tracking target, such as map information, as described below. The storage unit 14 is preferably realized by a non-volatile recording medium, but may also be realized by a volatile recording medium. The recording medium may be, for example, a semiconductor memory or a magnetic disk.

[0051] The accumulation unit 15 accumulates the latest position of the target to be followed, which is identified by the identification unit 13, in the storage unit 14. When only the latest position is used among the positions stored in the storage unit 14, the accumulation unit 15 may accumulate the latest position by overwriting, or may accumulate the latest position separately from the positions stored up to that point. In the latter case, it is preferable that the latest position is accumulated so that it is possible to know which is the latest position. In addition, when the latest position of a predetermined range is stored in the storage unit 14, the accumulation unit 15 may delete other positions so that only the latest position of the predetermined range is stored in the storage unit 14, or may accumulate the latest positions sequentially so that it is possible to know which is the latest position of the predetermined range. In addition, the accumulation unit 15 may accumulate all the latest positions identified by the identification unit 13, or may accumulate some of the latest positions. In the latter case, for example, the latest position may be accumulated at a predetermined time interval, or the latest position may be accumulated at a predetermined distance.

[0052] In addition, when the latest position of the tracking target stored by the storage unit 15 is a position in the local coordinate system of the moving body 1, if the moving body 1 moves after the position is stored, the relationship between the moving body 1 after the movement and the stored position becomes unclear. Therefore, the storage unit 15 may store, for example, information indicating the difference in movement of the moving body 1 from when the latest position is stored to when the next latest position is stored in the storage unit 14. The information indicating the difference in movement may be, for example, information indicating the position and direction of the moving body 1 at the time when the next position is stored, based on the position and direction of the moving body 1 at the time when a certain position is stored. In other words, it may be information indicating the position and direction of the moving body 1 at the time when the next position is stored in the local coordinate system of the moving body 1 at the time when a certain position is stored. The information indicating the difference in movement may be obtained, for example, using the number of rotations of the wheels 31a and 31b, which are driving wheels described later. Note that a method of acquiring the current position and current direction using the number of rotations of the driving wheels is already known, and the description thereof will be omitted.

[0053] Also, for example, instead of storing information indicating the difference in movement in the storage unit 14, the accumulation unit 15 may constantly update the positions stored in the storage unit 14 so that they become positions in the current local coordinate system. In this case, the accumulation unit 15 may update each position stored in the storage unit 14 so that each time the moving body 1 moves by a small amount, it becomes a position in the local coordinate system after the small amount of movement. This update may also be performed using the number of rotations of the wheels 31a and 31b, which are the drive wheels.

[0054] Furthermore, the accumulation unit 15 may accumulate, for example, the position of the tracking target in the global coordinate system in the storage unit 14. In this case, the accumulation unit 15 may convert the position of the moving body 1 in the local coordinate system into a position in the global coordinate system by using, for example, the current position of the moving body 1 in the global coordinate system acquired by a current position acquisition unit included in the moving body 1, and accumulate the converted position in the storage unit 14.

[0055] The moving mechanism 16 moves the moving body 1, and may have wheels 31a to 31d and motors 32a and 32b provided on the back side of the main body 10 as shown in FIG. 2. When the wheels 31a to 31d are not particularly distinguished from one another, they may be called wheels 31. When the motors 32a and 32b are not distinguished from one another, they may be called motors 32. The number of wheels 31 is not limited as long as it is three or more. The wheels 31 may be omnidirectional wheels. As shown in FIG. 2, the wheels 31a and 31b are driving wheels driven by the motors 32a and 32b, respectively, and the wheels 31c and 31d are driven wheels that can turn. In this embodiment, a case will be described in which the traveling direction of the moving body 1 is determined by the difference in the rotation speed of the driving wheels 31a and 31b, but the moving mechanism 16 may have a steering mechanism that can change the direction of the wheels 31c and 31d, for example. All of the wheels 31 may be driving wheels. In this case, some of the wheels 31, for example, the wheels 31c and 31d on the front side, may be steering wheels, or all of the wheels 31 may be steering wheels, and the moving mechanism 16 may have a steering mechanism capable of changing the direction of the steering wheels. The motor 32 or the wheels 31a and 31b may be provided with an encoder for acquiring the number of rotations of the wheels 31a and 31b.

[0056] The movement control unit 17 controls the moving mechanism 16 so as to follow the target identified by the identification unit 13. The control of the moving mechanism 16 may be, for example, control of the start of movement, control of the stop of movement, control of the direction of movement, etc. For example, when the position of the target to be followed is identified by the identification unit 13, the movement control unit 17 may control the moving mechanism 16 so as to move toward the latest position of the target to be followed identified by the identification unit 13. When the target to be followed stops, the movement control unit 17 may control the moving mechanism 16 so as to stop at a position where the distance between the target to be followed and the moving body 1 is a predetermined distance. That is, the movement control unit 17 may perform movement control to follow the target to be followed so that the distance between the target to be followed and the moving body 1 is a predetermined distance. Note that the movement control unit 17 does not necessarily need to know the position in the global coordinate system in order to perform movement control to follow the target to be followed. Therefore, the moving body 1 does not need to have a current position acquisition unit that acquires the current position in the global coordinate system, such as a configuration such as a Global Positioning System (GPS) or Simultaneous Localization and Mapping (SLAM), but may have a current position acquisition unit to convert the position in the local coordinate system to the position in the global coordinate system, as described above.

[0057] The movement control unit 17 may control the moving mechanism 16 so that the distance between the moving body 1 and the object to be followed is constant. The distance is preferably close enough that a third party can recognize that the moving body 1 and the object to be followed are moving together, such as 1 meter to 3 meters. The movement control unit 17 may also control the speed and acceleration of the moving body 1 within a range below the upper limit of the speed and acceleration of the moving body 1. The upper limit of the speed and acceleration may be predetermined, or may be dynamically set based on the weight of the object to be transported. The predetermined upper limit may be determined according to the capacity of the motor 32 of the moving mechanism 16, for example. The dynamically set upper limit may be set so that the weight of the object to be transported increases, and the upper limit may be set so that the upper limit is lower within the range below the predetermined upper limit, for example, in order to prevent the load of the object to be transported from collapsing, and the upper limit is lower as the curvature of the trajectory of the moving body 1 increases.

[0058] For example, the movement control unit 17 may stop the moving body 1 when the target to be followed pauses, and may resume the moving of the moving body 1 to follow the target to be followed when the target to be followed resumes moving. Whether the target to be followed pauses may be determined by, for example, the identification unit 13. When the target to be followed pauses, stopping the moving body 1 can be prevented from moving finely in response to the minute movements of the target to be followed (for example, movements of short distances such as 10 centimeters or 20 centimeters) in the case where the moving body 1 is following the target to be followed so that the distance between the moving body 1 and the target to be followed is constant, when the target to be followed pauses. For example, when a person to be followed is waiting for a traffic light, the moving body 1 can be prevented from moving finely in response to the minute movements of the person.

[0059] For example, when the position of the target to be tracked cannot be identified by the identification unit 13, the movement control unit 17 may control the movement mechanism 16 to move along a route corresponding to the latest position of the target to be tracked stored by the storage unit 15. Here, each of the cases will be described: (1) when the route is set in advance in a global coordinate system, (2) when the route is generated using map information, and (3) when the route is set in advance in a local coordinate system.

[0060] (1) When a route is already set in the global coordinate system It is considered that the movement path of the target to be followed is often determined in advance. For example, when the moving body 1 moves while following the target to be followed indoors such as in a warehouse, it is considered that the movement path of the target to be followed when turning a corner of a shelf or the like is usually the same movement path in many cases. Therefore, it is considered that it becomes possible to move close to the target to be followed by setting a route in the global coordinate system at each corner where the position of the target to be followed cannot be identified, selecting a route according to the latest position of the target to be followed, and moving the moving body 1 along the selected route.

[0061] In this case, a plurality of routes may be set in advance in the global coordinate system. The plurality of routes may be local routes, for example, routes for each corner. Information indicating the plurality of routes may be stored in the storage unit 14. When the position of the target to be followed cannot be identified by the identification unit 13, the movement control unit 17 may select a route corresponding to the latest position of the target to be followed, stored by the storage unit 15, from among the plurality of routes set in advance, and control the moving mechanism 16 to move along the selected route. The movement control unit 17 may select, for example, a route that is closest to the latest position of the target to be followed, from among the plurality of routes set in advance. In this case, since the plurality of routes are set in the global coordinate system, it is preferable that the moving body 1 has a current position acquisition unit. The movement control unit 17 may convert the latest position of the target to be followed into a position in the global coordinate system using the current position acquired by the current position acquisition unit, and use the converted position to select the above-mentioned route. In this case, the route along which the moving body 1 moves is a route corresponding to the latest position of the target to be followed, from among the plurality of routes set in advance in the global coordinate system. Then, the movement control unit 17 may control the movement mechanism 16 so that the moving object 1 moves along the route. The multiple routes may be, for example, larger routes. In this case, the movement control unit 17 may select a route according to the latest position of the tracking target, and move along the selected route to a position around a corner.

[0062] (2) When generating a route using map information In this case, map information may be stored in the storage unit 14. This map information may be map information that can be used for route generation. The map information that can be used for route generation may be, for example, map information indicating the position of a passageway through which the moving body 1 can move, or may be map information indicating the position of an obstacle such as a wall or an object that the moving body 1 cannot move. In addition, the movement control unit 17 may predict the current position of the target to be followed according to the latest position of the target to be followed stored by the storage unit 15. This prediction may be performed, for example, by extrapolating the latest multiple positions of the target to be followed. Then, the movement control unit 17 may use, for example, the map information stored in the storage unit 14 to generate a route from the position of the moving body 1 when the position of the target to be followed cannot be identified by the identification unit 13 to a position predicted according to the latest position of the target to be followed, and control the movement mechanism 16 so that the moving body 1 moves along the generated route.

[0063] (3) When a route is already set in the local coordinate system As described above, when movement control is performed so that the distance between the target to be tracked and the moving body 1 is constant, the positional relationship between the target to be tracked and the moving body 1 when the position of the target to be tracked cannot be specified can be divided into several patterns. For example, the pattern can be divided into a pattern in which the target to be tracked cannot be specified because the target to be tracked has turned left, and a pattern in which the target to be tracked cannot be specified because the target to be tracked has turned right. Therefore, it is considered that the moving body 1 can move close to the target to be tracked by setting a route for the moving body 1 according to each pattern in the local coordinate system of the moving body 1, selecting a route according to the latest position of the target to be tracked, and moving the moving body 1 along the selected route.

[0064] In this case, a plurality of routes may be set in advance in the local coordinate system of the moving body 1. The plurality of routes may be, for example, a route for turning left at a corner, a route for turning right at a corner, and the like. Information indicating the plurality of routes may be stored in the storage unit 14. Then, when the identifying unit 13 cannot identify the position of the target to be followed, the movement control unit 17 may select a route corresponding to the latest position of the target to be followed, stored by the storage unit 15, from among the plurality of routes set in advance, and control the movement mechanism 16 to move along the selected route. For example, the movement control unit 17 may select a route that is closest to the latest position of the target to be followed, from among the plurality of routes set in advance. In this case, the route along which the moving body 1 moves is the route corresponding to the latest position of the target to be followed, from among the plurality of routes set in advance in the local coordinate system of the moving body 1. Then, the movement control unit 17 may control the movement mechanism 16 to move the moving body 1 along that route.

[0065] The movement control unit 17 may move the moving body 1 a predetermined distance, for example, by movement control along a route. The predetermined distance may be, for example, determined in advance, or may be the distance between the moving body 1 and the target to be followed when the moving body 1 moves while following the target to be followed. It is preferable that the predetermined distance is set so that the moving body 1 can identify the position of the target to be followed by moving the predetermined distance along the route.

[0066] After the movement control along the route has been performed, the identification unit 13 may again identify the position of the target to be tracked in the local coordinate system of the moving body 1 by using the measurement result of the distance measuring sensor 11 corresponding to the area of ​​the target to be tracked identified in the photographed image acquired by the image sensor 12. Then, the movement control unit 17 may resume the movement control for tracking the target to be tracked by using the identified position of the target to be tracked.

[0067] Furthermore, the movement control unit 17 may detect an obstacle using, for example, at least one of the measurement results of the distance measuring sensor 11 and the captured image acquired by the image sensor 12, and control the movement mechanism 16 to avoid collision with the detected obstacle. Avoidance of collision with an obstacle may be performed, for example, by decelerating or stopping, or by changing the movement path to avoid the obstacle. Note that movement control to a predetermined target position, movement control for avoiding collision with an obstacle, and the like are already known, and detailed description thereof will be omitted.

[0068] The output unit 18 may output an instruction to the person to be tracked. For example, when the moving object 1 starts moving, the output unit 18 may output an instruction to the person to be tracked to move in front of the image sensor 12.

[0069] The output unit 18 may output, for example, an operation instruction related to a visually recognizable operation to the target to be followed. The operation instruction may instruct the target to perform a specific operation so that the target can be uniquely identified in the captured image. The output of this operation instruction may be performed, for example, when the moving object 1 starts moving, or after the moving object 1 moves along a route according to the latest position of the target to be followed. The output of the operation instruction may be performed, for example, when the target to be followed cannot be uniquely identified using the captured image, or may be performed regardless of whether the target to be followed can be uniquely identified. In the former case, for example, a candidate for the target to be followed (for example, a person, etc.) existing near the center of the captured image is specified as the target to be followed, and the operation instruction may be output when multiple candidates for the target to be followed exist near the center of the captured image.

[0070] When the target to be followed is a person, the action instruction may be an instruction to perform a physical action. The physical action may be, for example, a gesture, such as jumping or twisting the body. The gesture is not particularly limited, and may be, for example, raising a hand, repeatedly opening and closing the palm, or forming a specific shape such as a scissors shape. It is preferable that the person to be followed performs an action such that the action according to the action instruction is photographed by the moving body 1. For example, when forming a specific shape of the hand and the moving body 1 is present behind the person to be followed, the person to be followed may form the hand in a shape according to the action instruction behind himself / herself.

[0071] When the target to be tracked is a moving object other than a person, the operation instruction may be, for example, an instruction to perform an operation related to light of the target to be tracked, an operation such as turning, etc. The operation related to light may be, for example, an operation related to turning on, turning off, blinking, etc. In the operation of turning on, turning off, or blinking the light, for example, the color of the light, the blinking interval, the lighting interval, the turning off interval, etc. may be determined by the operation instruction.

[0072] When the target to be followed is a person, the output of the operation instruction may be, for example, a sound output of a voice corresponding to the operation instruction, or may be a transmission of the voice. In the former case, a voice such as "Please raise your right hand" or "Please make a scissors shape" may be output from a speaker of the moving body 1. In the latter case, the voice may be transmitted to earphones, headphones, a neck speaker, or the like worn by the person to be followed, and the voice may be output from the earphones, or the like. When the target to be followed is a moving body other than a person, the output of the operation instruction may be, for example, a transmission of the operation instruction to the target to be followed. The same may be true for the output of instructions other than the operation instruction.

[0073] When the output unit 18 outputs an action instruction, the identification unit 13 may identify the target to be followed that has performed an action in response to the action instruction in the captured image acquired by the image sensor 12. In this case, for example, the captured image may be a moving image. The identification unit 13 may identify the target to be followed that has performed the action by identifying the action in response to the action instruction in the captured image, which is a moving image. Note that the recognition of the action in response to the action instruction may be performed using a trained learning model, such as 3D-CNN or (2+1)D-CNN. The identification unit 13 may also identify the target to be followed that has performed an action in the captured image, which is a still image. As an example, the identification unit 13 may identify a person who is the target to be followed who has raised one hand in response to the action instruction in the captured image, which is a still image.

[0074] Here, the case has been described where the target to be tracked is identified in the captured image in response to an operation instruction being output after the moving body 1 has moved along a route corresponding to the latest position of the target to be tracked, but this is not necessarily the case. For example, the identification unit 13 may store in advance in the storage unit 14 characteristic information of the target to be tracked identified in the captured image acquired by the image sensor 12, and after the moving body 1 has moved along a route corresponding to the latest position of the target to be tracked, use the characteristic information to identify the target to be tracked in the captured image acquired by the image sensor 12.

[0075] The feature information stored in the storage unit 14 is preferably feature information on a target object that is not performing an operation according to an operation instruction. The feature information is not particularly limited, and may include, for example, a feature amount acquired from a region of the target object in a captured image. The feature information may include, for example, one or more feature amounts selected from a group consisting of an average luminance value, a variance of luminance values, a kurtosis of luminance values, a skewness of luminance values, a higher-order local autocorrelation (HLAC), a SIFT feature amount, and a histogram of oriented gradients (HOG). In addition, when the target object is a person, the specification unit 13 may perform skeletal recognition on the person and acquire the skeletal features (for example, the relative relationship between arm length, shoulder width, etc.). The skeletal features may also be included in the feature information. In addition, when the captured image is a moving image, features related to the movement of the skeletal features may also be acquired, and the features related to the movement of the skeletal features may also be included in the feature information. In addition, when features such as the size and moving speed of the target object are acquired using the measurement results of the distance measurement sensor 11, these features may also be included in the feature information. The identification unit 13 may also identify the target to be tracked by using the features of the target to be tracked acquired using the measurement results of the distance measuring sensor 11. In this case, the target to be tracked may be identified according to the feature information by using both the captured image and the distance measurement results.

[0076] When identifying the target to be tracked using the feature information, the identification unit 13 may, for example, identify a region of the candidate target to be tracked in the captured image, acquire feature information from the candidate region, and determine that the candidate region is the region to be tracked when the similarity between the acquired feature information and the feature information stored in the storage unit 14 exceeds a threshold. Note that, when the feature information also includes features related to the distance measurement result, the identification unit 13 may, for example, determine as the target to be tracked a candidate target to be tracked whose similarity with the feature information exceeds a threshold in both the captured image and the distance measurement result.

[0077] When the moving body 1 cannot follow the target, the output unit 18 may output a deceleration instruction to the target to decelerate. For example, when the target moves at a speed or acceleration exceeding the upper limit of the speed or acceleration of the moving body 1, the moving body 1 cannot follow the target so that the distance between the target and the moving body 1 is constant. Therefore, the output unit 18 may output a deceleration instruction, for example, when the distance between the target and the moving body 1 gradually increases. The case where the moving body 1 cannot follow the target may be, for example, when the distance between the target and the moving body 1 exceeds a threshold.

[0078] When the target to be followed is a person, the output of the deceleration instruction may be, for example, a sound output of a voice corresponding to the deceleration instruction (e.g., "Please move more slowly") or the transmission of that voice. The deceleration instruction may be, for example, output in the same manner as an action instruction. When the target to be followed is a moving object other than a person, the output of the deceleration instruction may be, for example, the transmission of a deceleration instruction to the target to be followed.

[0079] The output unit 18 may output, for example, a reason corresponding to the deceleration instruction together with the deceleration instruction. The reason corresponding to the deceleration instruction may be, for example, a reason why the upper limit of the speed or acceleration at that time is set. In this case, for example, the upper limit of the speed and acceleration may be associated with and stored in the moving body 1. Then, the reason corresponding to the upper limit of the speed or acceleration at that time may be output by the output unit 18. For example, in the case where the upper limit of the speed or acceleration is set according to the capacity of the motor 32, when the moving body 1 cannot follow the target in a situation where at least one of the speed or acceleration is at the upper limit, a reason that the moving body 1 cannot move any faster due to its capacity may be output together with the deceleration instruction. Also, for example, in the case where the upper limit of the speed or acceleration is changed to a lower value according to the weight of the object to be transported, when the moving body 1 cannot follow the target in a situation where at least one of the speed or acceleration is at the upper limit, a reason that the moving body 1 cannot move any faster in order to prevent the load of the object to be transported from collapsing may be output together with the deceleration instruction.

[0080] The output unit 18 may not repeat the output of the same reason. Not repeating the output of the same reason may mean, for example, not repeating the output for the same reason a predetermined number of times or more, or not repeating the output for the same tracking target a predetermined number of times or more, or not repeating the output for the same reason a predetermined number of times or more in a predetermined period. The predetermined number of times may be, for example, two times, or three or more times. When the predetermined number of times is two times, for example, each reason may be output only once.

[0081] In the case where the output of the same reason is not repeated a predetermined number of times or more, correspondence information that associates the reason with the number of times the reason is output may be stored. Then, for example, when outputting a certain reason, the output unit 18 may not output the reason if the number of times the reason corresponding to the reason is equal to or greater than a predetermined number of times. Note that, in the case where the output of the same reason is not repeated a predetermined number of times or more within a predetermined period, the correspondence information may associate the reason with the output time point of the reason. Then, for example, when outputting a certain reason, the output unit 18 may count the number of times the reason is output within a predetermined period using the output time point corresponding to the reason, and may not output the reason if the count result is equal to or greater than a predetermined number of times.

[0082] In the case where the output of the same reason for the same tracking target is not repeated a predetermined number of times or more, correspondence information that associates the reason with the number of times the reason is output for each identifier of the tracking target may be stored. Then, the output unit 18 may use, for example, correspondence information corresponding to the tracking target that the moving object 1 is following and moving, so as not to repeat the output of the same reason a predetermined number of times or more. Note that the identifier of the tracking target may be, for example, input to the moving object 1 by the tracking target at the start of the movement, or may be characteristic information of the tracking target identified in the captured image. For example, the above-mentioned information may be used as the characteristic information.

[0083] In addition, when the distance between the moving body 1 and the target to be followed increases because the target does not decelerate even if a deceleration instruction is output, or because the deceleration is insufficient, the output unit 18 may output, for example, an instruction to the target to return to the moving body 1. If the target to be followed returns in response to the instruction, the movement may be continued, and if not, the moving body 1 may stop moving. In addition, when stopping the movement, the output unit 18 may output an instruction to the target to move in front of the image sensor 12. If the target to be followed moves in front of the image sensor 12 in response to the instruction, the target to be followed in the captured image and the position of the target to be followed using the distance measurement result may be specified, as in the case of starting the movement, and tracking of the target to be followed may be resumed. On the other hand, if the target to be followed does not move in front of the image sensor 12 in response to the instruction, the moving body 1 may stop completely.

[0084] Here, the output by the output unit 18 may be, for example, a transmission via a communication line, a sound output by a speaker, or a display on a display device. The output unit 18 may or may not include a device that performs the output (for example, a speaker or a communication device). The output unit 18 may be realized by hardware, or may be realized by software such as a driver that drives those devices.

[0085] The transporting platform 2 has a main body 20, a plurality of wheels 21 fixed to the rear side of the main body 20, and a frame portion 22 provided on the front side and the rear side of the main body 20 in the traveling direction. The main body 20 is a loading platform, and the transport object 3 is placed on the loading platform. The number of wheels 21 is not limited as long as it is two or more. All the wheels 21 may be rotatable driven wheels, or some of the wheels may be rotatable driven wheels and the rest may be fixed wheels that cannot be rotated. The transporting platform 2 may be towed by the moving body 1 by connecting the connecting portion 10a of the moving body 1 and the connecting portion 20a of the transporting platform 2. The connecting portion 10a of the moving body 1 and the connecting portion 20a of the transporting platform 2 may be connected by a connecting shaft extending in the vertical direction. In this case, at least one of the connecting portions 10a and 20a may be provided to be rotatable around the connecting shaft. The transport vehicle 2 may have any configuration as long as it can carry the object 3 to be transported and can be towed by the moving body 1. For example, the transport vehicle 2 may not be provided with the frame portion 22, and the frame portion 22 may be provided around the entire periphery of the main body 20.

[0086] The transport object 3 is not particularly limited, but may be, for example, a cardboard box, a folding container, or the like, may be placed on a pallet, or may be any other transport object.

[0087] Next, the operation of the moving object 1 will be described with reference to the flowchart of FIG. 4A. (Step S101) The output unit 18 outputs an action instruction to a tracking target.

[0088] (Step S102) The image sensor 12 captures a captured image of the surroundings. This captured image may be a moving image.

[0089] (Step S103) The distance measuring sensor 11 measures the distances to surrounding objects in a plurality of directions.

[0090] (Step S104) The identification unit 13 identifies the target to be tracked that has performed an action in accordance with the output action instruction in the captured image acquired in step S102, and identifies a position in the local coordinate system corresponding to the area of ​​the identified target to be tracked, using the measurement results acquired in step S103.

[0091] (Step S105) The accumulation unit 15 accumulates in the memory unit 14 the latest position identified in step S104.

[0092] (Step S106) The movement control unit 17 controls the movement mechanism 16 so as to move to the latest position identified in step S104.

[0093] (Step S107) The distance measuring sensor 11 judges whether or not to measure the distance to the surrounding object in a plurality of directions. If the measurement is to be performed, the process proceeds to step S108, and if not, the process proceeds to step S122. The distance measuring sensor 11 may, for example, periodically judge to perform the measurement.

[0094] (Step S108) The distance measuring sensor 11 measures the distances to surrounding objects in a plurality of directions.

[0095] (Step S109) The identification unit 13 identifies the position of the target in the local coordinate system by using the measurement result acquired in step S108.

[0096] (Step S110) The identification unit 13 judges whether or not the position of the target to be tracked has been identified. If the position has been identified, the process proceeds to step S111, and if not, the process proceeds to step S115.

[0097] (Step S111) The accumulation unit 15 accumulates the latest position identified in step S109 in the storage unit 14. The accumulation unit 15 may accumulate, for example, the movement difference in the storage unit 14 together with the latest position.

[0098] (Step S112) The identification unit 13 judges whether the target to be tracked is temporarily stopped or not. If the target to be tracked is temporarily stopped, the process proceeds to step S114, and if not, the process proceeds to step S113.

[0099] (Step S113) The movement control unit 17 controls the movement mechanism 16 so as to move to the latest position identified in step S109. Then, the process returns to step S107.

[0100] (Step S114) The identification unit 13 and the like perform a process related to the temporary stop of the tracking target. Then, the process returns to step S107. Details of this process will be described later with reference to the flowchart in FIG. 4B.

[0101] (Step S115) The movement control unit 17 controls the moving mechanism 16 to move along a route according to the latest position of the target to be followed stored in the storage unit 14. By this movement control, for example, the moving body 1 may move to a position where the target to be followed can be identified.

[0102] (Step S116) The output unit 18 outputs an action instruction to the follow-up target.

[0103] (Step S117) The image sensor 12 captures a captured image of the surroundings. This captured image may be a moving image.

[0104] (Step S118) The distance measuring sensor 11 measures the distances to surrounding objects in a plurality of directions.

[0105] (Step S119) The identification unit 13 identifies the target to be tracked that has performed an action in accordance with the output action instruction in the captured image acquired in step S117, and identifies a position in the local coordinate system corresponding to the area of ​​the identified target to be tracked, using the measurement results acquired in step S118.

[0106] (Step S120) The accumulation unit 15 accumulates in the memory unit 14 the latest position identified in step S119.

[0107] (Step S121) The movement control unit 17 controls the movement mechanism 16 so as to move to the latest position identified in step S119. Then, the process returns to step S107.

[0108] (Step S122) The movement control unit 17 judges whether or not to output a deceleration command. If the deceleration command is to be output, the process proceeds to step S123, and if not, the process returns to step S107. Note that the movement control unit 17 may judge to output a deceleration command when the distance between the target to be followed and the moving body 1 exceeds a threshold value in a situation where at least one of the speed and the acceleration has reached an upper limit, for example.

[0109] (Step S123) The output unit 18 outputs a deceleration instruction to the tracking target.

[0110] (Step S124) The output unit 18 judges whether or not to output the reason for deceleration. If the reason for deceleration is to be output, the process proceeds to step S125, and if not, the process returns to step S107. Note that the output unit 18 may, for example, judge not to output the reason if the reason to be output has already been output a predetermined number of times, and may judge to output the reason if not.

[0111] (Step S125) The output unit 18 outputs the reason for deceleration. Then, the process returns to step S107. Note that the deceleration instruction and the reason may be output together. In this case, before the deceleration instruction is output, it may be determined whether or not to output the reason.

[0112] The order of the processes in the flowchart of Fig. 4A is an example, and the order of the steps may be changed as long as the same results can be obtained. In addition, in the flowchart of Fig. 4A, when an instruction to end the movement is input to the moving object 1, the series of processes may be ended. Also, the series of processes may be ended by power off or an interrupt to end the processes.

[0113] FIG. 4B is a flowchart showing details of the process (step S114) relating to the temporary suspension of the target to be followed up in the flowchart of FIG. 4A.

[0114] (Step S201) The image sensor 12 captures an image of the surroundings.

[0115] (Step S202) The distance measuring sensor 11 measures distances to surrounding objects in multiple directions.

[0116] (Step S203) The identification unit 13 identifies the target to be tracked in the captured image acquired in step S201, and identifies a position in the local coordinate system corresponding to the area of ​​the identified target to be tracked using the measurement result acquired in step S202. The first identification of the target to be tracked using the captured image in the process related to the temporary stop of the target to be tracked may be performed, for example, by identifying a candidate of the target to be tracked (e.g., a person, etc.) present in the area of ​​the captured image corresponding to the position of the target to be tracked in the local coordinate system of the moving body 1. Furthermore, the second or subsequent identification of the target to be tracked using the captured image in the process related to the temporary stop of the target to be tracked may be performed, for example, by tracking the position of the target to be tracked previously identified.

[0117] (Step S204) The identification unit 13 judges whether the target to be followed has resumed its movement. If the target to be followed has resumed its movement, the process returns to step S107 in the flowchart of FIG. 4A. If not, the process returns to step S201.

[0118] Next, the operation of the moving body 1 according to this embodiment will be described using a specific example. In this specific example, the moving body 1 moves by following the movement of a human being who is the object to be followed. First, it is assumed that the user who is the object to be followed operates the moving body 1 to input an instruction to start following and starts walking. Then, the output unit 18 outputs an instruction to raise the right hand to the object to be followed (step S101). It is assumed that the user who is the object to be followed raises the right hand in response to the output of the instruction. Then, the image sensor 12 acquires a photographed image of the periphery of the moving body 1 shown in FIG. 6 and passes it to the identification unit 13 (step S102). In addition, the distance measurement sensor 11 measures the distance to an object around the moving body 1 and passes it to the identification unit 13 (step S103). It is assumed that the measurement result of this distance is as shown in FIG. 7. In FIG. 7, the measurement points are indicated by black circles in the xy coordinate system which is the local coordinate system of the moving body 1.

[0119] The identification unit 13 identifies a person in the captured image shown in Fig. 6 using a person detection model, identifies the area of ​​the person 5 who is raising his right hand, which is an action according to the action instruction, and identifies the azimuth range of the identified area. The identification unit 13 also identifies measurement points corresponding to the azimuth range of the area of ​​the person 5 identified in the captured image in the distance measurement results shown in Fig. 7. In this specific example, it is assumed that a set of measurement points located on both sides of the y-axis is identified as the measurement points corresponding to the azimuth range of the area of ​​the person 5. Then, the identification unit 13 passes the position of the measurement point 6 closest to the moving object 1 out of the set of measurement points to the accumulation unit 15 and the movement control unit 17 as the latest position of the tracking target (step S104).

[0120] Assume that the position of the measurement point 6 in the local coordinate system is (X11, Y11). Then, the accumulation unit 15 accumulates the position of the target to be followed, the time T11 at that time, and the difference in movement in the storage unit 14 (step S105). The time may be acquired from a clock unit or a timer (not shown). At this point, the movement has just started, so the difference in movement may all be 0. The movement control unit 17 controls the moving mechanism 16 to move to the position (X11, Y11) of the target to be followed received from the identification unit 13 (step S106). The movement control unit 17 may acquire the difference in movement of the moving body 1 after the latest position of the target to be followed is identified by the identification unit 13 using the number of rotations of the wheels 31a and 31b, which are the drive wheels of the moving mechanism 16.

[0121] During the movement of the moving object 1, the distance measuring sensor 11 measures the distance to the object around the moving object 1 and passes it to the identification unit 13 (steps S107 and S108). The identification unit 13 uses the distance measurement result to identify the position of the target to be followed (step S109). The identified position, the time at that point in time, and the difference in movement are stored in the storage unit 14 by the accumulation unit 15 (steps S110 and S111). If the target to be followed is not temporarily stopped, movement control is performed using the identified position (steps S112 and S113). In this way, the moving object 1 moves while following the person 5, which is the target to be followed, as shown in FIG. 9A. For convenience of explanation, in FIG. 9A, the moving directions of the moving object 1 and the person 5 are indicated by arrows, and the transport vehicle 2 is omitted.

[0122] By repeating such movement control, the position of the target to be tracked and the movement difference are stored in the memory unit 14 as shown in Fig. 8A. Note that the time changes as time passes, such as T11, T12, and T13. In this specific example, it is also assumed that the latest three pieces of information are stored in the memory unit 14. The movement difference includes a position difference (a13, b13) and the like, and an angle difference θ13 and the like.

[0123] 9B, it is assumed that the person 5 to be tracked turns a corner of the wall 7 and the person 5 cannot be detected by the moving object 1. Then, the identifying unit 13 cannot identify the position of the person 5 to be tracked (step S109). Therefore, the position of the person to be tracked at the latest time T23 is no longer stored in the storage unit 14 as shown in FIG. 8B.

[0124] For example, when the moving object 1 loses the target to be followed and moves along a route as in (1) above, the movement control unit 17 may convert the position P22 corresponding to the latest position (X22, Y22) of the target to be followed in the local coordinate system into a position in the global coordinate system using the current position of the moving object 1 in the global coordinate system acquired by a current position acquisition unit (not shown), select a route that passes through a position closest to the latest position of the target to be followed from among a plurality of routes in the global coordinate system previously stored in the storage unit 14, and control the movement mechanism 16 so that the moving object 1 moves along the selected route (step S115). For example, if the route 51 shown in FIG. 9C is selected, the moving object 1 is controlled to move along the route 51.

[0125] For example, when the moving object 1 loses the target to be followed and moves along a route as in (2) above, the movement control unit 17 predicts the current position P23 of the target to be followed as shown in Fig. 9D using positions P21 and P22 corresponding to the positions (X21, Y21) and (X22, Y22) of the target to be followed shown in Fig. 8B, respectively. This position prediction may be performed, for example, by extrapolating the positions P21 and P22 stored in the storage unit 14. Then, the movement control unit 17 may generate a route 52 passing through the current position of the moving object 1 and the positions P21, P22, and P23, and control the movement mechanism 16 so that the moving object 1 moves along the route 52 (step S115).

[0126] For example, when the moving body 1 loses the target to be tracked and moves along a route as in (3) above, the movement control unit 17 may select, from among routes 53, 54 in the local coordinate system shown in FIG. 9E which are stored in advance in the memory unit 14, route 53 which passes through a position closest to position P22 which corresponds to the latest position (X22, Y22) of the target to be tracked in the local coordinate system, and control the movement mechanism 16 so that the moving body 1 moves along the selected route (step S115).

[0127] After the movement control unit 17 has performed movement control along the route, similar to when the movement started, operation instructions are output, photographed images are acquired, distances are measured, etc., and movement control is again performed so that the moving body 1 moves by following the target to be followed (steps S116 to S121).

[0128] In addition, when the target to be followed is temporarily stopped, a process corresponding to the temporary stop (step S114) is performed. Specifically, the acquisition of a photographed image by the image sensor 12, the measurement of the distance by the distance measuring sensor 11, the identification of the target to be followed in the photographed image, and the identification of the position of the target to be followed using the distance measurement result corresponding to the identified area are repeated (steps S201 to S203). In this way, when the target to be followed is temporarily stopped, the moving body 1 is stopped without performing the movement control. Therefore, when the target to be followed is temporarily stopped, it is possible to avoid the moving body 1 moving only a little in response to a slight change in the position of the target to be followed. Then, the identification unit 13 uses the position of the target to be followed to determine whether the target to be followed has resumed its movement (step S204), and when the target to be followed has resumed its movement, the movement control for following the target to be followed is performed again (steps S107 to S113).

[0129] Furthermore, when the moving body 1 is moving and the distance between the target and the moving body 1 exceeds a threshold value even though the moving body 1 is moving at the upper limit speed, the movement control unit 17 determines to output a deceleration instruction, and transmits to the output unit 18 a message indicating that the deceleration instruction is to be output and a reason corresponding to the upper limit speed at that time (step S122). Upon receiving these, the output unit 18 outputs a deceleration instruction to the target (step S123). The target to be followed moves more slowly in response to the output deceleration instruction, and the moving body 1 becomes able to appropriately follow the target.

[0130] In addition, if the output of the reason is not repeated a predetermined number of times or more, the output unit 18 also outputs the received reason (steps S124, S125). In response to this output, the number of outputs corresponding to the output reason may be incremented by 1. For example, if the upper limit of the speed is based on the capacity of the motor 32, a reason such as "it cannot move any faster" may be output as a voice. On the other hand, if the upper limit of the speed is set lower than the upper limit based on the capacity of the motor 32 according to the weight of the transport object, a reason such as "it cannot move faster because the weight of the transport object is large" may be output as a voice. In this way, by outputting the reason based on the deceleration instruction, the user who is the follow-up target can know the maximum speed based on the capacity of the motor 32 of the moving object 1, or the fact that the user must move more slowly when the weight of the transport object is large, and can use this as a reference for subsequent movements. In addition, the output unit 18 can avoid outputting the same reason unnecessarily repeatedly by not repeating the output of the same reason a predetermined number of times or more.

[0131] In this specific example, as shown in Figs. 8A and 8B, the case where the time is stored in the memory unit 14 together with the position of the target to be tracked and the difference in movement is described, but the time does not have to be stored. This is because the time is not necessary if the data is stored so that the newer position of the target to be tracked can be known. In addition, in this specific example, the case where the position of the target to be tracked is stored each time the position of the target to be tracked is specified is described, but this is not the case. The position of the target to be tracked may be stored less frequently than the position of the target to be tracked is specified. The position of the target to be tracked may be stored at a predetermined time interval, for example, or may be stored only once each time the position of the target to be tracked is specified a predetermined number of times.

[0132] As described above, according to the moving object 1 of this embodiment, after a person is identified by the image sensor 12, the position of the person can be identified using the measurement result of the distance measuring sensor 11, and the position of the target to be tracked can be identified more accurately. As a result, it becomes possible to move by appropriately tracking the target to be tracked. In addition, by storing the latest position of the identified target to be tracked, even if the position of the target to be tracked cannot be identified, it is possible to move along a route according to the stored latest position. As a result, by approaching the target to be tracked that has become unidentifiable due to a corner or an obstacle, if the target to be tracked can be identified again, it becomes possible to continue tracking the target to be tracked.

[0133] In addition, by outputting the operation instruction, it becomes possible to more accurately identify the target to be tracked in the captured image. For example, even if the target to be tracked is a person and the captured image contains multiple people, it becomes possible to more accurately identify the target to be tracked by identifying the person to be tracked in response to the operation instruction.

[0134] Furthermore, when the moving body 1 is unable to follow the target, a deceleration instruction is output, so that the moving body 1 can appropriately follow the target. In this case, the reason corresponding to the deceleration instruction is also output, so that, for example, the person who is the target to be followed can learn how fast to move when being followed by the moving body 1. Furthermore, by preventing the output of the reason from being repeated more than a predetermined number of times, it is possible to prevent the same reason from being output so repeatedly that, for example, the person who is the target to be followed finds it annoying.

[0135] In addition, when the target to be tracked stops temporarily, the position of the target to be tracked is repeatedly determined using the captured images and distance measurement results, and movement control is not performed, thereby preventing the moving body 1 from moving unnecessarily in small increments.

[0136] The action instruction may be, for example, an instruction to perform a predetermined action while making a specific utterance. As an example, the action instruction may be an instruction such as "Raise your right hand while saying 'Yes'". In this case, the voice uttered by the person to be followed may be received by, for example, a microphone of the moving body 1, or may be collected by a microphone in headphones or the like worn by the person to be followed and transmitted, and then received by the moving body 1. Then, the identification unit 13 may identify, as the person to be followed, the person performing the action according to the action instruction at the timing when the specific utterance is made.

[0137] In addition, in this embodiment, the case where the operation instruction is output to enable a predetermined target to be identified in the captured image has been mainly described, but the operation instruction may also be output for the purpose of recruiting targets to be followed. In this case, the moving body 1 that is not following the target to be followed may repeatedly output a sound such as "If you wish to be followed, please raise your hand" from a speaker. Then, when a person raises their hand in response, the moving body 1 may identify that person as a target to be followed in the captured image and start following the target to be followed.

[0138] In addition, in the present embodiment, the case where the target to be tracked is a person has been mainly described, but for example, a specific person may be the target to be tracked. In this case, the identification unit 13 may perform face authentication or the like using the captured image, and may set only a specific person as the target to be tracked. The target to be tracked may not be a person, but may be another moving body. In this case, for example, a figure or a marker that can identify the moving body may be attached to the other moving body, and the identification unit 13 may recognize the figure or the marker in the captured image to identify the position of the moving body to be tracked. Note that, for example, a figure or a marker may not be attached to the moving body to be tracked, and the identification unit 13 may identify the moving body to be tracked using features such as the shape of the moving body to be tracked. In this case, the identification of the other moving body in the captured image may be performed, similarly to the identification of a person, using, for example, pattern matching, segmentation, a detection model, or the like.

[0139] In addition, in the present embodiment, the image sensor 12 is mainly described as being monocular, but the image sensor 12 may be, for example, a stereo camera. In this case, it is also possible to specify the position of the target to be tracked using the captured image. Therefore, the specification unit 13 may specify the final position of the target to be tracked using the position of the target to be tracked acquired using the distance measurement result and the position of the target to be tracked acquired using the captured stereo image. The specification of the final position may be, for example, specification of a representative position of both positions. The representative position may be an intermediate position between both positions, or may be a position resulting from combining both positions using their respective reliability.

[0140] In addition, in the present embodiment, a case has been mainly described in which when the position of the target to be tracked cannot be specified, movement control is performed using the stored past positions of the target to be tracked, but this is not necessarily the case. The moving body 1 does not necessarily have to perform such control. In this case, the moving body 1 does not necessarily have to include the memory unit 14 or the accumulation unit 15.

[0141] In addition, in the present embodiment, the case where the output unit 18 outputs an operation instruction or a deceleration instruction has been mainly described, but the output unit 18 may not output one of the instructions, for example. Also, the moving body 1 may not output an instruction such as an operation instruction or a deceleration instruction. In this case, the moving body 1 may not be provided with the output unit 18.

[0142] In addition, in the present embodiment, the moving body 1 is mainly described as a traveling body that travels on land, but this is not essential. The moving body 1 may be, for example, an aircraft flying in the air, or may be an aquatic moving body (for example, a ship, etc.) that travels on water. The aircraft may be, for example, a rotorcraft, an airplane, an airship, or other aircraft. From the viewpoint of being movable to any position, it is preferable that the aircraft is a rotorcraft. The rotorcraft may be, for example, a helicopter, or a multicopter having three or more rotors (rotors). The multicopter may be, for example, a quad rotor having four rotors, or may have any other number of rotors. When the moving body 1 is an aircraft, the movement mechanism 16 may have, for example, a propeller, and a driving means such as a motor or an engine that drives the propeller. Furthermore, when the moving body 1 is a water moving body, the moving mechanism 16 may have, for example, a screw, a driving means such as a motor or engine for driving the screw, a rudder, and a steering mechanism capable of changing the direction of the rudder.

[0143] In addition, in the above embodiments, each process or function may be realized by centralized processing in a single device or a single system, or may be realized by distributed processing in multiple devices or multiple systems.

[0144] In addition, in the above embodiments, the transfer of information between each component may be performed, for example, by one component outputting information and the other component receiving information if the two components transferring the information are physically different, or, if the two components transferring the information are physically the same, by shifting from a processing phase corresponding to one component to a processing phase corresponding to the other component.

[0145] In the above embodiment, information related to the processing performed by each component, such as information accepted, acquired, selected, generated, transmitted, or received by each component, and information such as thresholds, formulas, and addresses used by each component in processing, may be temporarily or long-term stored in a recording medium (not shown) even if not specified in the above description. Furthermore, each component or a storage unit (not shown) may store information in the recording medium (not shown). Furthermore, each component or a reading unit (not shown) may read information from the recording medium (not shown).

[0146] In the above embodiment, if the information used by each component, such as the thresholds, addresses, and various setting values ​​used by each component in processing, may be changed by the user, the user may be able to change the information as appropriate, even if not specified in the above description. If the information is changeable by the user, the change may be realized, for example, by a reception unit (not shown) that receives a change instruction from the user, and a change unit (not shown) that changes the information in response to the change instruction. The reception unit (not shown) may receive the change instruction from an input device, may receive information transmitted via a communication line, or may receive information read from a predetermined recording medium.

[0147] Also, in the above embodiment, when two or more components included in the mobile body 1 have a communication device, an input device, etc., the two or more components may have a physically single device, or may have separate devices.

[0148] In the above embodiment, each component may be configured by dedicated hardware, or a component that can be realized by software may be realized by executing a program. For example, each component may be realized by a program execution unit such as a CPU reading and executing a software program recorded on a recording medium such as a hard disk or a semiconductor memory. During execution, the program execution unit may execute the program while accessing the storage unit or the recording medium. The software for realizing the moving body 1 in the above embodiment is a program as follows. That is, this program may be a program for making a computer function as a specifying unit that specifies the position of the target to be followed in a local coordinate system of the moving body using the measurement results of a distance measuring sensor that measures the distance to surrounding objects in multiple directions of the moving body corresponding to the area of ​​the target to be followed identified in a captured image acquired by an image sensor that acquires a captured image of the surroundings of the moving body at the start of the movement of the moving body that moves by following the target to be followed, and a movement control unit that controls a moving mechanism that moves the moving body so as to follow the target to be followed identified by the specifying unit.

[0149] In addition, in the above program, the functions realized by the program do not include functions that can only be realized by hardware. For example, the functions realized by the program at least do not include functions that can only be realized by hardware in an acquisition unit that acquires information, a storage unit that stores information, an output unit that outputs information, etc.

[0150] This program may be executed by being downloaded from a server or the like, or may be executed by reading out a program recorded on a predetermined recording medium (for example, an optical disk such as a CD-ROM, a magnetic disk, a semiconductor memory, etc.). This program may be used as a program constituting a program product.

[0151] Furthermore, the program may be executed by a single computer or a plurality of computers. That is, the program may be executed by centralized processing or distributed processing.

[0152] 10 is a diagram showing an example of a computer system 901 that executes the above program to realize the moving object 1 according to the above embodiment. The above embodiment can be realized by computer hardware and a computer program executed thereon.

[0153] 10, computer system 901 includes MPU (Micro Processing Unit) 911, ROM 912 such as flash memory, RAM 913 connected to MPU 911 for temporarily storing instructions of application programs and providing a temporary storage space, hard disk 914, and bus 915 for interconnecting MPU 911, ROM 912, etc. Programs such as a boot-up program, application programs, system programs, and data may be stored in ROM 912 or hard disk 914. Programs are loaded into RAM 913 at the time of execution. Programs may be loaded directly from a network. Computer system 901 may also include a wireless communication module and an input device such as a touch panel.

[0154] The program does not necessarily include an operating system (OS) or a third party program that causes the computer system 901 to execute the functions of the mobile object 1 according to the above embodiment. The program may include only an instruction portion that calls appropriate functions or modules in a controlled manner to achieve a desired result. How the computer system 901 operates is well known, and a detailed description will be omitted.

[0155] The above-mentioned embodiments are merely examples for specifically implementing the present invention, and are not intended to limit the technical scope of the present invention. The technical scope of the present invention is indicated by the claims, not by the description of the embodiments, and is intended to include modifications within the literal scope of the claims and the scope of equivalent meanings. [Explanation of symbols]

[0156] 1. Mobile 11 Distance sensor 12 Image Sensor 13 Specific section 14 Storage section 15 Storage section 16 Moving mechanism 17 Movement control section 18 Output section

Claims

1. A moving object that moves while following a target to be followed, A distance measuring sensor for measuring distances to surrounding objects in multiple directions; an image sensor for capturing an image of the surroundings of the moving object; an identification unit that, when the moving body starts moving, identifies a position of the target to be tracked in a local coordinate system of the moving body using a measurement result of the distance measuring sensor corresponding to a region of the target to be tracked identified in a photographed image acquired by the image sensor, and, while the moving body is moving, identifies the position of the target to be tracked in the local coordinate system using the measurement result of the distance measuring sensor; A moving mechanism that moves the moving body; a movement control unit that controls the movement mechanism so as to follow the target identified by the identification unit; A storage unit; a storage unit that stores in the storage unit a latest position of the target identified by the identification unit, the movement control unit controls the movement mechanism to move along a route corresponding to the latest position of the tracking target stored by the storage unit when the identification unit cannot identify the position of the tracking target; The route is a route corresponding to the latest position of the target to be followed, among a plurality of routes preset in a local coordinate system of the moving body.

2. A computer that can access the storage unit, an identification unit that, when a moving body that moves while tracking a target to be tracked starts moving, identifies a position of the target to be tracked in a local coordinate system of the moving body using measurement results of a distance measuring sensor that measures distances to surrounding objects in multiple directions of the moving body, the distance measuring sensor measuring distances to surrounding objects in a region of the target to be tracked that is identified in an image captured by an image sensor that captures an image of the surroundings of the moving body, and, while the moving body is moving, identifies the position of the target to be tracked in the local coordinate system using the measurement results of the distance measuring sensor; a movement control unit that controls a movement mechanism that moves the moving object so as to follow the target identified by the identification unit; functioning as an accumulation unit that accumulates the latest position of the target to be tracked identified by the identification unit in the storage unit; the movement control unit controls the movement mechanism to move along a route corresponding to the latest position of the tracking target stored by the storage unit when the identification unit cannot identify the position of the tracking target; The route is a route corresponding to the latest position of the target to be tracked, among a plurality of routes preset in a local coordinate system of the moving body.

Images