Moving object, control method of moving object, and computer program

Abstract

To provide a moving object that can be moved without losing track of its next destination position.SOLUTION: A moving object comprises: travel route generation means for generating a travel route to a first destination; and detection means for detecting a second destination to be reached next after the first destination. The detection means can quickly move the moving object without losing track of the position of the second destination by successively detecting a position of the second destination while the moving object is traveling to the first destination based on the travel route, even if the second destination moves.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to a moving object, a control method for a moving object, a computer program, and the like. [Background technology]

[0002] Conventionally, there are known mobile objects that move autonomously in various places such as homes, office buildings, houses, logistics centers, etc. Such mobile objects can perform autonomous driving by generating a moving route according to the purpose from the output information of various sensors mounted on the mobile object.

[0003] One application of autonomous mobile objects is to follow people and support their daily lives and work. For example, Patent Literature 1 discloses a technology for generating a route to move while ensuring a field of view to a target to be followed. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2010-15194 A Summary of the Invention [Problem to be solved by the invention]

[0005] One use case of a mobile object supporting a person is when the object needs to bring something from a specific location (e.g., bring coffee from the kitchen). In this case, the mobile object needs to move to the first destination (the kitchen) and then generate a route to the second destination (the person giving the instructions).

[0006] However, the conventional technology disclosed in the above-mentioned Patent Document 1 discloses a technology for following a person, but does not consider the case where there are multiple destinations as described above. Therefore, if the instructor moves while the moving object is going to pick up the specified item, the instructor may be lost.

[0007] Therefore, an object of the present invention is to provide a mobile body that can move without losing sight of the position of its next destination. [Means for solving the problem]

[0008] A moving body according to one aspect of the present invention comprises: A driving route generating means for generating a driving route to a first destination; a detection means for detecting a second destination to be reached after the first destination, The detection means is characterized in that it sequentially detects the position of the second destination while traveling to the first destination based on the traveling route. Effect of the Invention

[0009] According to the present invention, it is possible to provide a mobile body that can move without losing sight of the position of its next destination. [Brief description of the drawings]

[0010] [Figure 1] 1 is a functional block diagram showing a configuration example of an autonomous driving system 150 that can be mounted on a moving body 1 according to a first embodiment. [Diagram 2] FIG. 2 is a diagram illustrating an example of a moving path of a moving object 1. [Diagram 3] 5 is a flowchart illustrating an example of a method for controlling a moving object by a travel route generating unit 43 in the first embodiment. [Figure 4] This is a continuation of the flowchart in Figure 3. [Diagram 5] 11A and 11B are diagrams illustrating an example of a method for calculating a visual field margin. [Figure 6] 1 is a schematic diagram showing a configuration example of a moving object 1 and a monitoring device 2 arranged on the moving object 1 in an embodiment 1. FIG. [Figure 7] 1A and 1B are diagrams illustrating the optical characteristics of the optical system. [Figure 8] 2 is a functional block diagram showing an example of the configuration of a monitoring device 2 in the first embodiment. FIG. [Figure 9]10 is a flowchart illustrating an operation of a travel route generating unit 43 in the second embodiment. [Figure 10] 10 is a continuation of the flowchart in FIG. 9. [Figure 11] This is a continuation of the flowchart in Figure 10. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. However, the present invention is not limited to the following embodiment. In the drawings, the same members or elements are given the same reference numerals, and duplicated descriptions thereof will be omitted or simplified.

[0012] <Embodiment 1> Fig. 1 is a functional block diagram showing a configuration example of an autonomous driving system 150 that can be mounted on a moving body 1 according to embodiment 1. Some of the functional blocks shown in Fig. 1 are realized by causing a CPU or the like as a computer (not shown) included in the moving body 1 to execute a computer program stored in a memory (not shown) as a storage medium.

[0013] However, some or all of them may be realized by hardware. A dedicated circuit (ASIC) or a processor (reconfigurable processor, DSP) may be used as the hardware. Furthermore, the functional blocks shown in Fig. 1 do not have to be built in the same housing, and may be configured by separate devices connected to each other via signal paths. The above explanation for Fig. 1 also applies to Fig. 8.

[0014] The autonomous driving system 150 includes a monitoring device 2 that captures images of the surroundings of the moving body, a map data generating device 3, and an autonomous driving control device 4. The moving body 1 is equipped with at least a part of the autonomous driving system 150 and is capable of autonomous driving.

[0015] In this embodiment, the moving object 1 performs the following operation upon receiving a predetermined instruction from an instructor (user): moving to a predetermined location (first destination), and then moving to the instructor (second destination). The moving object 1 is, for example, a mobile robot such as an AMR (Autonomous Mobile Robot) or an AGV (Automated Guided Vehicle).

[0016] The moving body 1 of the present embodiment can be used as a robot that monitors the surrounding situation while assisting human actions, particularly in the home, such as serving food and carrying furniture and objects, etc. However, the moving body of the present embodiment may be a drone or a car, and is not limited to the AMR or AGV described above.

[0017] The autonomous driving system 150 includes, as hardware, a calculation unit such as a CPU as a computer, and a storage unit as a storage medium such as a RAM, a ROM, a flash memory, or a hard disk drive.

[0018] These hardware elements are not shown in the drawings. The storage unit mainly stores programs required for the moving object 1 to perform autonomous driving, mainly in the ROM, and necessary data is mainly stored in the flash memory and the hard disk drive.

[0019] The autonomous driving system 150 may also include a communication device that communicates with an external device. Instead of or in addition to the CPU, a programmable logic device (PLD) or an application specific integrated circuit (ASIC) may be used.

[0020] Alternatively, a DSP (Digital Signal Processor) may be used. The operation of the monitoring device may be controlled by the CPU or the like, or may be controlled by a controller including a CPU, an MPU, or the like that is provided separately from the CPU.

[0021] 1, at least one of the blocks other than the monitoring device 2 and the driving unit 45 may be mounted on a device (a remote device or a server) other than the moving object 1. In this case, the moving object 1 and the remote device or server may be configured to be able to communicate mainly wirelessly.

[0022] The monitoring device 2 is a camera composed of an imaging element, an optical system, etc., and captures images of the surroundings of the moving object 1 and transmits the images as video signals to the map data generating device 3 and the autonomous driving control device 4. An example of the configuration of the monitoring device 2 will be described later.

[0023] The map data generating device 3 includes a distance measuring sensor 31 and a map creating unit 32. The distance measuring sensor 31 is a sensor for measuring the distance to an object, and uses, for example, a stereo camera or a LiDAR (Light Detection and Ranging).

[0024] The map creation unit 32 creates a map 33 required for estimating the self-position and orientation and generating a route based on at least one of the output image from the monitoring device 2 and the measurement values ​​of the distance measuring sensor 31.

[0025] The map 33 is three-dimensional spatial data such as three-dimensional point cloud data representing the characteristics of structures such as objects existing in space, floors, walls, and ceilings of buildings, and coordinate data of characteristic points. The map data generating device 3 can update the map 33 at any time while the mobile object 1 is traveling.

[0026] The autonomous driving control device 4 is a device that is at least partially mounted on the moving body 1 and drives the moving body 1, and has a detection unit 41, a self-position and attitude estimation unit 42, a driving path generation unit 43, a moving body control unit 44, and a drive unit 45.

[0027] The detection unit 41 sequentially detects objects existing around the moving body 1 by image recognition based on the output image of the monitoring device 2. The objects to be detected are, for example, objects that are obstacles to the moving body 1's travel, such as a desk, a chair, or a person, or objects that are the second destination. The detection unit 41 transmits to the travel path generation unit 43 the type of the detected object (whether it is an obstacle or a destination) as well as the position of each object on the image.

[0028] The detection unit 41 detects an instruction from an instructor wirelessly or from an image. For example, an instruction based on a body movement (hereinafter, referred to as a gesture) of a person present around the moving object 1 may be detected from an output image of the monitoring device 2.

[0029] In this case, the detection unit 41 stores information regarding a plurality of gestures and the instructions associated with them in advance, and determines whether a human gesture detected from the output image of the monitoring device 2 matches the gesture information previously recorded in a recording unit (not shown).

[0030] For example, the detection unit 41 determines that the instructor is making a gesture of standing still for a certain period of time while pointing at a specific location (first destination). In that case, it determines that the instruction is to first move to the specific location (first destination), receive a specific package, and then return to the instructor (second destination).

[0031] Furthermore, when the detection unit 41 determines that a person is making a gesture motion of raising a hand toward the moving body 1 and standing still for a certain period of time, it determines that an instruction to move to the instructor has been received. Note that the means for detecting an instruction from an instructor is not limited to a gesture, and for example, the instruction may be detected by detecting the instructor's voice or a predetermined sound from a microphone (not shown) and recognizing that an instruction that has been stored in advance in association with the voice has been received. The detection unit 41 transmits the detected instruction to the travel route generation unit 43.

[0032] Furthermore, the detection unit 41 has a function of detecting a specific object or person from the output image of the monitoring device 2. For example, the detection unit 41 identifies a person who has given an instruction by gesture by referring to person data recorded (registered) in advance in a recording unit (not shown). This makes it possible to recognize who gave what instruction even if there are multiple people who give instructions to the moving object 1.

[0033] While the moving body 1 is moving, the self-position and attitude estimation unit 42 estimates the position and attitude of the moving body 1 on which the autonomous driving control device 4 is mounted, using the output image from the monitoring device 2, the measurement values ​​of the ranging sensor 31, and the map 33.

[0034] The self-position and orientation estimation unit 42 extracts characteristics of structures around the moving body 1 based on, for example, the output image from the monitoring device 2 and the measurement values ​​of the distance measuring sensor 31, and estimates its own position and orientation by comparing them with the map 33.

[0035] The driving route generation unit 43 generates a driving route based on the measurement values ​​of the distance measurement sensor 31 , the self-position and orientation estimation result by the self-position and orientation estimation unit 42 , the map information of the map 33 , and the detection result of the detection unit 41 .

[0036] Based on the instruction detected by the detection unit 41, the driving route generating unit 43 first generates a driving route to the first destination. That is, the driving route generating unit 43 functions as a driving route generating means for generating a driving route to the first destination. At this time, the driving route generating unit 43 sequentially generates a route so that the second destination is included in the shooting range as much as possible. The operation of the driving route generating unit 43 will be described in detail later.

[0037] The moving body control unit 44 controls the moving direction, speed, etc. of the moving body 1 equipped with the autonomous driving control device 4 based on the driving route generated by the driving route generation unit 43. The drive unit 45 drives the wheels, etc. of the moving body 1 equipped with the autonomous driving control device 4 based on the control from the moving body control unit 44.

[0038] The map 33 can be accessed from both the map data generation device 3 and the autonomous driving control device 4, and may be stored in either the map data generation device 3 or the autonomous driving control device 4, or may be stored in both.

[0039] In addition, since the moving object 1 can be used to transport luggage, it may be provided with luggage detection means (not shown) for detecting the type and condition of luggage being transported, and may be configured to transmit the luggage loading status to the travel route generation unit 43. The luggage detection means may be configured, for example, by a pressure sensor (not shown).

[0040] Alternatively, the detection unit 41 may be configured to detect that a target baggage has been loaded from the output image of the monitoring device 2. When baggage detection is realized from the image, the monitoring device 2 is installed so that the location where the baggage is loaded is included in the imaging range of the monitoring device 2.

[0041] Next, a specific example of the movement path of the moving object 1 will be described. Fig. 2 is a diagram for explaining one example of the movement path of the moving object 1. Here, it is assumed that an instructor instructs the moving object 1 to acquire (pick up) an object at a first destination G1, and then transport it to a second destination, which is the instructor. At this time, a case will be described in which, while the moving object 1 is moving to the first destination G1, the second destination G2, which is the instructor, moves to a position p3 via a route tr1.

[0042] First, the moving object 1 generates a route tr2 to a first destination G1 based on an instruction such as a gesture from an instructor detected by the detection unit 41, and starts moving along the route tr2.

[0043] When the moving object 1 reaches the position p1, the second destination G2, which is the indicator, is assumed to have moved to the position p2. At this time, the second destination G2 at the position p2 seen by the moving object 1 from the position p1 will soon be hidden behind the obstacle b1 and will no longer be visible.

[0044] In this embodiment, the moving body 1 generates a route again to keep the second destination G2 in view, generates a route tr3, and starts moving along the route tr3. This operation allows the moving body 1 to continuously grasp the position of the second destination G2, which is the indicator, even while moving to the first destination G1.

[0045] After the mobile unit 1 arrives at the first destination G1, the mobile unit 1 generates a route tr4 to the second destination G2 based on the position of the second destination G2 that it has already grasped, and moves along this route. Note that it is assumed that G2 has moved to position p3 at this point.

[0046] Next, a process flow for generating a route along which the moving object 1 can move while continuing to grasp the position of the second destination will be described.

[0047] Fig. 3 is a flowchart for explaining an example of a control method of a moving object by the driving route generating unit 43 in the embodiment 1, and explains an example of an operation performed by the driving route generating unit 43 when an instruction is given by an instructor to move to a second destination after moving to a first destination. Fig. 4 is a continuation of the flowchart in Fig. 3.

[0048] The operations of the steps in the flowcharts of FIG. 3 and FIG. 4 are performed in sequence by a CPU serving as a computer within the autonomous driving system 150 executing a computer program stored in a memory.

[0049] First, in step S01, the travel route generating unit 43 waits for an instruction from an instructor transmitted from the detecting unit 41. When the second destination in the instruction from the instructor is a destination to which the vehicle does not move (e.g., a predetermined place on a map), the travel route generating unit 43 may generate an optimal route (e.g., the shortest route) to the first destination and travel along the optimal route without following the flow of FIG.

[0050] That is, when the second destination is, for example, a destination that does not move, or when the second destination is an object whose movement trajectory is known in advance, a driving route to the first destination that does not depend on the margin of visibility may be generated. The margin of visibility will be described later.

[0051] In step S01, when the driving route generating unit 43 receives an instruction from the detection unit 41, in step S02, the driving route generating unit 43 generates a route to the first destination.

[0052] Specifically, the position of the first destination based on an instruction from the detection unit 41 is grasped from map information obtained from the map 33, and a route to the first destination is generated in combination with information on the self-position on the map obtained from the self-position and orientation estimation unit 42. Here, step S02 functions as a driving route generation step (driving route generation means) that generates a driving route to the first destination.

[0053] Next, in step S03, the travel route generation unit 43 instructs the mobile object control unit 44 to travel along the generated travel route to the first destination. In step S04, the travel route generation unit 43 grasps the positions of the second destination and obstacles on the map.

[0054] That is, the positions of the second destination and obstacles on the map are grasped based on the positions of the second destination and obstacles on the image detected by the detection unit 41, the three-dimensional spatial information obtained from the map 33, and the estimation results of the self-position and orientation on the map obtained from the self-position and orientation estimation unit 42.

[0055] Here, step S04 functions as a detection step (detection means) for detecting a second destination to be reached after the first destination, and the detection step sequentially detects the position of the second destination while traveling to the first destination based on the traveling route.

[0056] Next, in step S05, the driving route generating unit 43 calculates the visual field margin to the second destination based on the calculated positions of the second destination and obstacles on the map. That is, the driving route generating unit 43 as a driving route generating means calculates the visual field margin to the second destination. Furthermore, it is determined whether the visual field margin to the second destination is equal to or smaller than a predetermined value.

[0057] Here, the calculation of the visual field margin in this embodiment will be described. Fig. 5 is a diagram for explaining an example of a method for calculating the visual field margin. Based on the second destination G2, obstacle b1, and its own position information detected in step S04, the travel route generation unit 43 measures an angle a1 formed between the direction of the second destination G2 as seen from the moving body 1 and the direction of the edge of the obstacle b1 closest to the moving body 1 as seen from the moving body 1. Then, this angle a1 is set as the visual field margin.

[0058] At this time, if another obstacle b2 exists, the angles a2 and a1 measured in the same manner are compared, and the smaller angle a1 is set as the visual field margin. Note that if the edge of the obstacle is closer to the moving body 1 than the second destination G2, there is a possibility that it will obstruct the view to the second destination G2, so the visual field margin is calculated.

[0059] On the other hand, if the edge of the obstacle is farther away from the second destination G2 as viewed from the moving body 1, it does not obstruct the view to the second destination G2 and is therefore excluded from the calculation of the visual field margin. If the determination in step S05 is YES, the process proceeds to step S07, and if the determination is NO, the process proceeds to step S06.

[0060] If the result of the determination in step S05 is NO, that is, if the visual field margin to the second destination is greater than the predetermined value, in step S06, the travel route generation unit 43 instructs the mobile object control unit 44 to continue travel on the current travel route. Then, the process proceeds to step S08.

[0061] On the other hand, if the determination in step S05 is YES, that is, if the visibility to the second destination is equal to or less than the predetermined value, in step S07, the driving route generation unit 43 regenerates a route to reach the first destination.

[0062] That is, the driving route is regenerated so as to head in a direction in which the visual field margin calculated in step S05 is increased, and the driver is instructed to travel along the regenerated driving route. In this manner, in this embodiment, when the visual field margin becomes equal to or smaller than a predetermined value, the driving route to the first destination is regenerated so as to increase the visual field margin to the second destination.

[0063] Also, the regenerated travel route may be limited so that the regenerated route falls within a predetermined time or a predetermined travel distance. That is, a travel route that can reach the first destination within a predetermined time or a predetermined travel distance may be regenerated. After step S07, the process proceeds to step S08. Next, in step S08, the travel route generation unit 43 determines whether the first destination has been reached.

[0064] If the result in step S08 is NO, the driving route generating unit 43 returns to step S02 and continues the processes from step S02 to step S07 until the first destination is reached. If the result in step S08 is YES, the process proceeds to S16 in FIG.

[0065] In step S16, the travel route generation unit 43 causes the moving object 1 to pick up (receive) the package based on the instruction from the instructor at the first destination using a pickup mechanism (not shown).

[0066] Here, when the baggage detection means detects that the predetermined items have been loaded, or when an instruction that loading is complete is received from a person near the first destination, it is determined that receipt of the predetermined items has been completed, and the process proceeds to step S09.

[0067] In step S09, even after the first destination is reached, the driving route generating unit 43 grasps the positions of the second destination and obstacles. Then, in step S10, the driving route generating unit 43 generates a route to the second destination based on the detected positions of the second destination and obstacles, and in step S11, the mobile object control unit 44 issues a driving instruction to the second destination.

[0068] If there is an instruction to interrupt the moving object 1, the driving route generating unit 43 may interrupt the driving to the second destination and perform route generation processing based on the interrupted instruction. For example, assume that while the moving object 1 is moving to the second destination, a different instructor B from the instructor (instructor A) at the second destination instructs the moving object 1 to pick up a specific item by a gesture of raising his / her hand.

[0069] In this case, when the detection unit 41 detects such an instruction while traveling, it transmits an interrupt instruction based on the content of the instruction to the travel route generation unit 43. In step S13, the travel route generation unit 43 constantly determines whether or not there is an interrupt instruction from the detection unit 41. If the result in step S13 is NO, the process proceeds to step S12.

[0070] If the answer is YES in step S13, that is, if it is determined that an interrupt instruction has been given, the process proceeds to step S14. In step S14, the travel route generation unit 43 interrupts route generation based on instructions from instructor A, and generates a route based on instructions from instructor B and issues travel instructions to the mobile body control unit 44.

[0071] In step S15, the travel route generating unit 43 determines whether or not the instruction from the instructor B has been completed, and if NO, returns to step S14 and repeats steps S14 and S15. If YES in step S15, proceeds to step S12.

[0072] In step S12, the travel route generating unit 43 resumes the interrupted instruction from the instructor A, generates a route to the instructor A (second destination) again by the travel route generating unit 43, and resumes the movement. Also, in step S12, it is determined whether the second destination has been reached.

[0073] If the result is NO in step S12, the process returns to step S09, and the processes in steps S09 to S12 are repeated. If the result is YES in step S12, the flow in Fig. 4 is terminated. Note that the flows in Figs. 3 and 4 are executed periodically.

[0074] 3 and 4, the travel route generation unit 43 of this embodiment allows the moving body 1 to travel along a route that allows the moving body 1 to travel to the first destination while detecting the position of the second destination designated by the moving body 1. Therefore, even if the second destination to which the moving body 1 is scheduled to travel after arriving at the first destination moves while the moving body 1 is traveling to the first destination, the moving body 1 can move quickly without losing sight of the position of the second destination.

[0075] Next, a description will be given of a preferred embodiment regarding the placement of the monitoring device 2 and a detailed configuration example of the monitoring device 2. As described above, when the moving object 1 is monitoring a second destination while heading toward a first destination, it is desirable for the monitoring device 2 to be able to capture images of the surroundings of the moving object 1 in all directions.

[0076] In addition, it is desirable that the monitoring device 2 be able to capture images with high resolution and low distortion in the horizontal direction so that the detection unit 41 can detect the instructor from the captured images even if the moving object 1 moves to a position away from the instructor. Based on this, a specific example of a preferable arrangement of the monitoring device 2 and a configuration of the monitoring device 2 in this embodiment will be described.

[0077] FIG. 6 is a schematic diagram showing an example configuration of a moving body 1 and a monitoring device 2 arranged on the moving body 1 in embodiment 1, where the monitoring device 2 has an imaging device 21 as a first imaging device and an imaging device 22 as a second imaging device.

[0078] The imaging device 21 is installed at a first position so that the optical axis of the optical system included in the imaging device 21 faces upward. The upward direction is, for example, toward the sky or the ceiling. The imaging device 22 is installed at a second position so that the optical axis of the optical system included in the imaging device 22 faces downward, which is the opposite direction to the optical axis of the imaging device 21. The downward direction is, for example, toward the ground or floor.

[0079] That is, the imaging devices 21 and 22 are arranged so that the optical axes of the respective optical systems are directed in opposite directions to each other. The optical axes being directed in opposite directions to each other means, for example, that the angle between the optical axes is 180°±α (for example, α=0° to 10°, etc.).

[0080] That is, imaging device 21 is positioned so as to capture a field of view in a first direction, and imaging device 22 is positioned in the opposite direction to imaging device 22 so as to capture a field of view in a second direction opposite to the first direction.

[0081] The optical axes of the imaging devices 21 and 22 are arranged so as to overlap on a substantially same straight line. By arranging the imaging devices 21 and 22 in this manner, it is possible to reduce positional deviation between images when the image synthesis unit 25 described later synthesizes the images captured by the imaging devices 21 and 22.

[0082] Incidentally, the imaging devices 21 and 22 are installed so that the optical axes are oriented in the opposite directions as described above, but it is desirable to place them as close as possible to each other so that the installation heights of the first and second positions are approximately the same. By installing the imaging devices 21 and 22 in this way, it becomes possible to capture images of the ranges to the sides adjacent to the monitoring device 2, thereby reducing blind spots.

[0083] 6, the imaging devices 21 and 22 are installed in close proximity to each other by being attached so as to sandwich the attachment portion 5a provided at the tip of the support portion 5. However, this is not limited to the above, and the imaging devices 21 and 22 may be supported so as to be in contact with each other. "In close proximity" also includes the imaging devices 21 and 22 being in direct contact with each other in this manner.

[0084] In addition, the optical systems of the imaging devices 21 and 22 in this embodiment generate an optical image (image circle) having a characteristic that the resolution of a second region on the outer periphery of a first region including the center of the optical system through which the optical axis passes is higher than the resolution of a first region.

[0085] The generated optical image is then focused on a light receiving surface of an imaging element, which will be described later. In this embodiment, the imaging device 21 and the imaging device 22 are assumed to have substantially the same configuration, and the configuration of the imaging device 21 will be described below as an example. However, this embodiment is not limited to the imaging device 21 and the imaging device 22 having substantially the same configuration.

[0086] As described below, the imaging device 21 includes an optical system 21a and an imaging element 21b. The optical system 21a has different imaging magnifications at a first angle of view 10a (first field of view) and a second angle of view 10b (second field of view) that is outside (on the peripheral side) of the first angle of view 10a.

[0087] As described below, the imaging device 22 includes an optical system 22a and an imaging element 22b. The optical system 22a has different imaging magnifications at a first angle of view 11a (first field of view) and a second angle of view 11b (second field of view) that is outside (on the peripheral side) of the first angle of view 11a.

[0088] The imaging surface (light receiving surface) of the imaging element 21b includes a first region for imaging an object included in the first angle of view 10a and a second region for imaging an object included in the second angle of view 10b. At this time, the number of pixels per unit angle of view in the second region is greater than the number of pixels per unit angle of view in the first region. That is, the resolution of the imaging device 21 at the second angle of view (second region) is higher than the resolution at the first angle of view (first region).

[0089] Thus, in this embodiment, the imaging device 21 captures an image in the upward direction (vertically upward direction) and operates as a first imaging device having a higher resolution at the peripheral angle of view than at the central angle of view, while the imaging device 22 captures an image in the downward direction (vertically downward direction) and operates as a second imaging device having a higher resolution at the peripheral angle of view than at the central angle of view.

[0090] The combined angle of view of the first angle of view 10a and the second angle of view 10b corresponds to the imaging angle of view of the imaging device 21, which is 180° or more. The imaging angle of view of the imaging device 22 is also 180° or more, and the imaging angles of view of the imaging devices 21 and 22 overlap in part at the periphery. Therefore, the surroundings of the moving object 1 can be captured in a wide range (360° in this embodiment) without interruption.

[0091] 7(A) and (B) are diagrams for explaining the optical characteristics of the optical system, and the optical characteristics of the optical system 21a will be explained with reference to Fig. 7(A) and Fig. 7(B). Fig. 7(A) is a diagram showing the image height y at each half angle of view on the imaging surface (light receiving surface) of the imaging element 21b in the form of contour lines. Fig. 7(B) is a diagram showing the relationship between the half angle of view θ and the image height y (projection characteristics of the optical system 21a) in the first quadrant of Fig. 7(A).

[0092] 7B, the optical system 21a is configured such that its projection characteristic y(θ) differs between angles of view less than a predetermined half angle of view θa and angles of view equal to or greater than the half angle of view θa. Therefore, the optical system 21a is configured such that, when the increase in image height y per unit of half angle of view θ is defined as the resolution, the resolution differs depending on the angle of view (the area on the light receiving surface of the image sensor).

[0093] This local resolution can be expressed as the differential value dy(θ) / dθ of the projection characteristic y(θ) at the half angle of view θ. For example, the greater the slope of the projection characteristic y(θ) in Fig. 7B, the higher the resolution. Also, Fig. 7A shows that the greater the interval between the contour lines of the image height y at each half angle of view, the higher the resolution.

[0094] The optical system 21a of this embodiment has a projection characteristic in which the rate of increase of the image height y (the slope of the projection characteristic y(θ) in FIG. 7B) is small in the central region near the optical axis, and the rate of increase of the image height y becomes larger as the angle of view becomes larger in the peripheral region outside the central region.

[0095] 7A, a first region 21c including the optical axis center corresponds to an angle of view less than the half angle of view θa, and a second region 21d outside the first region corresponds to an angle of view equal to or greater than the half angle of view θa. The angle of view less than the half angle of view θa corresponds to the first angle of view 10a in FIG. 6, and the angle of view equal to or greater than the half angle of view θa corresponds to the second angle of view 10b in FIG. 6.

[0096] 6. Similarly, an angle of view less than the half angle of view θa of the optical system 22a of the imaging device 22 corresponds to the first angle of view 11a in FIG. 6, and an angle of view equal to or greater than the half angle of view θa corresponds to the second angle of view 11b in FIG.

[0097] As described above, the first region 21c is a relatively low-resolution region, and the second region 21d is a relatively high-resolution region. In addition, the first region 21c is a high-distortion region with relatively more distortion, and the second region 21d is a low-distortion region with relatively less distortion.

[0098] Therefore, in this embodiment, the first region 21c may be called a low-resolution region or a high-distortion region, and the second region 21d may be called a high-resolution region or a low-distortion region.

[0099] 7A and 7B are merely examples, and the present invention is not limited to these. For example, the low-resolution and high-resolution regions of the optical system do not have to be concentric, and each region may have a distorted shape.

[0100] The center of gravity of the low resolution region and the center of gravity of the high resolution region do not have to coincide with each other. The center of gravity of the low resolution region and the center of gravity of the high resolution region may be offset from the center of the light receiving surface of the image sensor. In the optical system of this embodiment, it is sufficient that the low resolution region is formed near the optical axis, and the high resolution region is formed on the peripheral side near the optical axis.

[0101] The value of the half angle of view θa can be set appropriately. That is, there is no clear boundary between the first region 21c and the second region 21d. In this embodiment, even in the first region 21c, the resolution increases from the center side through which the optical axis passes toward the outer periphery.

[0102] The same is true in the second region 21d. Alternatively, in the graph of the projection characteristics shown in Fig. 7(B), at least a part of the first region 21c and / or at least a part of the second region 21d may be linear (with constant resolution).

[0103] The optical system 21a of this embodiment is configured to satisfy the following formula 1, where f is the focal length, θ is the half angle of view, y is the image height on the image plane, y(θ) is a projection characteristic that indicates the relationship between the image height y and the half angle of view θ, and θmax is the maximum half angle of view of the optical system. That is, the optical system 21a is configured so that the projection characteristic y(θ) is different from 2ftan(θ / 2) (stereoscopic projection method). 0.2<2×f×tan(θmax / 2) / y(θmax)<0.92 (Formula 1)

[0104] In an optical system with such optical characteristics, the magnification in the radial direction with respect to the optical axis can be adjusted by adjusting the projection characteristic y(θ). This allows the aspect ratio in the radial direction and circumferential direction with respect to the optical axis to be controlled, so that, unlike conventional fisheye lenses, it is possible to obtain a high-resolution image with little distortion in the peripheral area even with a wide angle of view.

[0105] Moreover, by satisfying formula 1, the resolution in the second region 21d can be made higher than that of a stereoscopic projection optical system. If the upper limit of formula 1 is exceeded, the resolution in the second region 21d decreases, and the difference with the resolution in the first region 21c becomes small, which is not preferable. If the lower limit of formula 1 is exceeded, it becomes difficult to satisfactorily correct various aberrations such as field curvature, which is also not preferable.

[0106] However, the above-mentioned formula 1 is just one example, and the optical system in this embodiment is not limited to this.

[0107] In this embodiment, by configuring the optical system as described above, high resolution can be obtained in the high resolution region, while the increase in image height y per unit half angle of view θ can be reduced in the low resolution region, making it possible to capture a wider angle of view. Therefore, while the imaging range is a wide angle of view equivalent to that of a fisheye lens, high resolution can be obtained in the high resolution region.

[0108] Next, a configuration of the monitoring device 2 will be described. Fig. 8 is a functional block diagram showing an example of the configuration of the monitoring device 2 in embodiment 1. As shown in Fig. 8, the monitoring device 2 has imaging devices 21 and 22, image processing units 23 and 24, and an image synthesis unit 25. However, functional blocks other than the imaging devices 21 and 22 may be provided in, for example, an external device or an external server.

[0109] The imaging device 21 is composed of an optical system 21a and an imaging element 21b. The imaging device 22 is composed of an optical system 22a and an imaging element 22b. The optical systems 21a and 22a of the imaging devices 21 and 22 are each composed of one or more optical lenses, and each has the optical characteristics shown in Figures 7(A) and (B).

[0110] The optical systems 21a and 22a form (focus) their respective optical images on the light receiving surfaces of the image pickup elements 21b and 22b. The image pickup elements 21b and 22b are implemented by using two-dimensional image sensors such as complementary metal oxide semiconductors (CMOS), and perform photoelectric conversion on the focused optical images to output image pickup signals.

[0111] In addition, on the light receiving surfaces of the imaging elements 21b and 22b, for example, R, G, and B color filters are arranged for each pixel in a Bayer array, and R, G, and B pixel signals are sequentially output from the imaging devices 21 and 22, respectively, as imaging signals.

[0112] The image processing units 23 and 24 perform various image processing such as white balance adjustment, demosaic processing, gain / offset adjustment, gamma correction, and color correction on the imaging signals output from the imaging devices 21 and 22, respectively.

[0113] The image synthesis unit 25 functions as a synthesis means for creating a synthetic image by synthesizing the images captured by the imaging devices 21 and 22 into one image after panorama expansion of the images. The panorama expansion method may be based on the equirectangular projection or the Mercator projection.

[0114] With the above configuration, the monitoring device 2 can seamlessly capture images in all directions around the moving object 1, and can capture high-resolution, low-distortion images in the horizontal direction. This enables the moving object 1 to accurately detect distant destinations and gestures of instructors, enabling it to travel over a wider range. <Embodiment 2>

[0115] In the first embodiment, a configuration has been described in which the position of the second destination can be detected even while the moving object 1 is moving to the first destination. However, depending on the position of an obstacle or the position of the second destination, the moving object 1 may not be able to reach the first destination unless the second destination is removed from the field of view of the moving object 1.

[0116] In light of the above, an operation when the moving object 1 is unable to detect the position of the second destination will be described in embodiment 2. Note that the configuration of the moving object 1 in embodiment 2 is the same as that of embodiment 1 shown in FIG.

[0117] In the second embodiment, when the moving object 1 loses sight of the second destination, the position where it lost sight (for example, the current location of the moving object 1, the last detected second destination, or the position of an obstacle that blocks the view to the second destination, etc.) is recorded in a storage unit. Then, after the moving object 1 reaches the first destination, it moves to the position where it lost sight and waits. Note that if the second destination is detected again, the same flow as in the first embodiment may be repeated.

[0118] FIG. 9 is a flowchart for explaining the operation of the driving route generating unit 43 in the second embodiment, FIG. 10 is a continuation of the flowchart in FIG. 9, and FIG. 11 is a continuation of the flowchart in FIG.

[0119] The operations of the steps in the flowcharts of FIGS. 9 to 11 are performed in sequence by a CPU serving as a computer in the autonomous driving system 150 executing a computer program stored in a memory.

[0120] 9 to 11 are flowcharts for explaining the operation of the travel route generating unit 43 in the case where the moving object 1 is traveling to a first destination and is unable to detect the position of a second destination to which the moving object 1 is subsequently headed. Note that the processing contents in the steps having the same reference numbers as in Figs. 3 and 4 are the same as those in the first embodiment, and therefore will not be explained here.

[0121] In Fig. 9, the driving route generating unit 43 is configured to detect a situation in which the second destination is lost. That is, before determining the position of the second destination in step S04, in step S21, the driving route generating unit 43 determines whether the second destination has been detected from the captured image by the detection unit 41. If the result in step S21 is NO, the process proceeds to step S23 in Fig. 10, and if the result in step S21 is YES, the process proceeds to step S04.

[0122] Also, if the answer is YES in step S05, that is, if the visual field margin to the second destination is equal to or less than a predetermined value, in step S22, the driving route generation unit 43 determines whether or not it is possible to regenerate a route to reach the first destination within a predetermined time or distance.

[0123] That is, it is determined whether or not there is a route that can widen the visual margin among the routes that reach the first destination within a predetermined time or distance. If the answer is YES in step S22, the process proceeds to step S07.

[0124] If the result of step S22 is NO, the process proceeds to step S23, as in the case of NO in step S21. In step S23, the driving route generating unit 43 determines that the second destination has been lost, and records the position on the map where the second destination has been lost in the recording unit as the third destination.

[0125] That is, when the detection means is no longer able to detect the second destination or when a driving route that increases the margin of visibility cannot be generated, a third destination is set and a driving route to the third destination is generated.

[0126] At this time, the location recorded as the third destination may be, for example, the current location of the moving body 1, the location of the last detected second destination, or both. Also, when the visual field margin is calculated in step S22, the travel route generating unit 43 may record the location of the obstacle used in the last visual field margin calculation. That is, the third destination includes any one of the current location of the moving body, the location of the last detected second destination, or the location of the obstacle used in the last visual field margin calculation.

[0127] In the next step S24, the travel route generation unit 43 instructs the moving object control unit 44 to travel to the first destination, and the moving object 1 starts traveling to the first destination.

[0128] Then, in step S25, the travel route generation unit 43 redetects the second destination during the movement to the first destination. If the result of step S25 is YES, that is, if the detection unit 41 detects the second destination again, the process returns to step S04, where the travel route generation unit 43 grasps the positions of the second destination and obstacles.

[0129] On the other hand, if the result in step S25 is NO, the process proceeds to step S26, where the travel route generation unit 43 determines whether the first destination has been reached. If the result in step S26 is NO, the process returns to step S24, and steps S24 to S26 are repeated until the result in step S26 is YES.

[0130] If step S26 returns YES, i.e., if it is determined that the first destination has been reached, the process proceeds to step S27. In step S27, the travel route generation unit 43 generates a route for moving to one of the third destinations recorded in step S23 and instructs the mobile body control unit 44 to travel, and the mobile body 1 starts moving to the third destination.

[0131] Then, in step S28, the travel route generation unit 43 redetects the second destination during the movement to the third destination. If the result of step S28 is YES, that is, if the detection unit 41 detects the second destination, the process proceeds to step S09, where the travel route generation unit 43 grasps the positions of the second destination and obstacles.

[0132] On the other hand, if the result in step S28 is NO, the process proceeds to step S29, where the travel route generation unit 43 determines whether or not the third destination has been reached. If the result in step S29 is NO, the process returns to step S27, and the processes in steps S27 to S29 are repeated until the result in step S29 is YES.

[0133] If the answer is YES in step S29, that is, after the third destination is reached, the flow of Fig. 10 ends, and the moving object 1 waits there. Alternatively, a voice output unit (not shown) may play a sound to inform the moving object that the destination has been lost. Alternatively, the process may not end, but continue to check for the detection of the second destination, and if the second destination is detected, the process may be resumed from step S09.

[0134] In the second embodiment, the flow described in Figures 9 to 11 allows the moving body 1 to continue moving toward the first destination even if it loses sight of the second destination, and after reaching the first destination, it becomes easier to detect the second destination again.

[0135] The third destination may be preset to a location with a wide field of view in accordance with the layout of the room, but by using the location where the second destination was last detected as the third destination, the second destination can be more likely to be redetected without any pre-setting.

[0136] Although the present invention has been described in detail above based on the preferred embodiments, the present invention is not limited to the above-mentioned embodiments, and various modifications and combinations of the above-mentioned embodiments are possible based on the spirit of the present invention, and are not excluded from the scope of the present invention. The present invention includes the following combinations.

[0137] (Configuration 1) A mobile body comprising: a driving route generation means for generating a driving route to a first destination; and a detection means for detecting a second destination to be reached after the first destination, wherein the detection means sequentially detects the position of the second destination while traveling to the first destination based on the driving route.

[0138] (Configuration 2) The moving body described in Configuration 1, characterized in that the driving route generation means calculates a visual field margin to the second destination, and when the visual field margin becomes equal to or less than a predetermined value, regenerates the driving route to the first destination so as to increase the visual field margin to the second destination.

[0139] (Configuration 3) The moving body according to configuration 2, wherein the travel route generating means regenerates the travel route that enables the moving body to reach the first destination within a predetermined time or a predetermined travel distance.

[0140] (Configuration 4) The moving body described in Configuration 3, characterized in that the driving route generation means sets a third destination and generates a driving route to the third destination when the detection means is no longer able to detect the second destination or when it is impossible to generate the driving route that increases the visual field margin.

[0141] (Configuration 5) The moving body described in Configuration 4, characterized in that the third destination includes one of the current location of the moving body, the last detected position of the second destination, or the position of an obstacle used in the last calculation of the visual field margin.

[0142] (Configuration 6) The moving body described in Configuration 2, characterized in that, when the second destination is a specified destination, the driving route generation means generates the driving route to the first destination independent of the visual field margin.

[0143] (Configuration 7) A moving body according to any one of configurations 1 to 6, further comprising a monitoring device that photographs the surroundings of the moving body, and the detection means sequentially detects the second destination based on an output image of the monitoring device.

[0144] (Configuration 8) The monitoring device of the moving body described in Configuration 7 is characterized in having a first imaging device that captures images in the upward direction and has a resolution at a peripheral angle of view that is higher than the high resolution at a central angle of view, and a second imaging device that captures images in the downward direction and has a resolution at a peripheral angle of view that is higher than the high resolution at a central angle of view.

[0145] (Configuration 9) When the image height of each optical system of the first imaging device and the second imaging device is y, the focal length is f, and the half angle of view is θ, the projection characteristic y(θ) is expressed as follows: 0.2<2×f×tan(θmax / 2) / y(θmax)<0.92 The moving body according to aspect 8, wherein the above-mentioned conditions are satisfied.

[0146] (Method) A method for controlling a moving body, comprising: a driving route generation step for generating a driving route to a first destination; and a detection step for detecting a second destination to be reached after the first destination, wherein the detection step sequentially detects the position of the second destination while traveling to the first destination based on the driving route.

[0147] (Program) A computer program for controlling each means of the moving object according to any one of configurations 1 to 9 by a computer.

[0148] In addition, in order to realize some or all of the controls in the first and second embodiments, a computer program that realizes the functions of the above-mentioned embodiments may be supplied to a moving body, an autonomous driving system, etc. via a network or various storage media.

[0149] The computer (or CPU, MPU, etc.) in the moving body or autonomous driving system may read and execute the program. In this case, the program and the storage medium storing the program constitute the present invention. [Explanation of symbols]

[0150] 1: Moving object 2: Monitoring device 3: Map data generator 4: Autonomous driving control device 31: Distance sensor 32: Map Creation Department 33: Map 41: Detection section 42:Self-position and orientation estimation unit 43: Route generation unit 44: Mobile control unit 45: Drive unit

Claims

1. a travel route generating means for generating a travel route to a first destination; a detection means for detecting a second destination to be reached after the first destination, The moving body, wherein the detection means successively detects a position of the second destination while the moving body is traveling to the first destination based on the travel route.

2. The driving route generation means calculates a visibility margin to the second destination, 2. The moving body according to claim 1, wherein, when the visual field margin becomes equal to or smaller than a predetermined value, the travel route to the first destination is regenerated so as to increase the visual field margin to the second destination.

3. 3. The mobile body according to claim 2, wherein the travel route generation means regenerates the travel route that enables the first destination to be reached within a predetermined time or a predetermined travel distance.

4. The moving body according to claim 3, characterized in that the driving route generation means sets a third destination and generates a driving route to the third destination when the detection means is no longer able to detect the second destination or when it is impossible to generate the driving route that increases the visual margin.

5. 5. The vehicle according to claim 4, wherein the third destination includes one of the current location of the vehicle, the last detected location of the second destination, or the location of an obstacle used in the last calculation of the visibility margin.

6. 3. The vehicle according to claim 2, wherein the driving route generating means generates the driving route to the first destination independent of the visual margin when the second destination is a predetermined destination.

7. A monitoring device is provided for capturing images of the surroundings of the moving object, 2. The vehicle according to claim 1, wherein said detection means successively detects said second destination based on an output image of said monitoring device.

8. The monitoring device includes: a first imaging device that captures an image in an upward direction and has a higher resolution at a peripheral angle of view than a high resolution at a central angle of view; a second imaging device that captures an image in a downward direction and has a higher resolution at a peripheral angle of view than a high resolution at a central angle of view; 8. The moving body according to claim 7, further comprising:

9. When the image height of each optical system of the first imaging device and the second imaging device is y, the focal length is f, and the half angle of view is θ, the projection characteristic y(θ) is expressed as follows: 0.2<2×f×tan(θmax / 2) / y(θmax)<0.92 9. The moving body according to claim 8, wherein the above expression is satisfied.

10. a travel route generating step of generating a travel route to a first destination; a detection step of detecting a second destination to be reached after the first destination, A method for controlling a moving body, wherein the detection step successively detects a position of the second destination while the moving body is traveling to the first destination based on the travel route.

11. A computer program for controlling each means of a moving body according to any one of claims 1 to 9 by a computer.

Images