Control device for a moving body, control method for a moving body, and storage medium

Abstract

According to the present invention, a control device for a mobile body, which can move on both a roadway and in a predetermined area different from the roadway, comprises: a road type recognizing unit that respectively sets, in a captured image of an external camera that captures images in the travel direction of the mobile body, a center virtual lane that includes an expected travel path of the mobile body, a right side virtual lane that is present to the right of the center virtual lane as seen from the mobile body, and a left side virtual lane that is present to the left of the center virtual lane as seen from the mobile body, and recognizes whether the mobile body is moving on a roadway or in the predetermined area on the basis of a result obtained by respectively performing space classification on the center virtual lane, the right side virtual lane, and the left side virtual lane; and a control unit that limits the speed to a first speed when the mobile body is moving on the roadway and limits the speed to a second speed that is lower than the first speed when the mobile body is moving in the predetermined area.

Description

Technical Field

[0001] The present invention relates to a control device for a moving body, a control method for a moving body, and a storage medium.

Background Art

[0002] Conventionally, practical applications have been advanced for moving bodies capable of moving on both sidewalks and roadways. In such moving bodies, it is necessary to set different upper limit speeds for sidewalks and roadways. In this regard, a document has been disclosed that examines the recognition of whether a moving body is moving on a sidewalk or a roadway (Patent Document 1).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In the conventional technology, there are cases where it is not possible to appropriately recognize whether a moving body is moving on a roadway or moving in a predetermined area different from the roadway.

[0005] The present invention has been made in consideration of such circumstances, and one of its purposes is to provide a control device for a moving body, a control method for a moving body, and a storage medium that can appropriately recognize whether a moving body is moving on a roadway or moving in a predetermined area different from the roadway.

Means for Solving the Problems

[0006] The control device for a moving body, the control method for a moving body, and the storage medium according to this invention employ the following configurations. (1) The control device for a moving body according to one aspect of the present invention is a control device for a moving body capable of moving in both a roadway and a predetermined area different from the roadway. In a captured image of an external camera that captures the traveling direction of the moving body, a central virtual lane including an assumed route of the moving body, a right virtual lane existing on the right side of the central virtual lane as viewed from the moving body, and a left virtual lane existing on the left side of the central virtual lane as viewed from the moving body are respectively set. Based on the results of spatial classification for each of the central virtual lane, the right virtual lane, and the left virtual lane, a road type recognition unit that recognizes whether the moving body is moving on the roadway or moving in the predetermined area, and a control unit that limits the speed when the moving body is moving on the roadway to a first speed and limits the speed when the moving body is moving in the predetermined area to a second speed lower than the first speed.

[0007] (2) In the aspect of (1) above, the road type recognition unit recognizes whether the central virtual lane shows the roadway or the predetermined area. When the reliability of the recognition result regarding the central virtual lane is equal to or higher than a reference, based on the recognition result regarding the central virtual lane, it recognizes whether the moving body is moving on the roadway or moving in the predetermined area. When the reliability of the recognition result regarding the central virtual lane is less than the reference, based on the results of the spatial classification for each of the left virtual lane and the right virtual lane, it recognizes whether the moving body is moving on the roadway or moving in the predetermined area.

[0008] (3) In the aspect of (1) or (2) above, as the spatial classification, the road type recognition unit recognizes, for each of the left virtual lane and the right virtual lane, whether it shows the roadway, the predetermined area, or outside the runway.

[0009] (4): In the aspect of (3) above, based on the combination of the results of performing the spatial classification for each of the left virtual lane and the right virtual lane, the road type recognition unit recognizes whether the moving object is moving on a lane or moving in the predetermined area.

[0010] (5): In the aspect of (3) above, when the road type recognition unit recognizes that one of the left virtual lane and the right virtual lane depicts a lane and the other depicts the predetermined area, based on the similarity between the image related to the left virtual lane and the image related to the central virtual lane, and the similarity between the image related to the right virtual lane and the image related to the central virtual lane, the road type recognition unit recognizes whether the moving object is moving on a lane or moving in the predetermined area.

[0011] (6): In the aspect of (3) above, when the road type recognition unit recognizes that one of the left virtual lane and the right virtual lane depicts outside the roadway and the other depicts a lane, based on the similarity between the image related to the left virtual lane and the image related to the central virtual lane, the similarity between the image related to the right virtual lane and the image related to the central virtual lane, and whether there is a partition at the boundary portion between the image related to the central virtual lane and the image recognized as depicting the lane, the road type recognition unit recognizes whether the moving object is moving on a lane or moving in the predetermined area.

[0012] (7): In the aspect of (1) above, the road type recognition unit inputs the information obtained by adding virtual lane designation information indicating which area corresponds to which virtual lane to the captured image into a learned model, thereby obtaining the result of recognizing whether the moving object is moving on a lane or moving in the predetermined area.

[0013] (8): In the aspect of (1) above, the road type recognition unit inputs the captured image into a learned model, thereby obtaining the result of recognizing whether the moving object is moving on a lane or moving in the predetermined area.

[0014] (9): In the aspect of (7) or (8) above, the learned model is such that the parameters of the layer that performs spatial classification for each virtual lane and the parameters of the layer that integrates the results of the spatial classification for each virtual lane are simultaneously learned by backpropagation processing using common learning data and teacher data.

[0015] (10): In the aspect of (1) above, when a series of edges indicating a lane boundary can be extracted from the captured image, the road type recognition unit sets the series of edges as the boundary between two adjacent virtual lanes among the plurality of virtual lanes.

[0016] (11): In the aspect of (10) above, for a region where the series of edges cannot be extracted from the captured image, the road type recognition unit assumes that at least a part of the plurality of virtual lanes extends with a specified width on the road surface, and sets the boundary between two adjacent virtual lanes among the plurality of virtual lanes.

[0017] (12): In the aspect of (1) above, the road type recognition unit performs repetitive processing at a predetermined cycle, and based on the past recognition results for each of the plurality of virtual lanes, recognizes whether the moving body is moving on a lane or moving in the predetermined region.

[0018] (13) A method for controlling a moving body according to another aspect of the present invention is a method for controlling a moving body performed by a control device of a moving body capable of moving in both a roadway and a predetermined area different from the roadway. In an imaging image of an external camera that images the traveling direction of the moving body, a central virtual lane including an assumed route of the moving body, a right virtual lane existing on the right side of the central virtual lane as viewed from the moving body, and a left virtual lane existing on the left side of the central virtual lane as viewed from the moving body are respectively set. Based on the results of spatial classification for each of the central virtual lane, the right virtual lane, and the left virtual lane, it is recognized whether the moving body is moving on the roadway or moving in the predetermined area. When the moving body is moving on the roadway, the speed is limited to a first speed, and when the moving body is moving in the predetermined area, the speed is limited to a second speed lower than the first speed.

[0019] (14) A storage medium according to another aspect of the present invention causes a processor of a control device of a moving body capable of moving in both a roadway and a predetermined area different from the roadway to set, in an imaging image of an external camera that images the traveling direction of the moving body, a central virtual lane including an assumed route of the moving body, a right virtual lane existing on the right side of the central virtual lane as viewed from the moving body, and a left virtual lane existing on the left side of the central virtual lane as viewed from the moving body. Based on the results of spatial classification for each of the central virtual lane, the right virtual lane, and the left virtual lane, it is recognized whether the moving body is moving on the roadway or moving in the predetermined area. When the moving body is moving on the roadway, the speed is limited to a first speed, and when the moving body is moving in the predetermined area, the speed is limited to a second speed lower than the first speed. A storage medium stores a program for executing the above.

Advantages of the Invention

[0020] (1) According to the aspects of (1) to (14), it is possible to appropriately recognize whether the moving body is moving on the roadway or moving in a predetermined area different from the roadway.

Brief Description of the Drawings

[0021]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Modes for Carrying Out the Invention

[0022] Hereinafter, with reference to the drawings, embodiments of a control device for a moving body, a control method for a moving body, and a program will be described. The moving body moves in both a roadway and a predetermined area different from the roadway. The moving body may be referred to as micromobility. An electric kick scooter is a type of micromobility. Also, the moving body may be a vehicle that can carry passengers, or an autonomous moving body capable of autonomous driving without a driver. The latter autonomous moving body is used, for example, for transporting luggage and the like. The predetermined area is, for example, a sidewalk. Also, the predetermined area may be part or all of a roadside strip, a bicycle lane, an open space, etc., or may include all of a sidewalk, a roadside strip, a bicycle lane, an open space, etc. In the following description, the predetermined area is assumed to include a sidewalk and an open space.

[0023] FIG. 1 is a diagram showing an example of the configuration of a moving body 1 and a control device 100 according to an embodiment. The moving body 1 is equipped with, for example, an external detection device 10, a moving body sensor 12, an operator 14, an internal camera 16, a positioning device 18, a mode changeover switch 22, a moving mechanism 30, a driving device 40, an external notification device 50, a storage device 70, and a control device 100. Note that some of these configurations that are not essential for realizing the functions of the present invention may be omitted. The moving body is not limited to a vehicle and may include small mobility such as running parallel to a walking user to carry luggage or lead a person, or may include other moving bodies capable of autonomous movement (such as a walking robot).

[0024] The external detection device 10 is various devices that take the traveling direction of the moving body 1 as the detection range. The external detection device 10 includes an external camera, a radar device, LIDAR (Light Detection and Ranging), a sensor fusion device, and the like. The external detection device 10 outputs information (image, position of an object, etc.) indicating the detection result to the control device 100.

[0025] The mobile body sensor 12 includes, for example, a speed sensor, an acceleration sensor, a yaw rate (angular velocity) sensor, an azimuth sensor, and an operation amount detection sensor attached to the operator 14. The operator 14 includes, for example, an operator for instructing acceleration and deceleration (such as an accelerator pedal or a brake pedal) and an operator for instructing steering (such as a steering wheel). In this case, the mobile body sensor 12 may include an accelerator opening sensor, a brake depression amount sensor, a steering torque sensor, etc. The mobile body 1 may be provided with an operator in a form other than the above as the operator 14 (for example, a non-circular rotary operator, a joystick, a button, etc.).

[0026] The internal camera 16 images at least the head of the passenger of the mobile body 1 from the front. The internal camera 16 is a digital camera that uses an imaging device such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor). The internal camera 16 outputs the captured image to the control device 100.

[0027] The positioning device 18 is a device that positions the position of the mobile body 1. The positioning device 18 is, for example, a GNSS (Global Navigation Satellite System) receiver, and based on the signal received from the GNSS satellite, identifies the position of the mobile body 1 and outputs it as position information. Note that the position information of the mobile body 1 may be estimated from the position of the Wi-Fi base station to which the communication device described later is connected.

[0028] The mode switch 22 is a switch operated by the occupant. The mode switch 22 may be a mechanical switch or a GUI (Graphical User Interface) switch set on a touch panel. The mode switch 22 accepts an operation to switch the driving mode to, for example, Mode A: an assist mode in which either a steering operation or acceleration / deceleration control is performed by the occupant and the other is automatically performed, and there may be Mode A-1 in which the steering operation is performed by the occupant and the acceleration / deceleration control is automatically performed, and Mode A-2 in which the acceleration / deceleration operation is performed by the occupant and the steering control is automatically performed; Mode B: a manual driving mode in which the steering operation and the acceleration / deceleration operation are performed by the occupant; or Mode C: an automatic driving mode in which the operation control and the acceleration / deceleration control are automatically performed.

[0029] The moving mechanism 30 is a mechanism for moving the moving body 1 on the road. The moving mechanism 30 is, for example, a wheel group including a steering wheel and drive wheels. Also, the moving mechanism 30 may be legs for multi-legged walking.

[0030] The driving device 40 outputs force to the moving mechanism 30 to move the moving body 1. For example, the driving device 40 includes a motor that drives the drive wheels, a battery that stores electric power supplied to the motor, a steering device that adjusts the steering angle of the steering wheel, and the like. The driving device 40 may be provided with an internal combustion engine, a fuel cell, etc. as a driving force output means or a power generation means. Also, the driving device 40 may further be provided with a braking device by frictional force or air resistance.

[0031] The external notification device 50 is provided, for example, on the outer plate portion of the moving body 1, and is a lamp, a display device, a speaker, etc. for notifying information toward the outside of the moving body 1. The external notification device 50 performs different operations depending on whether the moving body 1 is moving in a predetermined area or moving on a roadway. For example, the external notification device 50 is controlled to emit light from the lamp when the moving body 1 is moving in a predetermined area and not to emit light from the lamp when the moving body 1 is moving on a roadway. It is preferable that the emission color of this lamp is a color defined by regulations. The external notification device 50 may be controlled to emit light from the lamp in green when the moving body 1 is moving in a predetermined area and to emit light from the lamp in blue when the moving body 1 is moving on a roadway. When the external notification device 50 is a display device, the external notification device 50 displays, in text or graphics, that "the vehicle is traveling on the sidewalk" when the moving body 1 is traveling in a predetermined area.

[0032] FIG. 2 is a perspective view of the moving body 1 seen from above. In the figure, FW is the steering wheel, RW is the drive wheel, SD is the steering device, MT is the motor, and BT is the battery. The steering device SD, the motor MT, and the battery BT are included in the drive device 40. Also, AP is the accelerator pedal, BP is the brake pedal, WH is the steering wheel, SP is the speaker, and MC is the microphone. The illustrated moving body 1 is a single-seater moving body, and the passenger P is seated in the driver's seat DS and wearing the seat belt SB. The arrow D1 is the traveling direction (velocity vector) of the moving body 1. The external world detection device 10 is provided near the front end portion of the moving body 1, the internal camera 16 is at a position where it can image the head of the passenger P from in front of the passenger P, and the mode changeover switch 22 is provided on the boss portion of the steering wheel WH, respectively. Also, near the front end portion of the moving body 1, an external notification device 50 as a display device is provided.

[0033] Returning to FIG. 1, the storage device 70 is a non-transitory storage device such as, for example, a HDD (Hard Disk Drive), a flash memory, or a RAM (Random Access Memory). The storage device 70 stores map information 72, a program 74 executed by the control device 100, and the like. In the figure, the storage device 70 is shown outside the frame of the control device 100, but the storage device 70 may be included in the control device 100. Also, the storage device 70 may be provided on a server (not shown).

[0034] <First Embodiment> [Control Device] The control device 100 includes, for example, a road type recognition unit 120, an object recognition unit 130, and a control unit 140. These components are realized, for example, when a hardware processor such as a CPU (Central Processing Unit) executes a program (software) 74. Some or all of these components may be realized by hardware (including a circuit part; circuitry) such as an LSI (Large Scale Integration), an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or a GPU (Graphics Processing Unit), or may be realized by cooperation between software and hardware. The program may be stored in the storage device 70 in advance, or may be stored in a removable storage medium (non-transitory storage medium) such as a DVD or a CD-ROM, and may be installed in the storage device 70 when the storage medium is mounted on a drive device.

[0035] The road type recognition unit 120 recognizes whether the moving body 1 is moving on a road lane or moving in a predetermined area. The road type recognition unit 120 recognizes whether the moving body 1 is moving on a road lane or moving in a predetermined area, for example, by analyzing an image captured by the external camera of the external detection device 10. Note that the outputs of a radar device, LIDAR, a sensor fusion device, etc. may be used supplementarily. Details of the processing of the road type recognition unit 120 will be described later.

[0036] The object recognition unit 130 recognizes objects existing around the moving body 1 based on the output of the external detection device 10. An object includes some or all of moving bodies such as vehicles, bicycles, and pedestrians, road markings, steps, guardrails, road shoulders, roadway boundaries such as a median strip, structures installed on the road such as road signs and billboards, and obstacles such as (fallen) dropped objects existing on the roadway. When an image captured by the external camera of the external detection device 10 is input, for example, the object recognition unit 130 inputs the image captured by the external camera to a learned model that has been learned to output information such as the presence, position, and type of an object, thereby obtaining information such as the presence, position, and type of other moving bodies. The type of other moving bodies can also be estimated based on the size in the image, the intensity of the reflected wave received by the radar device of the external detection device 10, and the like. Further, the object recognition unit 130 obtains the speed of other moving bodies detected by the radar device using, for example, Doppler shift.

[0037] The control unit 140 controls the drive device 40 according to, for example, the set driving mode. Note that the moving body 1 may execute only some of the following driving modes, but in any case, the control unit 140 sets different speed limit values when the moving body 1 is moving on a road lane and when it is moving in a predetermined area. In that case, the mode change switch 22 may be omitted.

[0038] In mode A-1, the control unit 140 refers to the information on the travel path and the object based on the output of the object recognition unit 130. When the moving body 1 moves on the road, the control unit 140 maintains a certain distance or more from the object existing in front of the moving body 1. When the distance from the object existing in front of the moving body 1 is sufficiently long, the control unit 140 controls the motor MT of the drive device 40 so that the moving body 1 moves at the first speed V1 (for example, a speed of 10 [km / h] or more and less than several tens [km / h]). When the moving body 1 moves in a predetermined area, the control unit 140 maintains a certain distance or more from the object existing in front of the moving body 1. When the distance from the object existing in front of the moving body 1 is sufficiently long, the control unit 140 controls the motor MT of the drive device 40 so that the moving body 1 moves at the second speed V2 (for example, a speed of less than 10 [km / h]). Such a function is similar to the ACC (Adaptive Cruise Control) function of a vehicle with the first speed V1 or the second speed V2 as the set speed, and the technology used in ACC can be utilized. In mode A-1, the control unit 140 controls the steering device SD to change the steering angle of the steered wheels based on the operation amount of the operator 14 such as the steering wheel. Such a function is similar to the function of a power steering device, and the technology used in the power steering device can be utilized. Note that the moving body 1 may have a steering device in which the operator 14 and the steering mechanism are mechanically connected without performing electronic control for steering.

[0039] In mode A-2, the control unit 140 refers to the information on the travel path and the object based on the output of the object recognition unit 130, generates a target trajectory that can avoid the object within the travel path, and controls the steering device SD of the drive device 40 so that the moving body 1 moves along the target trajectory. Regarding acceleration and deceleration, the control unit 140 controls the motor MT of the drive device 40 based on the speed of the moving body 1 and the operation amount of the accelerator pedal or the brake pedal. When the moving body 1 moves on the road, the control unit 140 controls the motor MT of the drive device 40 with the first speed V1 as the upper limit speed (in the case of mode A-2, when the upper limit speed is reached, it means that the moving body 1 is not accelerated even if there is a further acceleration instruction). When the moving body 1 moves in a predetermined area, the control unit 140 controls the drive device 40 with the second speed V2 as the upper limit speed.

[0040] In mode B, the control unit 140 controls the motor MT of the drive device 40 based on the speed of the moving body 1 and the operation amount of the accelerator pedal or the brake pedal. When the moving body 1 is moving on the road, the control unit 140 controls the motor MT of the drive device 40 with the first speed V1 as the upper limit speed (in the case of mode B, when the upper limit speed is reached, it means that the moving body 1 will not be accelerated even if there is a further acceleration instruction), and when the moving body 1 is moving in a predetermined area, the control unit 140 controls the motor MT of the drive device 40 with the second speed V2 as the upper limit speed. Regarding steering, it is the same as in mode A-1.

[0041] In mode C, the control unit 140 refers to the information of the traveling path and the object based on the output of the object recognition unit 130, generates a target trajectory that can avoid the object within the traveling path, and controls the drive device 40 so that the moving body 1 moves along the target trajectory. Also in mode C, when the moving body 1 is moving on the road, the control unit 140 controls the drive device 40 with the first speed V1 as the upper limit speed, and when the moving body 1 is moving in a predetermined area, the control unit 140 controls the drive device 40 with the second speed V2 as the upper limit speed.

[0042] [Road type recognition] Hereinafter, the processing of the road type recognition unit 120 will be described. The road type recognition unit 120 defines a plurality of virtual lanes including at least a central virtual lane including the assumed traveling path of the moving body 1, a right virtual lane existing on the right side of the central virtual lane as seen from the moving body 1, and a left virtual lane existing on the left side of the central virtual lane as seen from the moving body 1 in the space on the traveling direction side of the moving body 1, and for each of the plurality of virtual lanes, based on the result of performing space classification based on the output of the external world detection device 10, it recognizes whether the moving body 1 is moving on the road or moving in a predetermined area.

[0043] The external detection device 10 used by the road type recognition unit 120 is an external camera that images the outside of the moving body. FIG. 3 is a diagram showing an example of the captured image IM of the external camera. In the figure, 200 is a roadway and 201 is a sidewalk. The region Lm in FIG. 3 represents a central virtual lane including the assumed travel path K of the moving body 1 (for example, on the extension line of the central axis of the moving body 1 at that time, but not limited to this, and may be a future travel path according to the steering angle when the steering angle is generated). The region Lr in FIG. 3 represents a right virtual lane Lr existing on the right side with respect to the central virtual lane Lm, and the region Ll represents a left virtual lane existing on the left side with respect to the central virtual lane Lm as viewed from the moving body 1.

[0044] The road type recognition unit 120 sets the central virtual lane Lm, the right virtual lane Lr, and the left virtual lane Ll on the captured image IM and performs processing on each region in the captured image IM. When determining this region, if a series of edges indicating the road boundary (hereinafter referred to as road boundary edges) can be extracted in the captured image IM, the road type recognition unit 120 sets the road boundary edges as the boundary between two adjacent virtual lanes among the plurality of virtual lanes. An edge is a pixel (feature pixel) whose pixel value (such as luminance, RGB value, etc.) difference from an adjacent pixel is larger than a reference. There are various methods for selecting adjacent pixels, but when extracting a line extending along the traveling direction as viewed from the moving body 1, it is preferable that the horizontal pixels of the pixel of interest are selected as adjacent pixels.

[0045] FIG. 4 is a diagram showing the range of positions of a series of edges that can be recognized as runway boundary edges in the captured image. The runway boundary edges should be arranged from the lower part of the captured image IM toward the top dead center DP. Also, since it is rare for the moving body 1 to move directly above the runway boundary, the runway boundary edges should be arranged from a position offset to either the left or right rather than the center of the lower part of the captured image IM toward the top dead center DP. The road type recognition unit 120 recognizes the edges arranged toward the top dead center DP in the illustrated region A1 as runway boundary edges indicating the boundary between the central virtual lane Lm and the left virtual lane Ll, and recognizes the edges arranged toward the top dead center DP in the region A2 as runway boundary edges indicating the boundary between the central virtual lane Lm and the right virtual lane Lr. In the figure, the line recognized as the runway boundary edge is shown by a dashed line. In the example of FIG. 3, since the step 202 between the roadway 200 and the sidewalk 201 is extracted as the runway boundary edge, the road type recognition unit 120 uses this runway boundary edge as the boundary line between the central virtual lane Lm and the left virtual lane Ll.

[0046] When one of the runway boundary edges that should exist on the left and right cannot be extracted, the road type recognition unit 120 uses a line offset by a specified width W (width in the assumed plane as viewed from above) to either the left or right from the extracted runway boundary edge as the boundary between two adjacent virtual lanes. FIG. 5 is a diagram showing how the boundary of the virtual lane is set by offsetting the extracted runway boundary edge by the specified width. In the figure, BD is the boundary between the central virtual lane Lm and the right virtual lane Lr to be set. Since the specified width W is the width in the assumed plane S, it becomes narrower toward the upper side in the captured image IM. The specified width W is set to, for example, the width of a typical roadway, but it may be variable according to the recognition result of whether the moving body 1 is moving on the roadway or in a predetermined area. For example, when it is recognized with high confidence that the moving body 1 is moving in a predetermined area (especially a sidewalk), the specified width W regarding at least the central virtual lane Lm may be set to the width of a typical sidewalk.

[0047] When no edge line can be extracted at all, the road type recognition unit 120 sets lines that are offset left and right from the assumed travel route K by a predetermined width (half of the specified width W) on the assumed plane, for example, and sets the lines obtained by converting them to the plane of the captured image IM as the boundaries between two adjacent virtual lanes.

[0048] In this way, the road type recognition unit 120 assumes that at least a part of a plurality of virtual lanes extends with a specified width on the road surface, and sets the boundaries between two adjacent virtual lanes among the plurality of virtual lanes.

[0049] Also, the road type recognition unit 120 repeats the process at a predetermined cycle, and may recognize whether the moving body 1 is moving on a lane or moving in a predetermined area while inheriting the past recognition results for each of the plurality of virtual lanes.

[0050] The road type recognition unit 120 of the first embodiment recognizes whether the moving body 1 is moving on a lane or moving in a predetermined area based on the result of spatial classification performed for each of the plurality of virtual lanes based on the output of the external detection device.

[0051] For example, the road type recognition unit 120 first recognizes whether the central virtual lane Lm captures a lane or a predetermined area. When the reliability of the recognition result regarding the central virtual lane Lm is equal to or higher than the reference, the road type recognition unit 120 recognizes whether the moving body 1 is moving on a lane or moving in a predetermined area based on the recognition result regarding the central virtual lane Lm.

[0052] For example, each time the road type recognition unit 120 recognizes a plurality of first events indicating that the moving body 1 is moving on a lane in the area of the central virtual lane Lm of the captured image IM, it adds points to the lane score Sr. When the lane score Sr is equal to or greater than the first threshold Th1, it recognizes that the moving body 1 is moving on a lane. At this time, the road type recognition unit 120 weights the points according to the confidence level when recognizing each of the plurality of first events and adds them to the lane score Sr. However, when the road type recognition unit 120 recognizes any of a plurality of second events indicating that the moving body 1 is moving on a sidewalk in the image captured by the external camera, it recognizes that the moving body 1 is moving on a sidewalk regardless of the lane score Sr. The first events include the absence of static obstacles such as roadside signs, the movement of vehicles, the presence of road markings, the presence of crosswalks, and being on the lower side of a step. The second events include the presence of static obstacles such as roadside signs, the presence of braille blocks, and being on the upper side of a step. At this time, the reliability of the recognition result regarding the central virtual lane Lm is calculated based on the value of the lane score Sr. For example, when the lane score Sr is less than the first threshold Th1 and no second event is recognized, the road type recognition unit 120 determines that the reliability of the recognition result regarding the central virtual lane Lm is less than the standard.

[0053] Alternatively, the road type recognition unit 120 may input the area of the central virtual lane Lm of the captured image IM into the first pre-trained model to obtain a result of recognizing whether the central virtual lane Lm, that is, the area where the moving body 1 is moving, is a lane or a predetermined area. The first pre-trained model is a model learned by machine learning so that when an image having the size of the central virtual lane Lm is input, it outputs information (recognition result) indicating whether the portion shows a lane or a predetermined area. In this case, the first pre-trained model is set to output information indicating the reliability of the output information together with the recognition result. If the reliability output when the area of the central virtual lane Lm of the captured image IM is input into the first pre-trained model is less than the reference value, it may be determined that the reliability of the recognition result regarding the central virtual lane Lm is less than the standard.

[0054] When the reliability of the recognition result regarding the central virtual lane Lm is less than the standard, based on the results of spatial classification for each of the left virtual lane Ll and the right virtual lane Lr, the road type recognition unit 120 recognizes whether the moving body 1 is moving on a lane or moving in a predetermined area. As the spatial classification, the road type recognition unit 120 recognizes, for each of the left virtual lane Ll and the right virtual lane Lr, whether it is an image of a lane, an image of a predetermined area, an image of outside the road (e.g., a building wall), or unknown. FIG. 6 is a diagram (part 1) for explaining the processing of the road type recognition unit 120. First, as described above, the road type recognition unit 120 performs recognition on an image of the area of the central virtual lane Lm (hereinafter referred to as the central virtual lane image). When the reliability is low, by inputting an image of the area of the left virtual lane Ll (hereinafter referred to as the left virtual lane image) into the second pre-trained model, a recognition result indicating whether the left virtual lane Ll is an image of a lane, a predetermined area, or outside the road is obtained. Similarly, the road type recognition unit 120 inputs an image of the area of the right virtual lane Lr (hereinafter referred to as the right virtual lane image) into the third pre-trained model to obtain a recognition result indicating whether the right virtual lane Lr is an image of a lane, a predetermined area, or outside the road. Each of the second pre-trained model and the third pre-trained model is a model learned by machine learning so as to output the above-described identification result when an image is input. That is, it is a model learned using the image as learning data and labels such as a lane, a predetermined area, and outside the road as teacher data.

[0055] Based on the combination of the results of spatial classification for each of the left virtual lane Ll and the right virtual lane Lm, the road type recognition unit 120 recognizes whether the moving body 1 is moving on a lane or moving in a predetermined area. More specifically, based on the combination of the identification results of the left virtual lane Ll and the right virtual lane Lr and the analysis result regarding the central virtual lane Lm, the road type recognition unit 120 recognizes whether the moving body 1 is moving on a lane or moving in a predetermined area.

[0056] FIG. 7 is a diagram (part 2) for explaining the processing of the road type recognition unit 120. As shown in (1) in the figure, when it is recognized that both the left virtual lane Ll and the right virtual lane Lr are images of lanes, the road type recognition unit 120 recognizes that the central virtual lane Lm is also an image of a lane, that is, the moving body 1 is recognized as moving in a lane. Also, as shown in (2) in the figure, when it is recognized that both the left virtual lane Ll and the right virtual lane Lr are images of a predetermined area, the road type recognition unit 120 recognizes that the central virtual lane Lm is also an image of a predetermined area, that is, the moving body 1 is recognized as moving in a predetermined area.

[0057] As shown in (3) to (5) in the figure, when the road type recognition unit 120 recognizes that one of the left virtual lane Ll and the right virtual lane Lr is a lane and the other is an image of a predetermined area, based on the similarity αl between the left virtual lane image and the central virtual lane image and the similarity αr between the right virtual lane image and the central virtual lane image, it is recognized whether the moving body 1 is moving in a lane or in a predetermined area. In the figure, the left virtual lane Ll is a predetermined area and the right virtual lane Lr is a lane, but for the reverse pattern, the left and right should be reversed. The similarity is an index value obtained by calculating the cosine similarity or the like for the feature amounts of each image obtained by a method using a CNN (Convolution Neural Network) or the like, and can take a value from 0 to 1, for example. In the following example, both the second threshold Th2 and the third threshold Th3 are positive values, and Th2 > Th3.

[0058] As shown in (3) in the figure, when the similarity αl is greater than the second threshold Th2 and the difference αl - αr obtained by subtracting the similarity αr from the similarity αl is greater than the third threshold Th3, the road type recognition unit 120 recognizes that the central virtual lane Lm is an image of a predetermined area, that is, the moving body 1 is recognized as moving in a predetermined area.

[0059] As shown in (4) in the figure, when the similarity αr is greater than the second threshold Th2 and the difference αr - αl obtained by subtracting the similarity αl from the similarity αr is greater than the third threshold Th3, the central virtual lane Lm is a copy of the lane, that is, it is recognized that the moving body 1 is moving in the lane.

[0060] As shown in (5) in the figure, when both the similarity αr and the similarity αl are less than or equal to the fourth threshold Th4, it is unclear whether the central virtual lane Lm is a copy of the lane or a copy of a predetermined area, that is, the recognition result is output that it is unclear whether the moving body 1 is moving in the lane or the predetermined area.

[0061] As shown in (6) to (8) in the figure, when the road type recognition unit 120 recognizes that one of the left virtual lane Ll and the right virtual lane Lr is outside the runway and the other is a copy of the lane, based on the similarity αl between the left virtual lane image and the central virtual lane image, the similarity αr between the right virtual lane image and the central virtual lane image, and whether there is a partition such as a guardrail at the boundary between the central virtual lane image and the image recognized as a copy of the lane, it is recognized whether the moving body 1 is moving in the lane or in the predetermined area. In the figure, the left virtual lane Ll is outside the runway and the right virtual lane Lr is the lane, but for the reverse pattern, the left and right should be reversed.

[0062] First, as shown in (6) in the figure, when there is a partition such as a guardrail at the boundary between the central virtual lane image and the right virtual lane image, the central virtual lane Lm is a copy of the predetermined area, that is, it is recognized that the moving body 1 is moving in the predetermined area.

[0063] As shown in (7) in the figure, when there is no partition such as a guardrail at the boundary between the central virtual lane image and the right virtual lane image and the similarity αr between the central virtual lane image and the right virtual lane image is less than or equal to the fourth threshold Th4, the central virtual lane Lm is a copy of the predetermined area, that is, it is recognized that the moving body 1 is moving in the predetermined area.

[0064] As shown in (8) in the figure, when there is no partition such as a guardrail at the boundary between the central virtual lane image and the right virtual lane image in the road type recognition unit 120, and the similarity αr between the central virtual lane image and the right virtual lane image is greater than the fourth threshold Th4, the central virtual lane Lm is an image of the lane, that is, it is recognized that the moving body 1 is moving on the lane.

[0065] According to the first embodiment described above, it is possible to recognize whether the moving body 1 is moving on the lane or moving in a predetermined area different from the lane by reflecting the characteristics of each virtual lane. Even in a case where it is difficult to make a judgment only based on the central virtual lane image, the recognition accuracy can be improved by referring to the left virtual lane image and the right virtual lane image. As a result, it is possible to appropriately recognize whether the moving body 1 is moving on the lane or moving in a predetermined area different from the lane.

[0066] <Second Embodiment> Hereinafter, the second embodiment will be described. In the control device 100 of the second embodiment, the road type recognition unit 120 uses an integrated learned model including a virtual lane setting layer to recognize whether the moving body 1 is moving on the lane or moving in a predetermined area different from the lane.

[0067] FIG. 8 is a diagram for explaining the content of the process of the road type recognition unit 120 using the integrated learned model. The road type recognition unit 120 inputs the captured image IM into the integrated learned model to obtain a result of recognizing whether the moving body 1 is moving on the lane or moving in a predetermined area. The integrated learned model includes, for example, a virtual lane setting layer, a central virtual lane recognition layer, a left virtual lane recognition layer, a right virtual lane recognition layer, and a recognition result integration layer. The integrated learned model is learned by a method described later based on a machine learning model with a connection structure as shown in the figure.

[0068] When the captured image IM is input, the virtual lane setting layer outputs the ranges of the central virtual lane Lm, the left virtual lane Ll, and the right virtual lane Lr in the captured image IM. The central virtual lane recognition layer functions in the same manner as the first learned model in the first embodiment, the left virtual lane recognition layer functions in the same manner as the second learned model in the first embodiment, and the right virtual lane recognition layer functions in the same manner as the third learned model in the first embodiment. The recognition result integration layer has a function corresponding to the process shown in FIG. 7 in the first embodiment, but does not necessarily function according to the specified rules illustrated in FIG. 7, and is configured to perform arithmetic processing according to the result of machine learning.

[0069] FIG. 9 is a diagram for explaining the learning process of the integrated learned model. The integrated learned model is generated by a learning device (not shown). The virtual lane setting layer has its parameters learned by the first backpropagation process using the captured image IM as learning data and the virtual lane designation information indicating which region in the captured image IM corresponds to which virtual lane as teacher data. The central virtual lane recognition layer, the left virtual lane recognition layer, the right virtual lane recognition layer, and the recognition result integration layer are collectively learned, for example, by the second backpropagation process. For example, the central virtual lane image input to the central virtual lane recognition layer, the left virtual lane image input to the left virtual lane recognition layer, and the right virtual lane image input to the right virtual lane recognition layer are used as learning data, and the label (recognition result) indicating either the lane or a predetermined region is used as teacher data. The parameters of the central virtual lane recognition layer, the left virtual lane recognition layer, the right virtual lane recognition layer, and the recognition result integration layer are learned by the second backpropagation process. In this way, the integrated learned model has the parameters of the layer that performs spatial classification for each virtual lane and the parameters of the layer that integrates the results of the spatial classification for each virtual lane learned at once by the second backpropagation process using common learning data and teacher data.

[0070] By training the integrated trained model in this way, it is possible to recognize whether the moving object 1 is moving on the lane or in a predetermined area different from the lane, reflecting the characteristics of each virtual lane. Even in a case where it is difficult to make a determination based only on the central virtual lane image, the recognition accuracy can be improved by referring to the left virtual lane image and the right virtual lane image. In addition, by using machine learning, it is possible to perform recognition using features that were not assumed by the creator of the rule-based rules, so there is a possibility of further improving the recognition accuracy. As a result, according to the second embodiment, it is possible to appropriately recognize whether the moving object 1 is moving on the lane or in a predetermined area different from the lane.

[0071] <Modification Example of the Second Embodiment> In the second embodiment, instead of inputting the captured image IM into the integrated trained model, the central virtual lane image, the right virtual lane image, and the left virtual lane image cut out from the captured image IM based on the information of the virtual lanes set in a rule-based manner as in the first embodiment are input into the integrated trained model # (the one shown in FIG. 8 excluding the virtual lane setting layer) to obtain a recognition result. FIG. 10 is a diagram showing the processing of a modification example of the second embodiment. The integrated trained model # is one in which the parameters are learned by the second backpropagation process described above. That is, also in the integrated trained model #, the parameters of the layer that performs spatial classification for each virtual lane and the parameters of the layer that integrates the results of the spatial classification for each virtual lane are learned at once by the second backpropagation process using common learning data and teacher data.

[0072] The embodiments described above can be expressed as follows. A storage medium that stores computer-readable instructions, A processor connected to the storage medium, the processor executing the computer-readable instructions to: In an imaging image of an external camera that images the traveling direction of a moving body capable of moving on both a road lane and a predetermined area different from the road lane, a central virtual lane including an assumed route of the moving body, a right virtual lane existing on the right side of the central virtual lane as viewed from the moving body, and a left virtual lane existing on the left side of the central virtual lane as viewed from the moving body are respectively set. Based on the results of spatial classification for each of the central virtual lane, the right virtual lane, and the left virtual lane, it is recognized whether the moving body is moving on a road lane or moving in the predetermined area. The speed when the moving body moves on a road lane is limited to a first speed, and the speed when the moving body moves in the predetermined area is limited to a second speed lower than the first speed. A control device for a moving body.

[0073] As described above, the embodiments for implementing the present invention have been described using the embodiments. However, the present invention is not limited to such embodiments, and various modifications and substitutions can be made without departing from the gist of the present invention.

Explanation of Reference Numerals

[0074] 10 External detection device 12 Moving body sensor 14 Operator 16 Internal camera 18 Positioning device 22 Mode switch 30 Moving mechanism 40 Driving device 50 External notification device 70 Storage device 100 Control device 120 Road type recognition unit 130 Object recognition unit 140 Control unit

Claims

1. A control device for a moving body capable of moving in both a lane and a predetermined area different from the lane, comprising: a road type recognition unit that recognizes whether the moving body is moving in the lane or in the predetermined area based on the results of spatial classification for each of a central virtual lane including an assumed travel path of the moving body, a right virtual lane existing on the right side of the central virtual lane as viewed from the moving body, and a left virtual lane existing on the left side of the central virtual lane as viewed from the moving body, in a captured image of an external camera that captures the traveling direction of the moving body; a control unit that limits the speed when the moving body moves in the lane to a first speed and limits the speed when the moving body moves in the predetermined area to a second speed lower than the first speed; A control device for a moving body comprising the above.

2. The road type recognition unit: recognizes whether the central virtual lane depicts the lane or the predetermined area; when the reliability of the recognition result regarding the central virtual lane is equal to or higher than a reference, recognizes whether the moving body is moving in the lane or in the predetermined area based on the recognition result regarding the central virtual lane; when the reliability of the recognition result regarding the central virtual lane is less than the reference, recognizes whether the moving body is moving in the lane or in the predetermined area based on the results of the spatial classification for each of the left virtual lane and the right virtual lane. The control device for a moving body according to Claim 1.

3. The road type recognition unit recognizes, as the spatial classification, for each of the left virtual lane and the right virtual lane, whether it depicts the lane, the predetermined area, or outside the runway. The control device for a moving body according to Claim 1 or 2.

4. The road type recognition unit recognizes whether the moving body is moving in the lane or in the predetermined area based on a combination of the results of the spatial classification for each of the left virtual lane and the right virtual lane. The control device for a moving body according to Claim 3.

5. When the road type recognition unit recognizes that one of the left virtual lane and the right virtual lane is an image of a traffic lane and the other is an image of the predetermined area, based on the similarity between the image related to the left virtual lane and the image related to the center virtual lane, and the similarity between the image related to the right virtual lane and the image related to the center virtual lane, it recognizes whether the moving body is moving in the traffic lane or in the predetermined area. The control device for a moving body according to claim 3.

6. When the road type recognition unit recognizes that one of the left virtual lane and the right virtual lane is an image of outside the road and the other is an image of a traffic lane, based on the similarity between the image related to the left virtual lane and the image related to the center virtual lane, the similarity between the image related to the right virtual lane and the image related to the center virtual lane, and whether there is a partition at the boundary between the image related to the center virtual lane and the image recognized as an image of the traffic lane, it recognizes whether the moving body is moving in the traffic lane or in the predetermined area. The control device for a moving body according to claim 3.

7. The road type recognition unit inputs the information obtained by adding virtual lane designation information indicating which area corresponds to which virtual lane to the captured image into a learned model, so as to obtain a result of recognizing whether the moving body is moving in a traffic lane or in the predetermined area. The control device for a moving body according to claim 1.

8. The road type recognition unit inputs the captured image into a learned model, so as to obtain a result of recognizing whether the moving body is moving in a traffic lane or in the predetermined area. The control device for a moving body according to claim 1.

9. The parameters of the layer for performing spatial classification for each virtual lane and the parameters of the layer for integrating the results of the spatial classification for each virtual lane in the learned model are simultaneously learned by backpropagation processing using common learning data and teacher data. The control device for a moving body according to claim 7 or 8.

10. When a series of edges indicating the road boundary can be extracted from the captured image, the road type recognition unit sets the series of edges as the boundary between two adjacent virtual lanes among the plurality of virtual lanes. The control device for a moving body according to claim 1.

11. For areas in the captured image where the series of edges cannot be extracted, the road type recognition unit sets the boundary between two adjacent virtual lanes among the plurality of virtual lanes on the assumption that at least a part of the plurality of virtual lanes extends with a specified width on the road surface. The control device for a moving body according to claim 10.

12. The road type recognition unit performs repetitive processing at a predetermined cycle, and based on the past recognition results for each of the plurality of virtual lanes, recognizes whether the moving body is moving on a lane or moving in the predetermined area. The control device for a moving body according to claim 1.

13. A control method for a moving body performed by a control device for a moving body capable of moving in both a lane and a predetermined area different from the lane, In the captured image of an external camera that captures the traveling direction of the moving body, setting a central virtual lane including the assumed travel route of the moving body, a right virtual lane existing on the right side of the central virtual lane as viewed from the moving body, and a left virtual lane existing on the left side of the central virtual lane as viewed from the moving body, respectively; Based on the results of spatial classification for each of the central virtual lane, the right virtual lane, and the left virtual lane, recognizing whether the moving body is moving on a lane or moving in the predetermined area; Limiting the speed when the moving body moves on a lane to a first speed, and limiting the speed when the moving body moves in the predetermined area to a second speed lower than the first speed; A control method for a moving body comprising the above.

14. In a processor of a control device for a moving body capable of moving in both a lane and a predetermined area different from the lane, In the captured image of an external camera that captures the traveling direction of the moving body, setting a central virtual lane including the assumed travel route of the moving body, a right virtual lane existing on the right side of the central virtual lane as viewed from the moving body, and a left virtual lane existing on the left side of the central virtual lane as viewed from the moving body, respectively; Based on the results of spatial classification for each of the central virtual lane, the right virtual lane, and the left virtual lane, recognizing whether the moving body is moving on a lane or moving in the predetermined area; Limiting the speed when the moving body moves on a lane to a first speed, and limiting the speed when the moving body moves in the predetermined area to a second speed lower than the first speed; A storage medium storing a program for causing [operation].

Images