MOBILE BODY CONTROL DEVICE, MOBILE BODY CONTROL METHOD, AND STORAGE MEDIUM

Abstract

This mobile body control apparatus comprises: a road type recognition unit that recognizes whether a region of interest is a roadway or a walkway on the basis of bicycle markers, which are for guiding the travel of a bicycle, that have been detected by a detection device for detecting the external situation of a mobile body capable of moving on both roadways and walkways; and a control unit that controls the mobile body on the basis of the recognition result by the road type recognition unit.

Description

[Technical Field]

[0001] The present invention relates to a control device for a mobile object, a control method for a mobile object, and a storage medium. [Background technology]

[0002] Conventionally, a driving support device for an electric vehicle has been disclosed that detects a direction in which the electric vehicle can move from image data captured by an imaging unit and controls the driving of the electric vehicle based on the detected direction (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2019-197328 Summary of the Invention [Problem to be solved by the invention]

[0004] However, the above-described driving assistance device for an electric vehicle may not be able to recognize the type of road in the target area, which may result in, for example, being unable to drive in accordance with the road type.

[0005] The present invention has been made in consideration of these circumstances, and one of its objectives is to provide a mobile body control device, a mobile body control method, and a storage medium that can accurately recognize the type of road in a target area and control the mobile body according to the road type. [Means for solving the problem]

[0006] The control device for a moving body, the control method for a moving body, and the storage medium according to the present invention employ the following configurations. (1): A control device for a mobile body according to one embodiment of the present invention is a control device for a mobile body that can move on both roadways and sidewalks, and is equipped with a road type recognition unit that recognizes whether an area of interest is a roadway or a sidewalk based on bicycle signs that guide bicycle passage detected by a detection device that detects the external conditions of the mobile body, and a control unit that controls the mobile body based on the recognition results of the road type recognition unit.

[0007] (2) In the above aspect (1), the road type recognition unit recognizes whether the area of interest is a roadway or a sidewalk based on the bicycle sign and the orientation of the bicycle sign.

[0008] (3): In the above-mentioned aspects (1) or (2), the road type recognition unit recognizes whether the area of interest is a roadway or a sidewalk based on the bicycle sign and signs placed near the bicycle sign.

[0009] (4): In any of the above aspects (1) to (3), the road type recognition unit recognizes whether the area of interest is a roadway or a sidewalk based on the width of the bicycle sign, the first road on which the bicycle sign is displayed, and the width of the second road adjacent to the first road.

[0010] (5): In any of the above aspects (1) to (4), the road type recognition unit recognizes the narrower of the first road and the second road as a sidewalk, and recognizes the wider road as a roadway.

[0011] (6): In any of the above aspects (1) to (5), when the bicycle marking is provided on a first road and there is a boundary object adjacent to the bicycle marking that separates the first road from a second road adjacent to the first road, the road type recognition unit recognizes whether the second road is a roadway or a sidewalk based on the presence or absence of vehicles on the first road and the second road, and / or the width of the first road and the width of the second road.

[0012] (7): In the above aspect (6), the boundary object is a curb, a fence, a guardrail, a plant, or a marking on a road.

[0013] (8): In the above aspect (7), the road type recognition unit recognizes the road on which no vehicle is present, of the first road and the second road, as a sidewalk.

[0014] (9): In the above aspect (8), the road type recognition unit recognizes the narrower road of the first road and the second road as a sidewalk.

[0015] (10): In any of the aspects of (7) above, the road type recognition unit identifies the narrower road between the first road and the second road, and if there is no vehicle on the identified road, recognizes the identified road as a sidewalk.

[0016] (11): In any of the above aspects (1) to (10), the road type recognition unit recognizes whether the area of interest is a roadway or a sidewalk based on the bicycle sign, the orientation of the bicycle sign, the width of the road on either side of the bicycle sign, and signs placed near the bicycle sign.

[0017] (12): In any of the above aspects (1) to (11), if the orientation of the bicycle marking can be recognized, the road type recognition unit recognizes whether the area of interest is a roadway or a sidewalk based on the bicycle marking and the orientation of the bicycle marking. If the orientation of the bicycle marking cannot be recognized, the road type recognition unit recognizes whether the area of interest is a roadway or a sidewalk based on the bicycle marking and one or both of the width of the road on either side of the bicycle marking and signs placed near the bicycle marking.

[0018] (13): In any of the above aspects (1) to (12), when the mobile body is located on a road that the road type recognition unit recognizes as a roadway, the control unit controls the mobile body in a roadway mode for traveling on the roadway, and when the mobile body is located on a road that the road type recognition unit recognizes as a sidewalk, the control unit controls the mobile body in a sidewalk mode for traveling on the sidewalk.

[0019] (14): In any of the above aspects (1) to (12), when the moving body is located on a road that the road type recognition unit recognizes as a roadway, the control unit limits the upper limit speed at which the moving body can travel to a first speed, and when the moving body is located on a road that the road type recognition unit recognizes as a sidewalk, the control unit limits the upper limit speed at which the moving body can travel to a second speed that is smaller than the first speed.

[0020] (15): In another aspect of the present invention, a method for controlling a moving body includes a control device for a moving body that can move on both roadways and sidewalks, which recognizes whether an area of interest is a roadway or a sidewalk based on bicycle signs that guide the passage of bicycles detected by a detection device that detects the external conditions of the moving body, and controls the moving body based on the results of the recognition.

[0021] (16): A storage medium storing a program according to another aspect of the present invention causes a control device of a mobile body that can move on both roadways and sidewalks to recognize whether an area of interest is a roadway or a sidewalk based on bicycle signs that guide the passage of bicycles detected by a detection device that detects the external conditions of the mobile body, and controls the mobile body based on the results of the recognition. [Effects of the Invention]

[0022] According to the above aspects (1) to (16), the control device can accurately recognize the type of road in the area of interest and can control the moving object according to the type of road.

[0023] According to the above aspect (12), the control device changes the process for recognizing the road type depending on the results of recognizing the orientation of the bicycle sign, thereby reducing the processing load while accurately recognizing the road type and enabling control of the moving object according to the road type. [Brief explanation of the drawings]

[0024] [Figure 1] 1 is a diagram illustrating an example of the configuration of a moving object 1 and a control device 100 according to an embodiment. [Figure 2] FIG. 2 is a perspective view of the moving body 1 seen from above. [Figure 3] FIG. 1 illustrates an example of a road including bicycle markings. [Figure 4] FIG. 10 shows another example of a road including bicycle markings. [Figure 5] FIG. 10 shows another example of a road including bicycle markings. [Figure 6] FIG. 10 is a diagram illustrating an example of a sign. [Figure 7] FIG. 10 is a diagram for explaining processing using various elements. [Figure 8] FIG. 10 is a diagram illustrating an example of a score derivation method. [Figure 9] 4 is a flowchart showing an example of the flow of processing executed by the control device 100. [Figure 10] 10 is a flowchart showing an example of the flow of processing executed by a control device 100 of a modified example. DETAILED DESCRIPTION OF THE INVENTION

[0025] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, embodiments of a mobile object control device, a mobile object control method, and a storage medium according to the present invention will be described with reference to the accompanying drawings.

[0026] Hereinafter, with reference to the drawings, embodiments of a mobile body control device, a mobile body control method, and a storage medium of the present invention will be described. A mobile body moves both on roadways and in a predetermined area different from the roadway. A mobile body is sometimes referred to as micromobility. An electric kick scooter is a type of micromobility. The predetermined area is, for example, a sidewalk. A mobile body may be a vehicle that can carry a passenger, or an autonomous mobile body capable of autonomous travel. The latter type of autonomous mobile body is used, for example, to transport luggage. A mobile body may include a small mobility vehicle that runs alongside a walking user to carry luggage or lead a user, or may include other autonomously moving mobile bodies (e.g., walking robots). The predetermined area may be part or all of a sidewalk, bicycle lane, public open space, etc., or may include all of the sidewalk, sidewalk, bicycle lane, public open space, etc. In the following description, the predetermined area is assumed to be a sidewalk. In the following description, the term "sidewalk" may be appropriately replaced with "predetermined area."

[0027] 1 is a diagram showing an example of the configuration of a moving object 1 and a control device 100 according to an embodiment. The moving object 1 is equipped with, for example, an external detection device 10, a moving object sensor 12, an operator 14, an internal camera 16, a positioning device 18, a mode selector switch 22, a movement mechanism 30, a drive unit 40, an external notification device 50, a storage device 70, and a control device 100. Some of these components that are not essential for realizing the functions of the present invention may be omitted.

[0028] The external detection device 10 is a device of various types whose detection range is in the traveling direction of the moving object 1. The external detection device 10 includes an external camera, a radar device, a LIDAR (Light Detection and Ranging), a sensor fusion device, etc. The external detection device 10 outputs information indicating the detection result (image, object position, etc.) to the control device 100.

[0029] The mobile body sensor 12 includes, for example, a speed sensor, an acceleration sensor, a yaw rate (angular velocity) sensor, a direction sensor, and an operation amount detection sensor attached to the operator 14. The operator 14 includes, for example, an operator for instructing acceleration / deceleration (e.g., an accelerator pedal or a brake pedal), and an operator for instructing steering (e.g., a steering wheel). In this case, the mobile body sensor 12 may include an accelerator opening sensor, a brake depression amount sensor, a steering torque sensor, etc. The mobile body 1 may also be provided with an operator of a type other than those described above as the operator 14 (e.g., a non-annular rotary operator, a joystick, a button, etc.).

[0030] Internal camera 16 captures an image of at least the head of an occupant of vehicle 1 from the front. Internal camera 16 is a digital camera that uses an imaging element such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor). Internal camera 16 outputs the captured image to control device 100.

[0031] Positioning device 18 is a device that measures the position of mobile object 1. Positioning device 18 is, for example, a Global Navigation Satellite System (GNSS) receiver, and identifies the position of mobile object 1 based on traffic signals received from GNSS satellites and outputs the position information. The position information of mobile object 1 may be estimated from the position of a Wi-Fi base station to which a communication device (described later) is connected.

[0032] The mode selector switch 22 is a switch operated by the occupant. The mode selector switch 22 may be a mechanical switch or a GUI (Graphical User Interface) switch set on a touch panel. The mode selector switch 22 accepts an operation to switch the driving mode between, for example, Mode A: an assist mode in which one of steering operation and acceleration / deceleration control is performed by the occupant and the other is performed automatically; Mode A-1: in which steering operation is performed by the occupant and acceleration / deceleration control is performed automatically; Mode A-2: in which acceleration / deceleration operation is performed by the occupant and steering control is performed automatically; Mode B: a manual driving mode in which steering operation and acceleration / deceleration operation are performed by the occupant; and Mode C: an automatic driving mode in which operation control and acceleration / deceleration control are performed automatically.

[0033] The locomotion mechanism 30 is a mechanism for moving the mobile object 1 on a road. The locomotion mechanism 30 is, for example, a group of wheels including steering wheels and drive wheels. The locomotion mechanism 30 may also be legs for multi-legged walking.

[0034] The drive unit 40 outputs force to the movement mechanism 30 to move the moving body 1. For example, the drive unit 40 includes a motor that drives the drive wheels, a battery that stores power to be supplied to the motor, a steering device that adjusts the steering angle of the steering wheels, etc. The drive unit 40 may also include an internal combustion engine or a fuel cell as a driving force output means or a power generation means. The drive unit 40 may further include a brake device that utilizes frictional force or air resistance.

[0035] The external notification device 50 is, for example, a lamp, a display device, a speaker, or the like, provided on an outer panel of the mobile object 1, and configured to notify information to the outside of the mobile object 1. The external notification device 50 operates differently depending on whether the mobile object 1 is traveling on a sidewalk or a roadway. For example, the external notification device 50 is controlled to emit a lamp when the mobile object 1 is traveling on a sidewalk and not emit a lamp when the mobile object 1 is traveling on a roadway. The light color of this lamp is preferably a color specified by law. The external notification device 50 may be controlled to emit a green lamp when the mobile object 1 is traveling on a sidewalk and emit a blue lamp when the mobile object 1 is traveling on a roadway. If the external notification device 50 is a display device, the external notification device 50 displays a message in text or graphics indicating that the mobile object 1 is traveling on a sidewalk when the mobile object 1 is traveling on a sidewalk.

[0036] FIG. 2 is a perspective view of the moving body 1 as seen from above. In the figure, FW is a steering wheel, RW is a driving wheel, SD is a steering device, MT is a motor, and BT is a battery. The steering device SD, motor MT, and battery BT are included in the drive device 40. AP is an accelerator pedal, BP is a brake pedal, WH is a steering wheel, SP is a speaker, and MC is a microphone. The moving body 1 shown in the figure is a single-seat moving body, and an occupant P is seated in the driver's seat DS and wearing a seat belt SB. Arrow D1 indicates the direction of travel (velocity vector) of the moving body 1. The external detection device 10 is located near the front end of the moving body 1, the internal camera 16 is located in a position where it can capture an image of the occupant P's head from in front of the occupant P, and the mode selector switch 22 is located in the boss portion of the steering wheel WH. An external notification device 50 serving as a display device is provided near the front end of the moving body 1.

[0037] Returning to FIG. 1, the storage device 70 is a non-transitory storage device such as a hard disk drive (HDD), flash memory, or random access memory (RAM). The storage device 70 stores map information 72, a program 74 executed by the control device 100, and the like. In the figure, the storage device 70 is illustrated outside the frame of the control device 100, but the storage device 70 may be included in the control device 100. The storage device 70 may also be provided on a server (not shown).

[0038] [Control device] The control device 100 includes, for example, a road type recognition unit 120, a recognition unit 130, and a control unit 140. These components are realized by, for example, a hardware processor such as a CPU (Central Processing Unit) executing a program (software) 74. Some or all of these components may be realized by hardware (including circuitry) such as an LSI (Large Scale Integration), an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or a GPU (Graphics Processing Unit), or may be realized by a combination of software and hardware. The program may be stored in the storage device 70 in advance, or may be stored in a removable storage medium (non-transitory storage medium) such as a DVD or CD-ROM, and installed in the storage device 70 by inserting the storage medium into a drive device.

[0039] The road type recognition unit 120 recognizes whether the moving object 1 is moving on a roadway or a sidewalk. The road type recognition unit 120 recognizes whether the moving object 1 is moving on a roadway or a sidewalk, for example, by analyzing an image captured by an external camera of the external detection device 10. The external camera may be a plurality of cameras (for example, a stereo camera). The output of a radar device, a LIDAR, a sensor fusion device, etc. may be used supplementarily.

[0040] The road type recognition unit 120 recognizes whether the moving object 1 is traveling on a roadway or a sidewalk based on an image captured by an external camera. For example, the road type recognition unit 120 classifies and labels each pixel in an image frame into a class using semantic segmentation, and imagines multiple virtually separated regions and their boundaries. The road type recognition unit 120 performs processing by classifying at least the regions of the image captured by the external camera into a region where the moving object 1 exists (hereinafter referred to as the current region) and a region adjacent to the current region where the moving object 1 does not exist (hereinafter referred to as the adjacent region). For example, the road type recognition unit 120 uses bicycle signs marked on the road to recognize whether the region of interest (e.g., the current region) is a sidewalk or a roadway. Details of this processing will be described later.

[0041] The recognition unit 130 recognizes objects present around the moving body 1 based on the output of the external detection device 10. The objects include some or all of the following obstacles: moving bodies such as vehicles, bicycles, and pedestrians; road boundary lines such as road markings, steps, guardrails, road shoulders, and median strips; road structures such as road signs (traffic signs) and billboards; and objects lying (fallen) on the moving body. The recognition unit 130 acquires information such as the presence, position, and type of other moving bodies by inputting images captured by the external camera of the external detection device 10 into a trained model that is trained to output information such as the presence, position, and type of an object when an image captured by the external camera of the external detection device 10 is input. The type of other moving bodies can also be estimated based on the size in the image or the intensity of the reflected wave received by the radar device of the external detection device 10. The recognition unit 130 acquires the speed of other moving bodies detected by the radar device using Doppler shift, for example.

[0042] The control unit 140 controls the drive unit 40 according to, for example, the set driving mode. The moving object 1 may execute only some of the driving modes described below, but in all cases, the control unit 140 sets different speed limits for when the moving object 1 moves on a roadway and when it moves on a sidewalk. In this case, the mode selector switch 22 may be omitted.

[0043] In mode A-1, the control unit 140 refers to the information on the course and objects based on the output of the recognition unit 130, and controls the motor MT of the drive unit 40 so that when the moving body 1 moves on the road, the distance to the object in front of the moving body 1 is maintained at a certain level or more, and if the distance to the object in front of the moving body 1 is sufficiently long, the moving body 1 moves at a first speed V1 (for example, a speed of more than 10 km / h and less than several tens of km / h).

[0044] When the moving object 1 moves on a sidewalk, the control unit 140 controls the motor MT of the drive unit 40 so that the moving object 1 maintains a certain distance from an object in front of the moving object 1 or more, and if the distance to the object in front of the moving object 1 is sufficiently long, the moving object 1 moves at a second speed V2 (for example, a speed less than 10 km / h). This function is similar to the adaptive cruise control (ACC) function of a vehicle in which the first speed V1 or the second speed V2 is set as a set speed, and techniques used in ACC can be utilized. In mode A-1, the control unit 140 controls the steering device SD to change the steering angle of the steered wheels based on the amount of operation of the operator 14, such as a steering wheel. This function is similar to the function of a power steering device, and techniques used in power steering devices can be utilized. Note that the moving object 1 may have a steering device in which the operator 14 and a steering mechanism are mechanically connected, rather than electronically controlling steering.

[0045] In mode A-2, the control unit 140 references the information about the traveling path and objects based on the output of the recognition unit 130, generates a target trajectory that allows the moving object 1 to avoid objects on the traveling path, and controls the steering device SD of the drive unit 40 so that the moving object 1 moves along the target trajectory. Regarding acceleration and deceleration, the control unit 140 controls the motor MT of the drive unit 40 based on the speed of the moving object 1 and the amount of operation of the accelerator pedal or the brake pedal. When the moving object 1 is moving on a roadway, the control unit 140 controls the motor MT of the drive unit 40 with a first speed V1 as the upper limit speed (in mode A-2, this means that the moving object 1 will not accelerate even if a further acceleration command is given once the upper limit speed has been reached), and controls the drive unit 40 with a second speed V2 as the upper limit speed when the moving object 1 is moving on a sidewalk.

[0046] In mode B, the control unit 140 controls the motor MT of the drive unit 40 based on the speed of the moving object 1 and the amount of operation of the accelerator pedal or the brake pedal. When the moving object 1 is moving on a roadway, the control unit 140 controls the motor MT of the drive unit 40 with a first speed V1 as the upper limit speed (in mode B, this means that the moving object 1 will not accelerate even if a further acceleration command is given once the upper limit speed has been reached), and when the moving object 1 is moving on a sidewalk, the control unit 140 controls the motor MT of the drive unit 40 with a second speed V2 as the upper limit speed. Steering is the same as in mode A-1. Hereinafter, the mode when traveling on a roadway may be referred to as mode B1 (second movement mode), and the mode when traveling on a sidewalk may be referred to as mode B2 (first movement mode).

[0047] In mode C, the control unit 140 references the information on the path and objects based on the output of the recognition unit 130, generates a target trajectory that allows movement while avoiding objects within the path, and controls the drive unit 40 so that the moving body 1 moves along the target trajectory. Even in mode C, the control unit 140 controls the drive unit 40 with a first speed V1 as the upper limit speed when the moving body 1 is moving on a roadway, and controls the drive unit 40 with a second speed V2 as the upper limit speed when the moving body 1 is moving on a sidewalk.

[0048] [Processing using bicycle signs] The road type recognition unit 120 recognizes whether an area of interest is a roadway or a sidewalk based on bicycle signs that are detected by the external detection device 10 and that guide bicycle passage. The control unit 140 controls the mobile object 1 based on the recognition result of the road type recognition unit 120. For example, when the mobile object 1 is located on a road that the road type recognition unit 120 recognizes as a roadway, the control unit 140 controls the mobile object 1 in a roadway mode for traveling on the roadway. The roadway mode is a mode in which the upper limit speed during traveling is limited to a first speed. When the mobile object 1 is located on a road that the road type recognition unit 120 recognizes as a sidewalk, the control unit 140 controls the mobile object 1 in a sidewalk mode for traveling on the sidewalk. The sidewalk mode is a mode in which the upper limit speed during traveling is limited to a second speed.

[0049] The road type recognition unit 120 recognizes whether the area of interest is a roadway or a sidewalk based on the bicycle markings and their orientation. FIG. 3 is a diagram showing an example of a road including bicycle markings. While FIG. 3 (and subsequent figures) uses a bird's-eye view from above for ease of explanation, each process may be performed on an image plane. As shown in FIG. 3, roads R1 and R2 exist, with road R1 being the roadway and road R2 being the sidewalk. A bicycle marking B1 is marked on road R1. A moving object 1 is traveling in the direction D1. The road type recognition unit 120 recognizes the bicycle markings and the direction indicated by the arrow AW1 on the bicycle marking. The guide direction indicated by the arrow AW1 is the same as the direction D1. The guide direction is the direction in which a bicycle should travel. For example, it is recommended that a bicycle travel along the road in the guide direction, above the bicycle markings. The road type recognition unit 120 recognizes that a road with a sidewalk on the left side of the guide direction (the right side if the law stipulates keeping to the right) is a sidewalk, and that a road on the right side is a roadway.

[0050] 4 is a diagram showing another example of a road including bicycle markings. The road type recognition unit 120 recognizes the direction indicated by the bicycle marking B2. The guide direction indicated by the bicycle marking B2 is the same as the direction D1. The road type recognition unit 120 recognizes that a road with a sidewalk on the left side of the guide direction is a sidewalk, and that a road on the right side is a roadway.

[0051] The storage device 70 stores characteristic information for various bicycle signs, such as bicycle sign B1 and bicycle sign B2. The characteristic information is information indicating characteristics (information indicating edge distribution, brightness distribution, etc.) obtained by image recognition processing of captured images of bicycle signs B1 and B2. The road type recognition unit 120 recognizes bicycle signs and the orientation indicated by the bicycle signs based on the characteristic information obtained from the captured images of bicycle signs B1 or B2 and the characteristic information stored in the storage device 70.

[0052] (Processing taking into account road width) The road width may be taken into consideration in the above process. The road type recognition unit 120 may recognize whether the area of interest is a roadway or a sidewalk based on the width of the bicycle sign and the roads to the left and right of the bicycle sign. For example, the road type recognition unit 120 may recognize the narrower road between a first road including a bicycle sign and a second road adjacent to the first road as a sidewalk, and the wider road as a roadway.

[0053] A second road adjacent to a first road includes, for example, a second road adjacent to the first road via a boundary object that separates the first road from the second road. A boundary object is, for example, a narrow object that is provided at the boundary between a sidewalk and a roadway, such as a curb, a fence, a guardrail, plants, or a dividing line marked on a road.

[0054] For example, as shown in FIG. 4, a boundary object OB is located between roads R1 and R2 to the left of bicycle sign B2. The road type recognition unit 120 may recognize the width of road R1 and the width of road R2, and recognize the narrower road R2 as a sidewalk. The width refers to the length of the road in the width direction. The width of road R1 may be, for example, the width excluding bicycle sign B2.

[0055] When a bicycle marking is provided on a first road (road R1 or road R2) and a boundary object exists adjacent to the bicycle marking that separates the first road from a second road adjacent to the first road, the road type recognition unit 120 may recognize whether the second road is a roadway or a sidewalk based on the presence or absence of vehicles on the first road and the second road and / or the width of the first road and the width of the second road. This is because the width can be determined more accurately when a boundary object exists. As will be described later, a road on which a vehicle exists is determined to be a roadway, and a road on which no vehicle exists is determined to be a sidewalk.

[0056] Even when bicycle signs are posted on the sidewalk as shown in FIG. 5 described later, the road with the shorter width between the first road and the second road may be recognized as the sidewalk as described above.

[0057] (Processing taking into account signs) The road type recognition unit 120 may recognize whether the area of interest is a roadway or a sidewalk based on signs placed near the bicycle markings. FIG. 5 is a diagram showing another example of a road including bicycle markings. In the example of FIG. 5, a bicycle marking B3 is placed on road R2, and a lane marking is placed next to bicycle marking B2 (in the opposite direction from the boundary object). Bicycles are recommended to ride between the lane marking and the boundary object (guide lane).

[0058] Bicycle sign B3 does not have a marking indicating the direction of travel. For example, if bicycle sign B3 is displayed on a sidewalk, bicycles can travel in either direction. For example, a bicycle can travel in the direction D1 or in the opposite direction to direction D1. For this reason, even if the road type recognition unit 120 recognizes bicycle sign B3, it may be difficult to determine whether the area of interest is a roadway or a sidewalk because the bicycle sign does not have a direction as described above.

[0059] Therefore, the road type recognition unit 120 recognizes whether the area of interest is a roadway or a sidewalk based on the sign C placed near the bicycle sign. The area near the bicycle sign refers to a guide lane or a sign placed near the guide lane.

[0060] FIG. 6 is a diagram showing examples of signs. Sign C1 indicates that road R2 is a road on which pedestrians and bicycles can travel. Sign C2 is a sign indicating that a pedestrian lane and a bicycle lane are separated. A pedestrian lane is a lane where pedestrians have priority or where pedestrians are encouraged (or permitted) to travel. A bicycle lane is a lane where bicycles have priority or where bicycles are encouraged (or permitted) to travel. Whether it is sign C1 or sign C2, when these signs C are placed near bicycle markings, it is clear that road R2 is not a roadway but a specified area different from the roadway.

[0061] As described above, the road type recognition unit 120 recognizes whether the area of interest is a roadway or a sidewalk based on the bicycle markings B3 and signs. In the above example, the road type recognition unit 120 refers to signs when the bicycle markings do not include information indicating a guide direction, but instead, even when the bicycle markings include information indicating a guide direction, the road type recognition unit 120 may refer to signs to recognize the type of road in the area of interest.

[0062] By using bicycle signs and road signs as described above, the road type recognition unit 120 can recognize the type of road with greater accuracy.

[0063] (Processing using elements contained in roads) The road type recognition unit 120 may use bicycle signs and various elements included in the road to recognize whether the area of interest is a roadway or a sidewalk.

[0064] Fig. 7 is a diagram for explaining processing using various elements. The explanation will focus on differences from Fig. 5. The various elements include, in addition to signs C, static obstacles (standing signs SB, mailboxes, telephone poles, trash cans, etc.), vehicles M, road markings RM, crosswalks PC, and the shapes of boundary objects.

[0065] For example, the road type recognition unit 120 determines that an area that satisfies many of the following conditions is a roadway, and that an area that does not satisfy many of the following conditions is a sidewalk. The conditions are, for example, that there are no static obstacles, that a vehicle M is stopped or moving, that there are no pedestrians, that a road marking RM is present on the road surface, and that a crosswalk PC is present. An area in which a boundary object OB is a curb and the step of the curb descends may be determined to be a roadway. For example, if the boundary object OB descends from road R2 to road R1, as shown by AT representing a cross section of a curb as viewed from the X direction in FIG. 7, road R1 may be determined to be a roadway.

[0066] The road type recognition unit 120 may recognize that the area of interest is a roadway if a bicycle sign is present and a predetermined number of the above conditions (one or more conditions) are met (see FIGS. 3 and 4).The road type recognition unit 120 may recognize that the area of interest is a sidewalk if a bicycle sign is present and a predetermined number of the above conditions (one or more conditions) are not met (see FIGS. 5 and 7).

[0067] As shown in FIG. 8, the road type recognition unit 120 may derive a score using information (bicycle signs, various elements) obtained from the recognition results of the recognition unit 130, and recognize whether the area of interest is a roadway or a sidewalk based on the derived score.

[0068] 8 is a diagram showing an example of a score derivation method. For example, the recognition result of the recognition unit 130 is applied to a score derivation method according to the recognition result, and a score according to the recognition result is derived. For example, a score according to bicycle signs and their orientation, a score according to the presence or absence of signs and the information indicated by the signs, a score according to road markings (the presence or absence of road markings and the information indicated by the road markings) or the presence or absence of pedestrian crossings, a score according to road width, a score according to the presence (or distribution) of traffic participants, a score according to the presence or absence of static obstacles, a score according to the shape of boundary objects, etc. Scores are given according to, for example, the presence or absence, type, size, width, etc. of elements.

[0069] The road type recognition unit 120 compares an integrated score obtained by integrating the scores derived as described above with a score threshold to recognize whether the area of interest is a roadway or a sidewalk. For example, if the integrated score indicates the likelihood of a roadway, the road type recognition unit 120 recognizes that the area of interest is a roadway if the integrated score is equal to or greater than the threshold. For example, if many of the above conditions are satisfied, the integrated score tends to be higher than if many of the above conditions are not satisfied.

[0070] For example, if the integrated score is a score indicating the likelihood of a sidewalk, the road type recognition unit 120 recognizes that the area of interest is a sidewalk if the integrated score is equal to or greater than a threshold. For example, if many of the above conditions are not met, the integrated score tends to be higher than if many of the above conditions are met.

[0071] [flowchart] 9 is a flowchart showing an example of the flow of processing executed by the control device 100. First, the recognition unit 130 recognizes the bicycle sign and its orientation (or lack thereof) (step S100). Next, the recognition unit 130 recognizes elements contained in the road other than the bicycle sign (such as the aforementioned signs and the presence or absence of vehicles) (step S102). Next, the road type recognition unit 120 derives a score based on the recognition results of steps S100 and S102 (step S104), and recognizes whether the area of interest is a roadway or a sidewalk based on the derived score (step S106).

[0072] Next, when the road type recognition unit 120 recognizes that the area of interest is a roadway, the control unit 140 moves the moving object 1 in roadway mode when moving through the area of interest (steps S108, S110). When the road type recognition unit 120 recognizes that the area of interest is a sidewalk, the control unit 140 moves the moving object 1 in sidewalk mode when moving through the area of interest (steps S108, S112). This completes the processing of one routine of this flowchart.

[0073] As described above, the control device 100 can accurately recognize the type of road in the area of interest, thereby enabling the control device 100 to control the moving object 1 in a manner appropriate to the surrounding situation.

[0074] [Variations] If the orientation of the bicycle marking can be recognized, the road type recognition unit may recognize whether the area of interest is a roadway or a sidewalk based on the bicycle marking and its orientation, and if the orientation of the bicycle marking cannot be recognized, may recognize whether the area of interest is a roadway or a sidewalk based on the bicycle marking, the width of the road on either side of the bicycle marking, or one or both of signs placed near the bicycle marking.

[0075] 10 is a flowchart showing an example of the flow of processing executed by the control device 100 of the modified example. First, the recognition unit 130 attempts to recognize the bicycle markings and their orientation (step S200). If the bicycle markings and their orientation are recognized, the road type recognition unit 120 recognizes the type of road based on the bicycle markings and their orientation (step S204). For example, if the bicycle markings and their orientation are recognized, the road type recognition unit 120 recognizes that the area where the bicycle markings are displayed is a roadway.

[0076] If the bicycle markings and their orientations cannot be recognized, the road type recognition unit 120 recognizes the type of road based on the bicycle markings and various elements of the road (step S206).

[0077] Next, when the road type recognition unit 120 recognizes that the area of interest is a roadway, the control unit 140 moves the moving object 1 in roadway mode when moving through the area of interest (steps S208, S210). When the road type recognition unit 120 recognizes that the area of interest is a sidewalk, the control unit 140 moves the moving object 1 in sidewalk mode when moving through the area of interest (steps S208, S212). This completes the processing of one routine of this flowchart.

[0078] As described above, the control device 100 of the modified example changes the process for recognizing the road type depending on the results of recognizing the orientation of the bicycle sign, thereby reducing the processing load while accurately recognizing the road type and enabling control of the moving object according to the road type.

[0079] In the above example, the road type recognition unit 120 has been described as deriving a score according to the recognition result by applying a score deriving method according to the recognition result. However, instead, an integrated score may be derived using a trained model. The trained model is a model that outputs a score according to input information on bicycle signs, their orientation, and various elements. This trained model is a model that has learned training data. The training data is information that associates information on bicycle signs, their orientation, various elements, and the type of road corresponding to that information (ground truth data indicating a roadway or a sidewalk). For example, when outputting a score that indicates that a target area is a roadway, the trained model outputs a high score when inputting information on bicycle signs corresponding to the roadway, their orientation, and various elements.

[0080] The trained model may also be a model that, when an image is input, outputs a score indicating whether the area shown in the image is a roadway or a sidewalk. For example, when an image of a roadway is input, this trained model outputs a score indicating that the area of interest is a roadway, and when an image of a sidewalk is input, it outputs a score indicating that the area of interest is a sidewalk. The training data used to train this trained model is, for example, an image of a roadway (roadway image) including information indicating bicycle signs and information indicating the orientation of the bicycle signs, and information associating this information with the road type corresponding to this information (e.g., ground truth data indicating a roadway). By training this training data, a score that takes into account the information indicating the bicycle signs and the orientation of the bicycle signs is output.

[0081] In this embodiment, bicycle signs may be used to determine the road type. For example, when an image is input, the trained model derives a score that estimates the area as a roadway or a sidewalk. The road type recognition unit 120 may correct the derived score using the presence or absence of bicycle signs and their orientation, and recognize the road type based on the correction result. For example, even if a score that estimates the area of interest as a sidewalk is derived, if the area of interest contains a bicycle sign indicating the direction of bicycle travel (e.g., bicycle sign B1 in FIG. 3 or bicycle sign B2 in FIG. 4), the road type recognition unit 120 may recognize the area of interest as not a sidewalk or may determine that the road type of the area of interest cannot be recognized. If the road type of the area of interest cannot be recognized, the road type recognition unit 120 may re-perform the process of recognizing the road type of the area of interest, or may perform the process of recognizing the road type of an area shifted from the area of interest.

[0082] The process by the control device 100 to recognize whether an area of interest is a roadway or a sidewalk using bicycle signs may be applied in various modes (modes A and C). For example, in various modes, when traveling on a sidewalk, the speed limit may be limited to the second speed.

[0083] As described above, the control device 100 can accurately recognize the type of road in the area of interest based on bicycle signs that guide the passage of bicycles detected by a detection device that detects the external conditions of the moving body, and can control the moving body according to the type of road.

[0084] The above-described embodiment can be expressed as follows. a storage medium for storing computer-readable instructions; a processor connected to the storage medium; The processor executes the computer-readable instructions to: Controlling a mobile object that can move on both a roadway and a predetermined area different from the roadway; Based on a bicycle sign that guides the passage of a bicycle detected by a detection device that detects the external situation of the moving body, the system recognizes whether the area of interest is a roadway or a sidewalk; controlling the moving object based on the result of the recognition; Control device for a moving object.

[0085] The above describes the form for carrying out the present invention using an embodiment, but the present invention is not limited to such an embodiment, and various modifications and substitutions can be made within the scope that does not deviate from the gist of the present invention. [Explanation of symbols]

[0086] 10 External sensing devices 12 Mobile Sensor 14 Controls 16 Internal Camera 18 Positioning equipment 22 Mode switch 30 Moving mechanism 40 Drive unit 50 External alarm device 70 Storage device 100 control device 120 Road type recognition unit 130 Recognition part 140 Control Unit

Claims

1. A control device for a mobile body that can move on both a roadway and a sidewalk, a road type recognition unit that recognizes whether a target area is a roadway or a sidewalk based on bicycle signs that guide bicycle traffic and are detected by a detection device that detects the external conditions of the moving object; a control unit that controls the moving object based on the result of the recognition by the road type recognition unit; A control device for a moving body comprising:

2. the road type recognition unit recognizes whether the area of interest is a roadway or a sidewalk based on the bicycle marking and the orientation of the bicycle marking; The control device for a moving body according to claim 1 .

3. the road type recognition unit recognizes whether the area of interest is a roadway or a sidewalk based on the bicycle sign and a sign provided near the bicycle sign; The control device for a moving body according to claim 1 .

4. the road type recognition unit recognizes whether the area of interest is a roadway or a sidewalk based on the widths of the bicycle sign, a first road on which the bicycle sign is displayed, and a second road adjacent to the first road; The control device for a moving body according to claim 1 .

5. the road type recognition unit recognizes the narrower road of the first road and the second road as a sidewalk, and recognizes the wider road as a roadway; The control device for a moving body according to claim 4.

6. When the bicycle marking is provided on a first road and a boundary object that separates the first road from a second road adjacent to the first road is present adjacent to the bicycle marking, the road type recognition unit recognizes whether the second road is a roadway or a sidewalk based on the presence or absence of vehicles on the first road and the second road and / or the width of the first road and the width of the second road. The control device for a moving body according to claim 1 .

7. The boundary object is a curb, a fence, a guardrail, a plant, or a marking on a road. The control device for a moving body according to claim 6.

8. the road type recognition unit recognizes a road on which no vehicle exists, of the first road and the second road, as a sidewalk; The control device for a moving body according to claim 7.

9. the road type recognition unit recognizes the narrower road of the first road and the second road as a sidewalk; The control device for a moving body according to claim 7.

10. the road type recognition unit identifies the narrower road from the first road and the second road, and recognizes the identified road as a sidewalk if no vehicle is present on the identified road; The control device for a moving body according to claim 7.

11. The road type recognition unit Recognizing whether the area of interest is a roadway or a sidewalk based on the bicycle sign, the direction of the bicycle sign, the width of the road on either side of the bicycle sign, and signs provided near the bicycle sign; The control device for a moving body according to claim 1 .

12. The road type recognition unit If the orientation of the bicycle marking can be recognized, the system recognizes whether the area of interest is a roadway or a sidewalk based on the bicycle marking and the orientation of the bicycle marking; If the orientation of the bicycle marking cannot be recognized, the system recognizes whether the area of interest is a roadway or a sidewalk based on the bicycle marking and one or both of the width of the road on either side of the bicycle marking and signs provided near the bicycle marking. The control device for a moving body according to claim 1 .

13. The control unit When the moving object is located on a road that the road type recognition unit has recognized as a roadway, the moving object is controlled in a roadway mode for traveling on the roadway; When the moving object is located on a road that the road type recognition unit has recognized as a sidewalk, the moving object is controlled in a sidewalk mode for traveling on the sidewalk. The control device for a moving body according to claim 1 .

14. The control unit When the moving object is located on a road that is recognized as a roadway by the road type recognition unit, an upper limit speed at which the moving object travels is limited to a first speed; When the moving object is located on a road that the road type recognition unit has recognized as a sidewalk, the upper limit speed at which the moving object travels is limited to a second speed that is lower than the first speed. The control device for a moving body according to claim 1 .

15. A control device for a mobile body that can move on both a roadway and a sidewalk, Based on a bicycle sign that guides the passage of a bicycle detected by a detection device that detects the external situation of the moving body, the system recognizes whether the area of interest is a roadway or a sidewalk; controlling the moving object based on the result of the recognition; A method for controlling a moving object.

16. A control device for a mobile object that can move on both roadways and sidewalks, Based on bicycle signs that guide the passage of bicycles detected by a detection device that detects the external conditions of the moving body, the system recognizes whether the area of interest is a roadway or a sidewalk; controlling the moving object based on the result of the recognition; A storage medium on which a program is stored.

Images