Sign recognition device, sign recognition method, and storage medium

Abstract

A sign recognition device recognizes a prescribed sign drawn so as to extend in a fixed direction on a movement path of a moving body that is capable of moving on both a roadway and a prescribed region different from the roadway. The sign recognition device extracts individual contour points positioned in a contour part of the region of the sign captured in a two-dimensional image which is captured by an imaging device that images at least the movement path, projects the position of each contour point extracted in the image into a two-dimensional projection space, detects the direction in which the sign is drawn on the basis of the direction of a contour line segment between two adjacent contour points, and outputs end point information indicating an end point of the region in which the sign is drawn.

Description

Sign recognition device, sign recognition method, and storage medium

[0001] The present invention relates to a sign recognition device, a sign recognition method, and a storage medium.

[0002] In recent years, efforts have been actively made to provide access to a sustainable transportation system that takes into account people in vulnerable positions among traffic participants. In order to achieve this, research and development focused on further improving traffic safety and convenience through research and development related to autonomous driving technology have been carried out.

[0003] For this reason, conventionally, the practical application of a moving body that can move on both sidewalks and roadways has been promoted. Such a moving body needs to move according to road signs drawn on sidewalks, roadways, etc. For this purpose, at least, it is important to recognize pavement markings drawn on the road surface on the moving route, particularly, guiding markings.

[0004] Regarding this, a temporary stop line detection device is disclosed which detects a temporary stop line candidate from a captured image of the front of a vehicle using deep learning or edge processing or the like, acquires both side end points in the length direction of the temporary stop line candidate and both side end points of the roadway at both outer sides in the width direction of the roadway, and discriminates whether the temporary stop line in front of the host vehicle is a temporary stop line on the host lane side or a temporary stop line on the oncoming lane side based on the positional relationship between both side end points of the temporary stop line candidate and both side end points of the roadway (for example, see Patent Document 1).

[0005] Japanese Patent Application Laid-Open No. 2021-123511

[0006] By the way, in autonomous driving technology, it is an issue to recognize road signs existing on the moving route of a moving body and safely run the moving body. In the conventional technology, it is necessary to acquire both end points of both side end points of a temporary stop line candidate and both side end points of the roadway from a captured image. However, in a situation where it is difficult to recognize a temporary stop line from an image, such as a sidewalk or a stop line with a special shape, it is conceivable that the temporary stop line may be misrecognized. However, in the conventional technology, the above-described situation has not been considered, and there are cases where a temporary stop line cannot be suitably recognized.

[0007] This invention was made based on the above-mentioned problem recognition and aims to provide a sign recognition device, a sign recognition method, and a storage medium that can more favorably recognize signage drawn on the travel path of a moving object. In other words, this invention aims to recognize road signs present on the travel path of a moving object and to enable the moving object to travel safely. And, in turn, it aims to contribute to the development of a sustainable transportation system.

[0008] The marking recognition device, marking recognition method, and storage medium according to this invention employ the following configuration: (1) A marking recognition device according to one aspect of this invention is a marking recognition device that recognizes a predetermined marking drawn so as to extend in a certain direction on the movement path of a moving body that can move both on a roadway and in a predetermined area different from the roadway, and is a marking recognition device that extracts each contour point located on the contour portion of the area of ​​the marking captured in a two-dimensional image captured by an imaging device that captures at least the movement path, projects the position of each contour point extracted in the image into a two-dimensional projection space, detects the orientation in which the marking is drawn based on the direction of the contour line segment between two adjacent contour points, and outputs endpoint information representing the endpoint of the area in which the marking is drawn.

[0009] (2) In the embodiment of (1) above, the system includes: an image acquisition unit that acquires the image and classifies the pixels constituting the acquired image into predetermined classes; a contour extraction unit that extracts each of the contour points located on the contour line of the marking from among the pixels classified into the classes; a projection transformation unit that projects each of the extracted contour points onto the projection space; a contour line orientation detection unit that detects the orientation of the contour line with respect to the direction of travel of the moving body based on the direction of the contour line segment; a contour line classification unit that groups together contour points representing the same contour line based on the distance between the contour line and the origin of the projection space; and a straightening unit that straightens the contour line represented by each of the contour points belonging to the same group and outputs endpoint information based on the straightened contour line.

[0010] (3) In the embodiment of (2) above, the contour extraction unit extracts a marking pixel that indicates it is a marking from among the pixels classified into the class, and detects the contour point in a predetermined direction.

[0011] (4) In the embodiment of (2) above, the contour line direction detection unit displays the direction of each contour line segment corresponding to each contour point in a histogram, and detects the direction of the contour line based on the distribution of directions of the contour line segments displayed in the histogram.

[0012] (5) In the embodiment of (4) above, the contour line direction detection unit detects the direction of the contour line segment with the highest distribution in the histogram as the direction of the contour line.

[0013] (6) In the embodiment of (2) above, the contour line classification unit calculates the distance for each of the contour points and groups contour points whose calculated distances are within a predetermined distance range into the same distance group.

[0014] (7) In the embodiment of (6) above, the contour line classification unit groups contour points in the same direction into the same direction group based on the direction of each contour line segment corresponding to each contour point belonging to the same distance group.

[0015] (8) In the embodiment of (2) above, the straightening unit estimates a straight line represented by each of the contour points grouped into the same group, projects each of the contour points of the estimated straight line onto the straight line, thereby straightening the contour line represented by each of the contour points grouped into the same group, and outputs information representing the contour points located at both ends of the straightened contour line as the endpoint information.

[0016] (9): A marking recognition method according to one aspect of the present invention is a marking recognition method comprising: a computer that recognizes a predetermined marking drawn so as to extend in a certain direction on the movement path of a mobile body that can move both on a roadway and in a predetermined area different from the roadway; extracts each contour point located on the contour portion of the area of ​​the marking as captured in a two-dimensional image captured by an imaging device that captures at least the movement path; projects the positions of each contour point extracted in the image into a two-dimensional projection space; detects the orientation in which the marking is drawn based on the direction of the contour line segment between two adjacent contour points; and outputs endpoint information representing the endpoint of the area in which the marking is drawn.

[0017] (10): A storage medium according to one aspect of the present invention is a storage medium that stores a program which causes a computer that recognizes a predetermined marking drawn so as to extend in a certain direction on the movement path of a mobile body that can move both on a roadway and in a predetermined area different from the roadway, to extract each contour point located on the contour of the area of ​​the marking as captured in a two-dimensional image captured by at least an imaging device that captures the movement path, to project the positions of each contour point extracted in the image into a two-dimensional projection space, to detect the orientation in which the marking is drawn based on the direction of the contour line segment between two adjacent contour points, and to output endpoint information representing the endpoint of the area in which the marking is drawn.

[0018] According to the embodiments (1) to (10) described above, the indicator markings drawn on the travel path can be recognized more favorably.

[0019] This figure shows an example of the configuration of a mobile body equipped with a marking recognition device according to an embodiment, and a control device for the mobile body. This is a perspective view of the mobile body from above. This figure shows an example of the functional configuration of the road marking recognition unit. This is a flowchart showing an example of the processing flow for recognizing instruction markings performed by the road marking recognition unit. This figure schematically shows an example of processing by the image acquisition unit and contour extraction unit of the road marking recognition unit. This figure schematically shows an example of processing by the homography conversion unit of the road marking recognition unit. This is a schematic diagram (1) showing an example of processing by the contour direction detection unit of the road marking recognition unit. This is a schematic diagram (2) showing an example of processing by the contour direction detection unit of the road marking recognition unit. This is a schematic diagram (1) showing an example of processing by the contour clustering unit of the road marking recognition unit. This is a schematic diagram (2) showing an example of processing by the contour clustering unit of the road marking recognition unit. This is a schematic diagram (3) showing an example of processing by the contour clustering unit of the road marking recognition unit. This is a schematic diagram (1) illustrating an example of processing by the straight-line fitting unit of the road marking recognition unit. This is a schematic diagram (2) illustrating an example of processing by the straight-line fitting unit of the road marking recognition unit.

[0020] The following describes embodiments of the marking recognition device, marking recognition method, and storage medium of the present invention with reference to the drawings. In the following description, the marking recognition device is assumed to be mounted on a mobile body. The mobile body moves both on the roadway and in a predetermined area separate from the roadway. The mobile body may be referred to as, for example, a micromobility. For example, an electric kick scooter is a type of micromobility. The predetermined area is, for example, a sidewalk. The predetermined area may be part or all of a road shoulder, bicycle lane, public open space, etc., or it may include all of the sidewalk, road shoulder, bicycle lane, public open space, etc. In the following description, the predetermined area will be assumed to be a sidewalk. In the following description, the term "sidewalk" may be replaced with "predetermined area" as appropriate.

[0021] [Overall Configuration] Figure 1 is a diagram showing an example of the configuration of a mobile body equipped with a sign recognition device according to an embodiment, and a control device for the mobile body. The mobile body 1 includes, for example, an external detection device 10, a mobile body sensor 12, an operator 14, an internal camera 16, a positioning device 18, a mode switching switch 22, an HMI (Human Machine Interface) 26, a moving mechanism 30, a drive device 40, an external notification device 50, a storage device 70, and a control device 100. The configuration shown in Figure 1 is merely an example, and some of the configuration (for example, some configurations that are not essential for realizing the functions of the present invention) may be omitted, or other configurations may be added.

[0022] The external environment detection device 10 detects the external conditions of the moving object 1. For example, the external environment detection device 10 is a variety of devices whose detection range covers at least a portion of the area around the moving object 1 (including the direction of travel). The external environment detection device 10 includes, for example, an external camera 11, a radar device (not shown), a LiDAR (Light Detection and Ranging) (not shown), a sensor fusion device (not shown), and the like.

[0023] The external camera 11 is a digital camera that utilizes a solid-state image sensor such as a CCD (Charge Coupled Device) or CMOS (Complementary Metal Oxide Semiconductor). The external camera 11 is mounted at any location that can capture images of at least the area in front of the moving object 1. The external camera 11 periodically and repeatedly captures images of the area around the moving object 1 at predetermined time intervals. The external camera 11 may also be a stereo camera.

[0024] The external camera 11 is an example of an "imaging device".

[0025] A radar device (not shown) emits radio waves such as millimeter waves around the moving object 1 and detects radio waves reflected by objects (reflected waves) to detect at least the object's position (distance and direction). The radar device can be attached to any location on the moving object 1. A LIDAR (not shown) irradiates light (or electromagnetic waves with a wavelength close to light) around the moving object 1 and measures the scattered light. The LIDAR detects the distance to the target based on the time from emission to reception. The irradiated light is, for example, pulsed laser light. The LIDAR can be attached to any location on the moving object 1. A sensor fusion device (not shown) performs sensor fusion processing on some or all of the detection results from the external camera 11, the radar device, and the LIDAR to detect (recognize) the object's position, type, speed, etc.

[0026] The external environment detection device 10 outputs information (images, object positions, etc.) indicating the detection results from the external camera 11, radar system, LIDAR, sensor fusion device, etc. to the control device 100. The external environment detection device 10 may output the detection results from the external camera 11, radar system, LIDAR, sensor fusion device, etc. directly to the control device 100.

[0027] The moving sensor 12 includes, for example, a speed sensor, an acceleration sensor, a yaw rate (angular velocity) sensor, a compass sensor, and a manipulated amount detection sensor attached to the operator 14.

[0028] The control element 14 receives driving operations from the occupant of the mobile vehicle 1. The control element 14 includes, for example, controls for instructing acceleration and deceleration (e.g., an accelerator pedal, a brake pedal, a dial switch or lever for speed adjustment) and controls for instructing steering (e.g., a steering wheel). In this case, the mobile vehicle sensor 12 may also include an accelerator opening sensor, a brake pedal pressure sensor, a steering torque sensor, etc. The mobile vehicle 1 may also be equipped with controls other than those described above as the control element 14 (e.g., a non-annular rotary control, a joystick, a button, etc.).

[0029] The internal camera 16 captures images of at least the heads of the occupants of the mobile vehicle 1 from the front. The internal camera 16 is a digital camera that uses a solid-state image sensor such as a CCD or CMOS. The internal camera 16 outputs the captured images to the control device 100.

[0030] The positioning device 18 is a device that determines the position of the mobile body 1. The positioning device 18 is, for example, a GNSS (Global Navigation Satellite System) receiver, which determines the position of the mobile body 1 based on signals received from GNSS satellites and outputs it as position information. The position information of the mobile body 1 may be estimated, for example, from the position of a Wi-Fi base station to which a communication device (not shown) mounted on the mobile body 1 is connected.

[0031] The mode selector switch 22 is a switch operated by the occupant. The mode selector switch 22 receives instructions regarding the switching of the driving state of the mobile unit 1. The mode selector switch 22 may be a mechanical switch or a GUI (Graphical User Interface) switch set on a touch panel. The mode selector switch 22 receives an operation to switch the driving mode to one of the following: Mode A: an assist mode in which either steering or acceleration / deceleration control is performed by the occupant, and the other is performed automatically (there may be Mode A-1 in which steering is performed by the occupant and acceleration / deceleration control is performed automatically, and Mode A-2 in which acceleration / deceleration control is performed by the occupant and steering control is performed automatically), Mode B: a manual driving mode in which steering and acceleration / deceleration are performed by the occupant, or Mode C: an automatic driving mode in which operation control and acceleration / deceleration control are performed automatically. The operation to switch the above modes (or driving states) may be received by a microphone (not shown) or an internal camera 16 provided on the mobile unit 1 instead of the mode selector switch 22. In this case, the control device 100 may accept a mode switching operation based on the analysis results of the occupant's voice input by a microphone (not shown), or it may accept a mode switching operation based on the occupant's gestures or mouth movements obtained as a result of the analysis of the occupant's front image captured by the internal camera 16.

[0032] The HMI 26 presents (or informs, notifies) various information to the occupant of the mobile unit 1 and accepts input operations from the occupant. The HMI 26 includes various display devices, speakers, microphones, buzzers, touch panels, switches, keys, lamps, etc. For example, the HMI 26 informs the occupant of the driving status of the mobile unit 1, which is controlled by the driving control unit 140, in different notification modes depending on the driving status. The HMI 26 may, for example, present information from the control device 100 or present information acquired from an external device via a communication device (not shown).

[0033] The mobility mechanism 30 is a mechanism for moving the mobile body 1 on a road. The mobility mechanism 30 is, for example, a group of wheels including steering wheels and drive wheels. The mobility mechanism 30 may also be legs for multi-legged walking.

[0034] The drive unit 40 outputs force to the moving mechanism 30 to move the moving body 1. The drive unit 40 includes, for example, a motor that drives the drive wheels, a battery that stores the power supplied to the motor, and a steering device that adjusts the steering angle of the steering wheels. The drive unit 40 may also be equipped with an internal combustion engine or a fuel cell as a means of outputting driving force or a means of generating power. The drive unit 40 may further be equipped with a braking device that uses frictional force or air resistance.

[0035] The external notification device 50 is, for example, a lamp, display device, speaker, etc., provided on the outer panel of the mobile body 1 to notify information to the outside of the mobile body 1. The external notification device 50 notifies the area around the mobile body (within a predetermined distance from the mobile body 1) of the running state of the mobile body 1, which is controlled by the running control unit 140, in different notification modes depending on the running state.

[0036] Figure 2 is a perspective view of the mobile body 1 from above. In Figure 2, FW is the steering wheel, RW is the drive wheel, SD is the steering mechanism, MT is the motor, and BT is the battery. The steering mechanism SD, motor MT, and battery BT are included in the drive system 40. In Figure 2, AP is the accelerator pedal, BP is the brake pedal, WH is the steering wheel, SP is the speaker, and MC is the microphone. The mobile body 1 shown in Figure 2 is a single-seat mobile body, and the occupant P is seated in the driver's seat DS and wearing a seat belt SB. The arrow α1 shown in Figure 2 is the direction of travel (velocity vector) of the mobile body 1. In the mobile body 1, the external environment detection device 10 is located near the front end of the mobile body 1, the internal camera 16 is located in a position that can capture images of the occupant P's head from in front of the occupant P, and the mode switching switch 22 is located on the boss of the steering wheel WH. In the mobile body 1, an external notification device 50, which serves as a display device, is provided near the front end of the mobile body 1. An HMI 26, which serves as a display device, is provided in front of the occupant P riding in the mobile vehicle 1. The external notification device 50 may be formed integrally with the speaker SP, and the HMI 26 may be formed integrally with the speaker SP and the microphone MC.

[0037] Returning to Figure 1, the storage device 70 is a storage device (a storage device equipped with a non-transient storage medium) that constitutes a storage device such as a semiconductor memory element such as ROM (Read Only Memory), RAM (Random Access Memory), or flash memory, or a hard disk drive (HDD). Map information 72, a program 74 executed by the control device 100, etc. are stored in the storage device 70. In Figure 1, the storage device 70 is shown outside the frame of the control device 100, indicating that it is configured to be located outside the control device 100, but the storage device 70 may be included in (located inside) the control device 100.

[0038] The control device 100 includes, for example, a road marking recognition unit 120 and a driving control unit 140. The road marking recognition unit 120 and the driving control unit 140 are each realized by a hardware processor, such as a CPU (Central Processing Unit), executing a program (software) 74. Some or all of these components may be realized by hardware (including circuitry) such as LSIs (Large Scale Integration), SOCs (System On Chips), Application Specific Integrated Circuits (ASICs), programmable logic devices (e.g., Simple Programmable Logic Devices (SPLDs) or Complex Programmable Logic Devices (CPLDs), Field Programmable Gate Arrays (FPGAs)), or by the cooperation of software and hardware. The program may be stored in the storage device 70 in advance, or it may be stored on a removable storage medium (non-transient storage medium) such as a DVD or CD-ROM, and installed in the storage device 70 when the storage medium is inserted into the drive device.

[0039] The road marking recognition unit 120 recognizes road markings, particularly directional markings, drawn on the road surface such as a sidewalk or roadway on which the moving object 1 is moving. For example, the road marking recognition unit 120 recognizes directional markings drawn on the road surface in front of the moving object 1 by analyzing the image captured by the external camera 11. At this time, the road marking recognition unit 120 performs an analysis process (hereinafter referred to as "image analysis processing") on the image captured by the external camera 11 to recognize subjects in the image using, for example, a DNN (Deep Neural Network) model generated by the function of deep learning, which is one of the machine learning functions of AI (Artificial Intelligence), thereby classifying the regions in the image into regions for each subject. An example of this image analysis processing is semantic segmentation. In this case, the road marking recognition unit 120 classifies each pixel in the image frame into a class (such as roadways, sidewalks, road markings, road signs, road surface markings (including directional markings), and obstacles) and assigns a label to it. The road marking recognition unit 120 then recognizes the area labeled with a directional marking within the area labeled with a roadway or sidewalk in the area corresponding to the front of the moving object 1. However, the image analysis processing in the road marking recognition unit 120 may also recognize the area of ​​the directional marking using image processing analysis techniques. The road marking recognition unit 120 outputs information representing the endpoints of the recognized area of ​​the directional marking (hereinafter referred to as "directional marking information") to the driving control unit 140.

[0040] The road marking recognition unit 120 is an example of a "marking recognition device." The instruction marking information is an example of "endpoint information."

[0041] The driving control unit 140 controls the drive unit 40 according to a driving mode determined by a mode selector switch 22 or the like, from among a plurality of pre-set driving modes. In this case, if the driving mode determined by the mode selector switch 22 or the like is a driving mode in which either or both steering operation and acceleration / deceleration control are performed automatically (mode A (assist mode) or mode C (automatic driving mode)), the driving control unit 140 may generate a target trajectory for the moving body 1 to travel automatically (without the operation of the driver (occupant P)) based on the instruction marking information output by the road marking recognition unit 120. The driving control unit 140 may then control the steering device SD of the drive unit 40 so that the moving body 1 moves along the generated target trajectory. Here, with respect to acceleration and deceleration, the driving control unit 140 may control the motor MT of the drive unit 40 based on the speed of the moving body 1 and the amount of operation of the accelerator pedal AP or brake pedal BP.

[0042] [Functional Configuration of Road Marking Recognition Unit] Figure 3 shows an example of the functional configuration of the road marking recognition unit 120. The road marking recognition unit 120 includes, for example, an image acquisition unit 121, a contour extraction unit 122, a homography conversion unit 123, a contour line direction detection unit 124, a contour line clustering unit 125, and a linear fitting unit 126.

[0043] The image acquisition unit 121 acquires an image of the front of the moving object 1, which is output by the external camera 11. The image acquisition unit 121 performs image analysis processing on the acquired image (hereinafter referred to as the "input image"), such as the semantic segmentation process described above, to classify each pixel constituting the frame of the input image into a class and assign a label representing the corresponding class. The image acquisition unit 121 outputs the image composed of each labeled pixel (hereinafter referred to as the "analyzed image") to the contour extraction unit 122. The image acquisition unit 121 may also generate an analyzed image in which the value of each pixel constituting the input image (pixel value) is a value representing the assigned label (label value), that is, subjects of the same class captured in the input image have the same pixel value (=label value), and output the generated analyzed image to the contour extraction unit 122.

[0044] The image acquisition unit 121 is an example of an "image acquisition unit".

[0045] The contour extraction unit 122 extracts the contour line of the indication mark from the analysis image output by the image acquisition unit 121. At this time, the contour extraction unit 122 extracts each pixel located at the end (outer periphery) of the region of the pixel to which the label of the indication mark is assigned (hereinafter referred to as "indication mark pixel") among the respective pixels constituting the analysis image, as a contour point forming the contour line. More specifically, the contour extraction unit 122 sequentially moves the position of the pixel in the so-called raster order from the upper left pixel to the lower right pixel of the analysis image, and determines whether the pixel at the moved position is an indication mark pixel. Then, the contour extraction unit 122 sets the pixel first determined to be an indication mark pixel as a reference contour point forming the contour line, and from the position of this reference contour point, for example, counterclockwise, other indication mark pixels located at the end (outer periphery) of the region of the indication mark pixel are detected, thereby extracting a plurality of contour points forming the contour line of the indication mark, that is, extracting a contour point group of the outer periphery of the indication mark. In the following description, the contour point group forming the contour line of the indication mark is also referred to as a "contour line". In the following description, the part described as "contour line" can be appropriately read as "contour point group". The "contour line" may be read as "each of the plurality of contour points forming the contour line of the indication mark".

[0046] The contour extraction unit 122 may extract contour lines by other means, not limited to this method. For example, the contour extraction unit 122 first determines whether each pixel in the entire analysis image is an indicator pixel (i.e., it determines whether each pixel is an indicator pixel for one frame), and detects indicator pixels belonging to the same indicator region. Then, the contour extraction unit 122 uses the upper left indicator pixel in the detected same indicator region as the reference contour point for the contour line, and extracts the contour line of the indicator by sequentially tracing the indicator pixels located at the edge (outer perimeter) of the same indicator region, for example, counterclockwise from the position of this reference contour point. The direction in which the contour extraction unit 122 detects other indicator pixels from the position of the reference contour point is not limited to the counterclockwise direction described above. For example, the contour line may be extracted by detecting other indicator pixels located at the edge (outer perimeter) of the indicator pixel region, for example, clockwise. In this case, in the following explanation, for example, descriptions regarding the direction of the contour line (contour segment) or the direction in which contour points are connected should be interpreted as the opposite direction as appropriate.

[0047] The contour extraction unit 122 outputs an analyzed image (hereinafter referred to as the "contour image") to the homography conversion unit 123, in which information indicating that the contour lines extracted from the analyzed image are contour lines of an indicator marking has been further added. The contour extraction unit 122 may also generate a contour image in which the pixel value (or label value) of each contour line pixel constituting the analyzed image is set to a value indicating that it is a contour line of an indicator marking (for example, pixel value = "1"), and the pixel value (or label value) of pixels other than contour lines is set to a value indicating that it is not a contour line of an indicator marking (for example, pixel value = "0"), and output the generated contour image to the homography conversion unit 123.

[0048] As described above, the contour extraction unit 122 extracts the indicated pixels (contour lines) of the contour line by detecting the indicated pixels counterclockwise, for example, from the position of the indicated pixel used as a reference. Therefore, when the contour extraction unit 122 extracts the contour line, it also detects the direction in which the contour points located at two adjacent contour points forming the contour line are connected, that is, the direction of the contour line segment between two adjacent contour points. For this reason, when generating the contour line image, the contour extraction unit 122 generates a contour line image in which the pixel value of the pixel of the contour line represents that it is the contour line of the indicated indication and also represents the value of the direction of the contour line (contour line segment), and outputs the contour line image to the homography conversion unit 123. Here, when the contour line (each of a plurality of contour points) is detected by detecting other indicated pixels counterclockwise from the position of the reference contour point, as the value representing the direction of the contour line (contour line segment), it is sufficient that at least the directions from the upper side to the lower side, from the left side to the right side, from the lower side to the upper side, and from the right side to the left side of the contour line image are represented. This is the same even when the contour line (each of a plurality of contour points) is detected by detecting other indicated pixels clockwise from the position of the indicated pixel used as a reference, for example.

[0049] The contour extraction unit 122 is an example of a "contour extraction unit".

[0050] The homography transformation unit 123 performs a projection transformation (homography transformation) on the contour line image output by the contour extraction unit 122. At this time, the homography transformation unit 123 may generate an isomorphism (hereinafter referred to as the "projected image") by projecting the positions of each extracted contour point represented by the two-dimensional contour line image onto a coordinate system of a two-dimensional projection space with a predetermined position of the moving body 1 (for example, the center position of the moving body 1) as the origin (hereinafter referred to as the "moving body coordinate system"). In this case, the homography transformation unit 123 generates a projected image by projecting at least each contour point (contour) included in the contour line image onto the moving body coordinate system. For the sake of simplicity in the following explanation, the homography transformation unit 123 will be assumed to generate a projected image. The homography transformation unit 123 outputs the generated projected image to the contour line direction detection unit 124. The method for generating the projected image in the homography transformation unit 123, that is, the method for projecting the contour line image, is an existing method, so a detailed explanation will be omitted.

[0051] The homography transformation unit 123 is an example of a "projection transformation unit".

[0052] The contour line direction detection unit 124 detects the orientation of the contour line of the indicator markings shown in the projected image output by the homography conversion unit 123 with respect to the direction of movement of the moving object 1. In other words, the contour line direction detection unit 124 uses the direction of movement of the moving object 1 (the direction of arrow α1 shown in Figure 2) as the reference direction and detects the direction of the contour line representing the connection of contour points included in the projected image. More specifically, the contour line direction detection unit 124 calculates the angle of the contour line segment based on the direction of the contour line segment between two adjacent contour points that form the contour line, which are included in the projected image. The contour line direction detection unit 124 then creates a histogram of the calculated angles of each contour line segment, with the contour line direction (angle) on the X axis and the number of contour line segments on the Y axis. The contour line direction detection unit 124 then detects the direction of the contour line with respect to the direction of movement of the moving object 1 based on the distribution of contour line segments in the direction (angle) of the contour line segments shown in the created histogram. More specifically, the contour line direction detection unit 124 detects the direction (angle) of the contour line segment with the largest number of contour line segments in the created histogram as the direction of the contour line relative to the direction of movement of the moving object 1. However, the contour line direction detection unit 124 may also detect the direction of the contour line relative to the direction of movement of the moving object 1 based on the characteristics of the direction (angle) of the contour line segments represented in the histogram, such as the shape of the distribution of contour line segments represented in the histogram or the proportion of the directions (angles) of the contour line segments.

[0053] Subsequently, the contour direction detection unit 124 excludes contour points representing contour lines in directions clearly different from the detected direction of the contour line relative to the direction of travel from subsequent processing. In other words, the contour direction detection unit 124 only includes contour points representing contour lines whose direction relative to the direction of travel in subsequent processing. The contour direction detection unit 124 generates a projection image (hereinafter referred to as the "target contour projection image") in which contour points to be excluded from subsequent processing have been removed from the contour lines included in the projection image, in other words, the projection image contains only the contour points to be included in subsequent processing. At this time, the contour direction detection unit 124 generates the target contour projection image by, for example, changing the pixel value (or label value) of the pixels of contour points to be excluded from subsequent processing to a value that indicates that it is not the indicated contour line (for example, pixel value = "0"). The contour direction detection unit 124 outputs the generated target contour projection image to the contour clustering unit 125.

[0054] The contour line direction detection unit 124 is an example of a "contour line direction detection unit".

[0055] The contour clustering unit 125 groups (clusters) the contour points representing each contour line to be processed, which are shown in the target contour projection image output by the contour direction detection unit 124. More specifically, the contour clustering unit 125 calculates the distance from the contour line of the target contour projection image to the origin of the moving object coordinate system (for example, the center position of the moving object 1 as described above). Then, the contour clustering unit 125 groups contour points that have a high similarity in the calculated distance (i.e., contour points that are close in the calculated distance (located within a predetermined distance range)) as clusters of contour points representing the same indicated contour line.

[0056] Furthermore, the contour clustering unit 125 determines the direction in which adjacent contour points are connected based on a value representing the direction of the contour line (contour segment) indicated by the contour point, and further groups (clusters) contour points representing the same indicator contour line that have been grouped together based on their positional relationship with the moving body 1. More specifically, the contour clustering unit 125 classifies the contour points into groups located closer to the moving body 1 (front side) and groups located further away from the moving body 1 (back side), based on a value representing the direction of the contour line (contour segment) indicated by the contour point. In other words, the contour clustering unit 125 determines whether the contour line (contour segment) is located on the front side or the back side in the direction of movement of the moving body 1, based on a value representing the direction of the contour line (contour segment) indicated by the contour point, and groups them accordingly.

[0057] The contour clustering unit 125 generates a target contour projection image (hereinafter referred to as the "contour group projection image") in which information representing the clustered group is further added to each contour point included in the target contour projection image, and outputs it to the linear fitting unit 126.

[0058] The contour clustering unit 125 is an example of a "contour classification unit".

[0059] The linear fitting unit 126 straightens (fits to a straight line) the contour lines of the indicator marks indicated by the contour points of each group in the contour group projection image output by the contour clustering unit 125. More specifically, the linear fitting unit 126 estimates the parameters of a linear model (straight line model) for the contour lines of the indicator marks indicated by the contour points (that is, the contour lines of the indicator marks, which are originally straight lines but are not straight lines due to the influence of noise etc. in the input image, represented by sequentially connecting the contour line segments between two adjacent contour points). Then, the linear fitting unit 126 projects each contour point onto the estimated straight line model, in other words, by projecting the contour points, it aligns the contour lines of the indicator marks indicated by the contour points to a straight line.

[0060] Furthermore, the linear fitting unit 126 extracts contour points located at both ends of the contour line of the indicator marking, which has become a straight line, based on the position of the projected (aligned to a straight line) contour points, as the endpoints of the contour line. At this time, the linear fitting unit 126 extracts contour points located at both ends for each contour line represented by the contour points of each group. The linear fitting unit 126 outputs the information representing the contour points at both ends of the contour line of each extracted group to the driving control unit 140 as indicator marking information representing the endpoints of the recognized indicator marking area.

[0061] The linear fitting section 126 is an example of a "straightening section".

[0062] With this functional configuration, the road marking recognition unit 120 recognizes the directional markings drawn on the road surface, such as a sidewalk or roadway, on which the mobile vehicle 1 is moving, by analyzing the image (input image) captured by the external camera 11. The road marking recognition unit 120 then outputs directional marking information representing the endpoints of the recognized directional marking area to the driving control unit 140. As a result, the driving control unit 140 can generate a target trajectory for when the mobile vehicle 1 automatically travels and control the drive unit 40 according to the driving mode, based on the information of the directional marking area indicated in the directional marking information output by the road marking recognition unit 120.

[0063] [Processing Flow of Road Marking Recognition Unit] Figure 4 is a flowchart showing an example of the processing flow for recognizing instruction markings performed by the road marking recognition unit 120. Figures 5 to 13 schematically show examples of processing by each component of the road marking recognition unit 120. In the following description, the processing of this flowchart shown in Figure 4 will be explained with reference to the examples of processing shown in Figures 5 to 13 as appropriate. The processing of this flowchart is repeatedly executed at predetermined time intervals when the external camera 11 images the area in front of the moving body 1. The processing of this flowchart may also be repeatedly executed, for example, at predetermined periodic time intervals when the road marking recognition unit 120 performs the processing to recognize instruction markings.

[0064] First, when it is time for the road marking recognition unit 120 to perform the process of recognizing the instruction marking, the image acquisition unit 121 acquires an image taken by the external camera 11 of the area in front of the moving object 1 as an input image. The image acquisition unit 121 then performs image analysis processing on the acquired input image to classify each pixel that makes up the frame of the input image into a class and generates an analyzed image with a label representing the corresponding class (step S100). The image acquisition unit 121 outputs the generated analyzed image to the contour extraction unit 122.

[0065] The contour extraction unit 122 extracts the contour lines of the indicator markings (more specifically, contour lines from the indicator marking pixels to which the indicator marking labels have been assigned) from the analysis image output by the image acquisition unit 121. The contour extraction unit 122 then generates a contour line image in which information indicating that each of the extracted contour lines is an indicator marking contour line is added (step S110). In the following description, it is assumed that the contour line image generated by the contour extraction unit 122 has pixel values ​​that indicate that the contour line is an indicator marking contour line, and also values ​​that indicate the direction of the contour line (contour line segment) in which the indicator marking pixels were detected in a counterclockwise direction.

[0066] Figure 5 is a schematic diagram illustrating an example of processing performed by the image acquisition unit 121 and the contour extraction unit 122 of the road marking recognition unit 120. Figure 5(a) shows an example of input image Im-I captured by the external camera 11 in front of the moving body 1. The input image Im-I shown in Figure 5(a) is an example of an input image captured when the moving body 1 is traveling on a sidewalk. Figure 5(b) shows an example of an analyzed image IM1 generated by the image acquisition unit 121 after performing image analysis processing. The analyzed image IM1 shown in Figure 5(b) is an example of an analyzed image in which the pixel value of each pixel is used as a label value representing the assigned label. The analyzed image IM1 shown in Figure 5(b) shows an example in which the guide stop line M1 on the side closer to the moving body 1 (foreground side) and the stop line M2 on the side further away from the moving body 1 (background side), both drawn on the road surface of the sidewalk, are classified into different classes of guide markings. Figure 5(c) shows an example of a contour line image IM2 generated by the contour extraction unit 122. The contour line image IM2 shown in Figure 5(c) is an example of a contour line image in which the contour lines of the instruction markings for the instruction stop line M1 and the stop line M2 have been extracted. The contour line image IM2 shown in Figure 5(c) also shows an example in which instruction marking pixels located inside the contour line of the instruction stop line M1, that is, located outside the edges (outer perimeter) of the area of ​​the instruction stop line M1.

[0067] The contour extraction unit 122 outputs the generated contour line image (see Figure 5(c)) to the homography conversion unit 123.

[0068] Returning to Figure 4, the homography conversion unit 123 performs a projection transformation (homography transformation) on the contour line image output by the contour extraction unit 122. The homography conversion unit 123 then generates a projected image (step S120).

[0069] Figure 6 schematically shows an example of processing by the homography transformation unit 123 of the road marking recognition unit 120. Figure 6(a) shows the same contour line image IM2 as in Figure 5(c), generated and output by the contour extraction unit 122. Figure 6(b) shows an example of a projected image IM3 obtained by homography transformation of the contour line image IM2. The projected image IM3 shown in Figure 6(b) is an example of an isomorphism (projected image) obtained by projecting the positions of each contour line of the indicator marking represented by the contour line image IM2 onto a two-dimensional projected space coordinate system (mobile body coordinate system) into a two-dimensional mobile body coordinate system where the center position C of the mobile body 1 is the origin, the direction of travel of the mobile body 1 (direction of arrow α1 shown in Figure 2) is the X axis, and the width direction of the mobile body 1 (width direction of the sidewalk) is the Y axis. In the projection image IM3 shown in Figure 6(b), although they are drawn on the same sidewalk surface, they are drawn at different positions in the depth direction ahead of the direction of travel of the moving object 1. Therefore, in the contour image IM2, the guide stop line M1 and the stop line M2 appear to have different lengths, but in the two-dimensional moving object coordinate system, they are represented as having the same length in the X-axis and Y-axis directions.

[0070] The homography conversion unit 123 outputs the generated projection image (see Figure 6(b)) to the contour direction detection unit 124.

[0071] Returning to Figure 4, the contour direction detection unit 124 calculates the angle of the contour segment based on the direction of the contour segment between two adjacent contour points included in the projection image output by the homography conversion unit 123, and creates a histogram of the calculated angles of each contour segment (step S130). Then, the contour direction detection unit 124 detects the direction of the contour based on the created histogram and generates a target contour projection image by deleting contour points in directions other than the detected contour direction (step S132).

[0072] Figures 7 and 8 schematically illustrate an example of processing by the contour line direction detection unit 124 of the road marking recognition unit 120. Figure 7(a) shows the projection image IM3a, which is the same projection image IM3 as in Figure 6(b), generated and output by the homography conversion unit 123, but only includes the range containing the guide stop line M1 and stop line M2 (range of the moving body coordinate system). Figure 7(b) shows an example of a histogram generated based on each contour point included in the projection image IM3a shown in Figure 7(a). The histogram shown in Figure 7(b) shows that when the direction of travel of the moving body 1 (X-axis direction) is taken as the reference direction (0°), there are contour line segments with contour line directions (angles) around 90° and around 180°. In the histogram shown in Figure 7(b), the number of contour segments with an angle of approximately 90° represents the number of contour segments between contour points extending in the Y-axis direction in Figure 7(a), and the number of contour segments with an angle of approximately 180° represents the number of contour segments between contour points extending in the X-axis direction in Figure 7(a). The contour direction detection unit 124 detects the angle of the contour segment with the largest number of contour segments N1 (for example, 85° in Figure 7(b)) from the histogram shown in Figure 7(b) as the contour direction θ with respect to the direction of travel of the moving object 1. An example of the contour direction θ detected by the contour direction detection unit 124 is also shown in the projected image IM3a shown in Figure 7(a).

[0073] Figure 8(a) shows the same histogram as Figure 7(b) generated by the contour line direction detection unit 124. The contour line direction detection unit 124 detects the angle of the contour line segment with the largest number of contour line segments N1 as the direction of the contour line θ with respect to the direction of travel of the moving body 1. Each contour point where a contour line segment with an angle of approximately 90° is obtained is subject to subsequent processing, and each contour point where a contour line segment with an angle of approximately 180° is excluded from subsequent processing. The histogram shown in Figure 8(a) schematically shows that each contour point where a contour line segment within region E1 is obtained is excluded from subsequent processing. Figure 8(b) shows an example of the target contour line projection image IM4 generated by the contour line direction detection unit 124. However, Figure 8(b) shows the target contour line projection image IM4a, which includes only the range (range of the moving body coordinate system) that contains the guidance stop line M1 and the stop line M2 in the target contour line projection image IM4. In the target contour projection image IM4a shown in Figure 8(b), only the contour lines extending in the vehicle width direction (Y-axis direction) of the moving body 1, where the contour line angle was around 90° in Figure 7(b) and Figure 8(b), are shown. The target contour projection image IM4a shown in Figure 8(b) also shows an example of the contour line direction θ detected by the contour line direction detection unit 124.

[0074] The contour line direction detection unit 124 outputs the generated target contour line projection image (see Figure 8(b)) to the contour line clustering unit 125.

[0075] Returning to Figure 4, the contour clustering unit 125 calculates the distance between the position of each contour point representing a contour line in the target contour projection image output by the contour direction detection unit 124 and the origin of the moving object coordinate system (step S140). Then, the contour clustering unit 125 clusters each contour point based on the calculated distance from the origin of the moving object coordinate system (step S142). Furthermore, the contour clustering unit 125 clusters each contour point based on the positional relationship between the position of the moving object 1 and the contour point (step S144). The contour clustering unit 125 generates a contour group projection image in which information representing the clustered group is added to each contour point included in the target contour projection image (step S146).

[0076] Figures 9 to 11 schematically illustrate an example of processing performed by the contour clustering unit 125 of the road marking recognition unit 120. Figure 9(a) shows the same target contour projection image IM4a as in Figure 8(b), generated and output by the contour direction detection unit 124. Figure 9(b) shows an example in which contour points, whose distances from the origin of the mobile coordinate system (center position C of the mobile body 1) are calculated, are plotted at corresponding positions on a one-dimensional graph with the distance from the origin of the mobile coordinate system to the contour line as the X-axis. As an example, Figure 9(a) shows the state in which the contour clustering unit 125 has calculated distance d1 as the distance between contour point P1 on the contour line of the stop line M2 and the center position C of the mobile body 1 (origin of the mobile coordinate system), and Figure 9(b) shows the state in which contour point P1 is plotted at the position corresponding to distance d1 on the one-dimensional graph.

[0077] Figure 10(a) shows the contour clustering unit 125 having completed the calculation of the distance between all contour points represented in the target contour projection image IM4a and the center position C of the moving body 1, and plotting each contour point on a one-dimensional graph. Figure 10(a) also shows the contour clustering unit 125 clustering each contour point into group G1 and group G2. Figure 10(b) shows an example of a contour group projection image IM5 representing the clustered contour points. However, Figure 10(b) shows a contour group projection image IM5a that includes only the range (range of the moving body coordinate system) that contains the guide stop line M1 and stop line M2 in the contour group projection image IM5. The contour line group projection image IM5a shown in Figure 10(b) is the contour line group projection image up to the point in time when the guidance stop line M1 is clustered as group G1 and the stop line M2 is clustered as group G2.

[0078] Groups G1 and G2 are examples of "distance groups".

[0079] Figure 11(a) shows the contour clustering unit 125 clustering each contour point based on the positional relationship between the position of the moving body 1 and the contour point. The contour points of the guidance stop line M1 belonging to group G1 are clustered into group G1F, which is closer to the moving body 1 (foreground side), and group G1B, which is further away from the moving body 1 (background side). The contour points of the stop line M2 belonging to group G2 are clustered into group G2F, which is closer to the moving body 1 (foreground side), and group G2B, which is further away from the moving body 1 (background side). In the guidance stop line M1 and stop line M2, the contour points belonging to group G1F and group G2F, which are closer to the moving body 1 (foreground side), are clustered as groups where the direction in which adjacent contour points are connected is from left to right. The contour points belonging to groups G1B and G2B, which are on the side furthest (backward) from the moving object 1 at the guidance stop line M1 and stop line M2, are clustered as groups where the direction in which adjacent contour points are connected is from right to left. Figure 11(b) shows an example of a contour line group projection image IM5 representing each clustered contour point. However, in Figure 11(b), the contour line group projection image IM5b is shown, which only includes the range that contains the guidance stop line M1 and stop line M2 (range of the moving object coordinate system). The contour line group projection image IM5b shown in Figure 11(b) is a contour line group projection image in which each contour point of the guide stop line M1, which is clustered as group G1, is further clustered (grouped) into group G1F and group G1B, respectively, and each contour point of the stop line M2, which is clustered as group G2, is further clustered (grouped) into group G2F and group G2B, respectively.

[0080] Group G1F, Group G1B, Group G2F, and Group G2B are all examples of "direction groups".

[0081] The contour clustering unit 125 outputs the generated contour group projection image (see Figure 11(b)) to the linear fitting unit 126.

[0082] Returning to Figure 4, the linear fitting unit 126 fits the contour lines of the indicator markings indicated by the contour points of each group represented in the contour group projection image output by the contour clustering unit 125 to a straight line (step S150). Then, the linear fitting unit 126 extracts the endpoints of both ends of the contour lines of the indicator markings based on the positions of the contour points of each group that have been fitted to a straight line (step S152).

[0083] Figures 12 and 13 schematically illustrate an example of processing by the linear fitting unit 126 of the road marking recognition unit 120. Figure 12(a) shows an example of a linear model estimated based on the position of each contour point included in the contour line group projection image. More specifically, Figure 12(a) shows an example of a linear model corresponding to the guidance stop line M1. Linear model L1F shown in Figure 12(a) is an example of a linear model corresponding to each contour point clustered in group G1F on the side closer to the moving object 1 (foreground side) of the guidance stop line M1, and linear model L1B is an example of a linear model corresponding to each contour point clustered in group G1B on the side farther from the moving object 1 (back side) of the guidance stop line M1. Figure 12(b) shows an example of a state in which the contour lines of the guidance markings indicated by each contour point are aligned to a straight line (fitted) by projecting each contour point onto the estimated corresponding linear model. In Figure 12(b), as an example, the contour point P2, which was offset from the linear model L1B in Figure 12(a), is fitted to the linear model L1B and becomes the contour point P2f.

[0084] Figure 13(a) shows the positional relationship between the linear model L1, which has been linearly fitted by the linear fitting unit 126 and the respective contour points, in the same state as in Figure 12(b). Figure 13(b) shows an example in which the linear fitting unit 126 extracts the endpoints at both ends of the contour lines of the instruction markings in the state of the positional relationship between the linear model L1 and the respective contour points after linear fitting. In Figure 13(b), the left endpoint M1-1 and the right endpoint M1-2 are extracted from the contour points fitted to the linear model L1F, and the right endpoint M1-3 and the left endpoint M1-4 are extracted from the contour points fitted to the linear model L1B. The four endpoints extracted in Figure 13(b) represent the positions of the instruction stop line M1 on the side closer to the moving object 1 (foreground side) drawn on the road surface of the sidewalk in the input image Im-I.

[0085] Returning to Figure 4, the linear fitting unit 126 outputs instruction marking information (see Figure 13(b)) to the driving control unit 140, which includes information representing the positions of the endpoints of the contour lines extracted from each group (step S154).

[0086] Then, the road marking recognition unit 120 returns the process to step S100. When it is time for the next step to perform the process of recognizing the instruction marking, the road marking recognition unit 120 repeats each of the processes from step S100 to step S154.

[0087] Through this process, the road marking recognition unit 120 analyzes the image (input image) captured by the external camera 11 to recognize the instruction markings drawn on the road surface, such as the sidewalk or roadway, on which the mobile vehicle 1 is moving. The road marking recognition unit 120 then outputs instruction marking information representing the endpoints of the recognized instruction marking area to the driving control unit 140. As a result, the driving control unit 140 can generate a target trajectory for when the mobile vehicle 1 automatically travels and control the drive unit 40 according to the driving mode, based on the information of the instruction marking area indicated in the instruction marking information output by the road marking recognition unit 120.

[0088] As described above, according to the marking recognition device of the embodiment, the road marking recognition unit 120 of the control device 100 recognizes the instruction markings drawn on the road surface in front of the mobile body 1 by acquiring and analyzing an image (input image) captured by an external camera 11 that captures images of at least the area in front of the mobile body 1. The marking recognition device of the embodiment then outputs instruction marking information representing the endpoint of the recognized instruction marking area to the driving control unit 140 of the control device 100. As a result, in the mobile body 1 equipped with the marking recognition device of the embodiment, the driving control unit 140 of the control device 100 can generate a target trajectory when the mobile body 1 is automatically traveling and control the drive unit 40 according to the driving mode based on the information of the instruction marking area indicated in the instruction marking information output by the road marking recognition unit 120.

[0089] The embodiment described above can be expressed as follows: A marking recognition device comprising: a storage medium for storing computer-readable instructions for recognizing a predetermined marking drawn to extend in a certain direction on the movement path of a mobile body that can move both on a roadway and in a predetermined area different from the roadway; and a processor connected to the storage medium, wherein the processor executing the computer-readable instructions to: extract each contour point located on the contour portion of the area of ​​the marking as captured in a two-dimensional image captured by an imaging device that captures at least the movement path; project the positions of each contour point extracted in the image onto a two-dimensional projection space; detect the orientation in which the marking is drawn based on the direction of the contour line segment between two adjacent contour points; and output endpoint information representing the endpoint of the area in which the marking is drawn.

[0090] In the embodiment, the case described was one in which the road markings drawn on the road surface of the sidewalk on which the mobile body 1 is traveling are not obscured by, for example, other mobile bodies or pedestrians, that is, the road marking recognition unit 120 can recognize the entire road marking. However, when the mobile body 1 is actually traveling, it is conceivable that part of the road marking may be obscured in the image captured by the external camera 11 (input image) (the entire road marking may not be captured) by other mobile bodies traveling in front of the mobile body 1, or by pedestrians or cyclists walking or crossing in front of the mobile body 1. Even in this case, for example, if the left and right edges of the road marking are captured in the input image, the road marking recognition unit 120 can recognize the entire road marking. This is because the road marking recognition unit 120 can output road marking information that represents the entire road marking by the left and right endpoints represented by each clustered (grouped) group. In this case, the processing of the road marking recognition unit 120 should be equivalent to the processing of the road marking recognition unit 120 described in the embodiment.

[0091] In the embodiment, the case described was the recognition of an instruction mark drawn on the road surface ahead of the sidewalk on which the mobile body 1 is traveling. However, the sidewalk or roadway on which the mobile body 1 is traveling may intersect with other sidewalks or other roadways, and instruction markings may also be drawn on the road surfaces of those other intersecting sidewalks or other roadways. Even in this case, if the entire or a part of the instruction marking is captured in the image (input image) taken by the external camera 11, the road marking recognition unit 120 can recognize that instruction marking. The processing of the road marking recognition unit 120 in this case should be equivalent to the processing of the road marking recognition unit 120 described in the embodiment.

[0092] In the embodiment, the case where the instruction marking recognized by the road marking recognition unit 120 is a stop line M1 or a stop line M2, that is, a stop line, was described. However, instruction markings drawn on the road surface such as sidewalks and roadways on which the moving body 1 travels may include things other than stop lines, such as pedestrian crossings. Even in this case, the road marking recognition unit 120 can recognize the instruction marking in the same way by extracting a number of endpoints corresponding to the shape of the instruction marking, based on the shape of the instruction marking and the orientation of the instruction marking relative to the direction of travel of the moving body 1. The processing of the road marking recognition unit 120 in this case should also be equivalent to the processing of the road marking recognition unit 120 described in the embodiment.

[0093] Although embodiments for carrying out the present invention have been described above using examples, the present invention is not limited in any way to these embodiments, and various modifications and substitutions can be made without departing from the spirit of the present invention.

[0094] 1... Mobile body 10... External detection device 12... Mobile body sensor 14... Operator 16... Internal camera 18... Positioning device 22... Mode switching switch 26... HMI 30... Movable mechanism 40... Drive unit 50... External notification device 70... Storage device 100... Control unit 11... External camera 72... Map information 74... Program 120... Road marking recognition unit 121... Image acquisition unit 122... Contour extraction unit 123... Homography conversion unit 124... Contour line direction detection unit 125... Contour line clustering unit 126... Linear fitting unit 140... Driving control unit

Claims

1. A marking recognition device for recognizing a predetermined marking drawn so as to extend in a certain direction on the movement path of a mobile body capable of moving both on a roadway and in a predetermined area separate from the roadway, the device extracts each contour point located on the contour portion of the area of ​​the marking as captured in a two-dimensional image taken by an imaging device that captures at least the movement path, projects the positions of each contour point extracted in the image into a two-dimensional projection space, detects the orientation in which the marking is drawn based on the direction of the contour line segment between two adjacent contour points, and outputs endpoint information representing the endpoint of the area in which the marking is drawn.

2. A marking recognition device according to claim 1, comprising: an image acquisition unit that acquires the image and classifies the pixels constituting the acquired image into predetermined classes; a contour extraction unit that extracts each of the contour points located on the contour line of the marking from among the pixels classified into the classes; a projection transformation unit that projects each of the extracted contour points onto the projection space; a contour line orientation detection unit that detects the orientation of the contour line with respect to the direction of travel of the moving body based on the direction of the contour line segment; a contour line classification unit that groups together contour points representing the same contour line based on the distance between the contour line and the origin of the projection space; and a straightening unit that straightens the contour line represented by each of the contour points belonging to the same group and outputs endpoint information based on the straightened contour line.

3. The marking recognition device according to claim 2, wherein the contour extraction unit extracts marking pixels that represent the marking from among the pixels classified into the class, and detects the contour points in a predetermined direction.

4. The marking recognition device according to claim 2, wherein the contour line direction detection unit displays the direction of each contour line segment corresponding to each contour point in a histogram, and detects the direction of the contour line based on the distribution of the directions of the contour line segments displayed in the histogram.

5. The marking recognition device according to claim 4, wherein the contour line direction detection unit detects the direction of the contour line segment with the highest distribution in the histogram as the direction of the contour line.

6. The marking recognition device according to claim 2, wherein the contour line classification unit calculates the distance for each of the contour points and groups contour points whose calculated distances are within a predetermined distance range into the same distance group.

7. The marking recognition device according to claim 6, wherein the contour line classification unit groups contour points in the same direction into the same direction group based on the direction of each contour line segment corresponding to each contour point belonging to the same distance group.

8. The marking recognition device according to claim 2, wherein the straightening unit estimates a straight line represented by each of the contour points grouped in the same group, projects each of the contour points of the estimated straight line onto the straight line, thereby straightening the contour line represented by each of the contour points grouped in the same group, and outputs information representing the contour points located at both ends of the straightened contour line as the endpoint information.

9. A method for recognizing a marking, comprising: a computer that recognizes a predetermined marking drawn to extend in a certain direction on the movement path of a mobile body capable of moving both on a roadway and in a predetermined area separate from the roadway; extracting each contour point located on the contour of the area of ​​the marking as captured in a two-dimensional image taken by an imaging device that captures at least the movement path; projecting the positions of each contour point extracted in the image into a two-dimensional projection space; detecting the orientation in which the marking is drawn based on the direction of the contour line segment between two adjacent contour points; and outputting endpoint information representing the endpoints of the area in which the marking is drawn.

10. A storage medium storing a program that causes a computer to recognize a predetermined marking drawn so as to extend in a certain direction on the movement path of a mobile body capable of moving both on a roadway and in a predetermined area separate from the roadway, to extract each contour point located on the contour of the area of ​​the marking as captured in a two-dimensional image taken by an imaging device that captures at least the movement path, to project the positions of each contour point extracted in the image into a two-dimensional projection space, to detect the orientation in which the marking is drawn based on the direction of the contour line segment between two adjacent contour points, and to output endpoint information representing the endpoints of the area in which the marking is drawn.

Images