Information processing device, information processing method, and program

Abstract

Provided is an information processing device (10) comprising: a recognition unit (11) that recognizes an area of a lane line of a road from a first image in which the road around a vehicle is captured; a generation unit (12) that generates a second image in which the recognized area of the lane line of the road is represented on a coordinate plane having the longitudinal direction and horizontal direction of the vehicle as two axes; and an estimation unit (13) that estimates a graphic of a line representing the lane line of the road on the basis of the second image.

Description

Information processing device, information processing method, and computer-readable medium

[0001] The present disclosure relates to an information processing device, an information processing method, and a non-transitory computer-readable medium on which a program is stored.

[0002] Patent Literature 1 discloses a technology for recognizing, when there are multiple lane candidate line candidates whose brightness is determined to be equal to or greater than a brightness threshold, the lane candidate closest to the vehicle as a lane marking line. This technology addresses the problem that when multiple line candidates with different brightnesses, such as white and yellow lines, are detected, only the white line may be recognized.

[0003] Japanese Patent Application Laid-Open No. 2019-20957

[0004] However, with the technology described in Patent Document 1, it may not be possible to properly detect the lane markings depending on their shape, for example, when the lane markings indicating one side of the lane in which the vehicle is traveling (e.g., center line, lane boundary line) are dashed lines.

[0005] In view of the above-mentioned problems, an object of the present disclosure is to provide an information processing device, an information processing method, and a non-transitory computer-readable medium storing a program that can appropriately detect lane markings.

[0006] In a first aspect of the present disclosure, an information processing device is provided that includes a recognition unit that recognizes the area of ​​road dividing lines from a first image of a road around a vehicle, a generation unit that generates a second image in which the recognized area of ​​road dividing lines is represented on a coordinate plane having the longitudinal and lateral directions of the vehicle as two axes, and an estimation unit that estimates linear figures representing the road dividing lines based on the second image.

[0007] In addition, a second aspect of the present disclosure provides an information processing method that recognizes an area of ​​road dividing lines from a first image of a road around a vehicle, generates a second image that represents the recognized area of ​​road dividing lines on a coordinate plane with the longitudinal and lateral directions of the vehicle as two axes, and estimates a linear figure representing the road dividing lines based on the second image.

[0008] In addition, a third aspect of the present disclosure provides a non-transitory computer-readable medium storing a program that causes a computer to execute a process of recognizing an area of ​​road dividing lines from a first image of a road around a vehicle, generating a second image that represents the recognized area of ​​road dividing lines on a coordinate plane having the longitudinal and lateral directions of the vehicle as two axes, and estimating linear figures representing the road dividing lines based on the second image.

[0009] According to one aspect, lane markings can be detected appropriately.

[0010] 1 is a diagram illustrating an example of the configuration of an information processing device according to an embodiment; FIG. 2 is a diagram illustrating an example of the hardware configuration of an information processing device according to an embodiment; FIG. 3 is a flowchart illustrating an example of processing of an information processing device according to an embodiment; FIG. 4 is a diagram illustrating an example of a captured image according to an embodiment; FIG. 5 is a diagram illustrating an example of a captured image after threshold processing according to an embodiment; FIG. 6 is a diagram illustrating an example of a binarized image in which dashed line portions of lane lines according to an embodiment are detected; FIG. 7 is a diagram illustrating an example of dashed lane lines on a bird's-eye view according to an embodiment; FIG. 8 is a diagram illustrating an example of an output image according to an embodiment;

[0011] The principles of the present disclosure will be described with reference to some exemplary embodiments. It should be understood that these embodiments are set forth for illustrative purposes only, to aid those skilled in the art in understanding and practicing the present disclosure, without implying any limitation on the scope of the present disclosure. The disclosure described herein can be implemented in various ways other than those described below. In the following description and claims, unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which the present disclosure belongs. Hereinafter, embodiments of the present disclosure will be described with reference to the drawings.

[0012] (First Embodiment) <Configuration> The configuration of an information processing device 10 according to an embodiment will be described with reference to Fig. 1. Fig. 1 is a diagram showing an example of the configuration of the information processing device 10 according to an embodiment. The information processing device 10 has a recognition unit 11, a generation unit 12, and an estimation unit 13. Each of these units may be realized by cooperation between one or more programs installed in the information processing device 10 and hardware such as a processor and memory of the information processing device 10.

[0013] The recognition unit 11 recognizes the area of ​​road dividing lines from a first image capturing the road around the vehicle. The generation unit 12 generates a second image representing the area of ​​road dividing lines recognized by the recognition unit 11 on a coordinate plane having two axes representing the longitudinal direction (e.g., forward direction) and lateral direction of the vehicle. The estimation unit 13 estimates linear figures representing the road dividing lines based on the second image.

[0014] (Embodiment 2) <Hardware Configuration> Fig. 2 is a diagram showing an example of the hardware configuration of an information processing device 10 according to an embodiment. In the example of Fig. 2, the information processing device 10 (computer 100) includes a processor 101, a memory 102, and a communication interface 103. These components may be connected via a bus or the like. The memory 102 stores at least a part of a program 104. The communication interface 103 includes an interface required for communication with other network elements.

[0015] When the program 104 is executed by the processor 101, memory 102, and other components in cooperation with each other, the computer 100 performs at least some of the processing of the embodiments of the present disclosure. The memory 102 may be of any type. As a non-limiting example, the memory 102 may be a non-transitory computer-readable storage medium. The memory 102 may also be implemented using any suitable data storage technology, such as semiconductor-based memory devices, magnetic memory devices and systems, optical memory devices and systems, fixed memory, and removable memory. Although only one memory 102 is shown in the computer 100, several physically different memory modules may be present in the computer 100. The processor 101 may be of any type. The processor 101 may include one or more of a general-purpose computer, a special-purpose computer, a microprocessor, a digital signal processor (DSP), and, as a non-limiting example, a processor based on a multi-core processor architecture. The computer 100 may have multiple processors, such as application-specific integrated circuit chips that are time-slaved to a clock that synchronizes the main processor.

[0016] Embodiments of the present disclosure may be implemented in hardware or special purpose circuits, software, logic, or any combination thereof. Some aspects may be implemented in hardware, while other aspects may be implemented in firmware or software that may be executed by a controller, microprocessor, or other computing device.

[0017] The present disclosure also provides at least one computer program product tangibly stored on a non-transitory computer-readable storage medium. The computer program product includes computer-executable instructions, such as instructions included in program modules, that execute on a target real or virtual processor or device to perform the processes or methods of the present disclosure. Program modules include routines, programs, libraries, objects, classes, components, data structures, etc. that perform particular tasks or implement particular abstract data types. The functionality of the program modules may be combined or divided among program modules as desired in various embodiments. The machine-executable instructions of the program modules may be executed in local or distributed devices. In a distributed device, the program modules may be located in both local and remote storage media.

[0018] The program code for executing the methods of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus. When the program code is executed by the processor or controller, the functions / acts in the flowcharts and / or implementing block diagrams are performed. The program code may be executed entirely on the machine, partly on the machine, as a stand-alone software package, partly on the machine and partly on a remote machine, or entirely on a remote machine or server.

[0019] The program can be stored and supplied to a computer using various types of non-transitory computer-readable media. Non-transitory computer-readable media include various types of tangible recording media. Examples of non-transitory computer-readable media include magnetic recording media, magneto-optical recording media, optical disk media, and semiconductor memory. Magnetic recording media include, for example, flexible disks, magnetic tapes, and hard disk drives. Magneto-optical recording media include, for example, magneto-optical disks. Optical disk media include, for example, Blu-ray discs, CD (Compact Disc)-ROM (Read Only Memory), CD-R (Recordable), and CD-RW (Rewritable). Semiconductor memory includes, for example, solid-state drives, mask ROM, PROM (Programmable ROM), EPROM (Erasable PROM), flash ROM, and RAM (Random Access Memory). The program may also be supplied to a computer by various types of temporary computer-readable media. Examples of temporary computer-readable media include electrical signals, optical signals, and electromagnetic waves. The temporary computer-readable medium can supply the program to the computer via a wired communication path such as an electric wire or an optical fiber, or via a wireless communication path.

[0020] <Processing> Next, an example of processing of the information processing device 10 according to the embodiment will be described with reference to Figs. 3 to 9. Fig. 3 is a flowchart showing an example of processing of the information processing device 10 according to the embodiment. Fig. 4 is a diagram showing an example of a captured image according to the embodiment. Fig. 5 is a diagram showing an example of a captured image after threshold processing according to the embodiment. Fig. 6 is a diagram showing an example of a binarized image in which dashed line portions of a lane marking line according to the embodiment have been detected. Fig. 7 is a diagram showing an example of dashed lane marking lines on a bird's-eye view according to the embodiment. Fig. 8 is a diagram showing an example of lanes on a bird's-eye view according to the embodiment. Fig. 9 is a diagram showing an example of an output image according to the embodiment.

[0021] In step S101, the recognition unit 11 acquires a still image (first image) captured by a camera. Here, the recognition unit 11 may acquire, for example, a frame of a video captured by a monocular camera mounted on a vehicle. In the example of FIG. 4 , the recognition unit 11 acquires a captured image 401 of a road ahead (the direction of travel when traveling straight) of a vehicle (hereinafter also referred to as "host vehicle") on which a camera is mounted.

[0022] Next, the recognition unit 11 recognizes dashed line regions that are dividing lines (e.g., center lines, lane boundaries) that indicate one side of the road lanes within the first image of the road ahead of the vehicle (step S102). Here, the recognition unit 11 may, for example, perform threshold processing based on the brightness of each pixel. In this case, the recognition unit 11 may, for example, convert pixel values ​​within a specific range of brightness into a specific color. This makes it possible, for example, to remove unnecessary background information and emphasize contours. Then, the recognition unit 11 may, for example, recognize dashed line regions that are dividing lines based on the contours of the regions in the image.

[0023] In the example of Fig. 5, threshold processing is performed on the captured image 401 in Fig. 4 to generate an image 501 in which the contours of the dashed line regions that are demarcation lines are emphasized. In the example of Fig. 6, a binarized image 601 in which the dashed line regions that are demarcation lines are distinguished from other regions is generated based on the image 501 in Fig. 5.

[0024] Next, based on the recognition result by the recognition unit 11, the generation unit 12 generates a bird's-eye view (bird's-eye view; "second image") in which the recognized dashed lines are mapped using two axes, the forward direction and the horizontal direction of the vehicle (step S103). Here, the generation unit 12 generates a bird's-eye view in which the values ​​of the two axes correspond to the distance from the vehicle in each direction of the two axes in real space. The bird's-eye view may be, for example, a view looking down vertically or a view looking down diagonally from above the vehicle.

[0025] Here, the generation unit 12 may generate the bird's-eye view by, for example, converting the coordinates of pixels in the captured image within the dashed lane marking area into coordinates in the bird's-eye view. In this case, for example, when the operator installs the camera on the vehicle, the camera may be mounted so that the optical axis direction of the camera (the center of the image) and the forward direction of the vehicle are aligned. In addition, information on the shooting conditions, such as the camera's angle of view, may be registered in the information processing device by the operator. Then, the generation unit 12 may determine a mathematical formula for converting the coordinates of each pixel in the captured image into the forward position and horizontal position of the vehicle in real space, based on the information on the shooting conditions, such as the camera's angle of view.

[0026] Furthermore, when the operator installs the device on the host vehicle, the forward position and horizontal position of the host vehicle in real space may be registered for each of the coordinates of multiple points in the captured image. In this case, the multiple points may be, for example, the four corner points of a trapezoid whose upper base is shorter than its lower base, where the far side in the forward direction of the vehicle is the upper base and the near side is the lower base. The generation unit 12 may then determine a mathematical formula for converting the coordinates of each pixel in the captured image into the forward position and horizontal position of the host vehicle in real space, based on information that associates the coordinates of each point in the captured image with the forward position and horizontal position of the host vehicle in real space.

[0027] For example, when detecting lane marking segments using coordinates in a captured image such as that shown in Figure 6, lane marking segments relatively far from the vehicle (e.g., line segment 611) are depicted as shapes that approximate points. Therefore, when detecting line segments using coordinates in a captured image, for example, by a Hough transform, problems may arise with the detection accuracy of lane marking segments relatively far from the vehicle. On the other hand, according to the present disclosure, the coordinates in the captured image are converted to coordinates in a bird's-eye view and then line segments are detected using a Hough transform, thereby improving the detection accuracy of lane marking segments relatively far from the vehicle, for example.

[0028] Next, the estimation unit 13 divides the bird's-eye view into a plurality of areas according to the distance in the forward direction of the vehicle (step S104). Here, the estimation unit 13 may divide the bird's-eye view into, for example, a first area including an area that is less than a specific distance in the forward direction of the vehicle, and a second area including an area that is equal to or greater than the specific distance in the forward direction of the vehicle.

[0029] In the example of FIG. 7, the estimation unit 13 divides the bird's-eye view 701 into areas 711, 712, 713, 714, and 715 in order of proximity to the vehicle, at specific distances (for example, 10 m) in accordance with the distance ahead of the vehicle.

[0030] Next, the generation unit 12 detects line segments of the lane markings for each area of ​​the bird's-eye view (step S105). Here, the generation unit 12 may detect line segments of the lane markings for each area by, for example, performing a Hough transform for each area. Note that the generation unit 12 is not limited to using a Hough transform, and may detect line segments using other known methods.

[0031] Next, the estimation unit 13 estimates a linear figure representing a road lane marking based on the line segments detected in each area (step S106). Here, the estimation unit 13 may estimate, for example, a line passing through a position on a first line segment detected in the first area and a position on a second line segment detected in the second area as one side of a lane on the road. This improves the detection accuracy of a linear figure representing a curved lane marking, for example, even when the road ahead of the host vehicle is curved.

[0032] In this case, the estimation unit 13 may detect, for example, a straight line passing through a specific point on the line segment detected in each area as one side of a lane on the road. The specific point may be, for example, a point on the line segment at a position closest to the midpoint in the forward direction of the host vehicle in each area. In this case, in the example of FIG. 7 , straight lines passing through points 721, 722, 723, 724, and 725 on the line segment at positions closest to the midpoint in the forward direction of the host vehicle (the vertical direction in FIG. 7 ) in each of areas 711, 712, 713, 714, and 715 are detected as one side of a lane on the road. As a result, for example, a left edge line 811 and a right edge line 812 of the lane on which the host vehicle is traveling are detected, as shown in the bird's-eye view 801 of FIG. 8 .

[0033] The estimation unit 13 may estimate a linear figure representing a road lane marking based on a line segment among the detected line segments that satisfies a predetermined angle condition. In this case, the estimation unit 13 may determine, among the multiple line segments detected in the second area, a line segment that satisfies a predetermined angle condition from the direction in which the first line segment detected in the first area extends toward the second area as a second line segment that forms one side of the same lane as the first line segment. This may improve the detection accuracy of curved lane markings, for example, even when the road ahead of the vehicle is curved. In this case, the estimation unit 13 may determine, among the multiple line segments detected in the second area, a line segment that exists within a specific angle (e.g., 15°) from the end of the first line segment that is closer to the second area and is centered on the direction in which the first line segment extends toward the second area as the second line segment. Furthermore, for example, if another line segment that satisfies a predetermined angle condition has been detected relative to a specific line segment, the estimation unit 13 may determine the reliability of the estimation of the specific line segment to be higher.

[0034] Furthermore, if the width of the detected lane of the road is less than a threshold, the estimation unit 13 may detect (correct or modify) a line of a specific width from one side of the lane based on the line segment on both sides of the lane that has a higher reliability in line segment detection by Hough transform as the line on the other side of the lane. This may improve the accuracy of detecting lanes relatively far from the vehicle, for example. In this case, for example, in the Hough transform, the estimation unit 13 may first project (transform) the coordinates (x, y) of each pixel on the lane segment of the lane marking in the bird's-eye view to a point (ρ, θ) in a two-dimensional polar coordinate space with a distance ρ and an angle θ. Here, the distance ρ may be, for example, the length of a perpendicular line drawn from the origin to a line passing through the coordinates (x, y). Furthermore, the angle θ may be, for example, the angle formed by a perpendicular line drawn from the origin to the line passing through the coordinates (x, y) and the x-axis. The estimation unit 13 may then detect line segments in the x-y coordinate system on the bird's-eye view based on the projected points (ρ, θ), for example. The estimation unit 13 may then determine that the greater the number of projected points (ρ, θ), the higher the reliability of the line segments in the x-y coordinate system on the bird's-eye view that are detected based on the points (ρ, θ).

[0035] The estimation unit 13 may also calculate the reliability of the estimation of a linear figure representing an estimated road lane marking and identify two linear figures representing lane markings forming both ends of the road lane based on the calculated reliability. For example, when a camera mounted on a vehicle continuously captures images, the estimation unit 13 may determine a higher reliability for the linear figure estimated in the current frame the closer the distance between the linear figure representing the lane marking estimated in a previous frame and the linear figure estimated in the current frame. The estimation unit 13 may also calculate the width between the linear figures representing the estimated road lane marks and calculate the reliability based on a comparison between the calculated width and a predetermined reference value representing the width (width) of the road lane. In this case, for example, when another line segment parallel to a detected specific line segment is detected at a distance equal to the predetermined reference value, the estimation unit 13 may determine a higher reliability for the specific line segment. The estimation unit 13 may estimate a dividing line forming the left edge of a lane on a road based on a linear figure with the highest reliability among linear figures within a first lateral distance in each area (for example, within 3 m lateral distance to the left from the center of the vehicle). Similarly, the estimation unit 13 may estimate a dividing line forming the right edge of a lane on a road based on a linear figure with the highest reliability among linear figures within a second lateral distance in each area (for example, within 3 m lateral distance to the right from the center of the vehicle).

[0036] Next, the estimation unit 13 outputs information based on the estimated road lane markings (step S107). Here, the estimation unit 13 may output, for example, information indicating the distance from one end of the lane in which the host vehicle is traveling to the host vehicle. Furthermore, the estimation unit 13 may output, for example, an image in which lines on both sides of the lane in which the host vehicle is traveling are superimposed on the captured image, as shown in FIG. 9. In the example of FIG. 9, the left edge line 811 and the right edge line 812 of the lane in which the host vehicle is traveling in FIG. 8 are converted from coordinates in the bird's-eye view to coordinates in the captured image, and lines 911 and 912 are superimposed on the captured image 401 in FIG. 4, to produce an image 901.

[0037] This can support, for example, the generation of simulation data for the development of an autonomous driving system or a driving assistance system that mimics the swaying of a vehicle actually driven. It can also support, for example, the verification of the operation of a vehicle equipped with an autonomous driving system or a driving assistance system when the vehicle is actually driven. It can also detect the driving lane of the vehicle in real time when the vehicle equipped with an autonomous driving system or a driving assistance system is actually driven.

[0038] <Modifications> The information processing device 10 may be a device contained in a single housing, but the information processing device 10 of the present disclosure is not limited to this. Each unit of the information processing device 10 may be realized by cloud computing configured with one or more computers, for example. Such an information processing device 10 is also included in the examples of the "information processing device" of the present disclosure.

[0039] The present disclosure is not limited to the above-described embodiment, and can be modified as appropriate within the scope of the present disclosure.

[0040] Some or all of the above embodiments can be described as, but are not limited to, the following supplementary notes. (Supplementary Note 1) An information processing device comprising: a recognition unit that recognizes an area of ​​a road's lane markings from a first image of a road around a vehicle; a generation unit that generates a second image in which the recognized area of ​​the road's lane markings is represented on a coordinate plane having two axes defined by the longitudinal and lateral directions of the vehicle; and an estimation unit that estimates a linear figure representing the road's lane markings based on the second image. (Supplementary Note 2) The information processing device described in Supplementary Note 1, wherein the estimation unit detects a plurality of line segments based on the area of ​​the road's lane markings represented in the second image, and estimates a linear figure representing the road's lane markings based on the detected plurality of line segments. (Supplementary Note 3) The information processing device according to Supplementary Note 2, wherein the estimation unit divides the second image into a first area including a range less than a predetermined distance in the longitudinal direction of the vehicle and a second area including a range equal to or greater than the predetermined distance in the longitudinal direction of the vehicle, and estimates a linear figure representing a lane marking of the road based on a first line segment detected in the first area and a second line segment detected in the second area. (Supplementary Note 4) The information processing device according to Supplementary Note 2 or 3, wherein the estimation unit estimates a linear figure representing a lane marking of the road based on a line segment that satisfies a predetermined angle condition among the plurality of detected line segments. (Supplementary Note 5) The information processing device according to Supplementary Note 3, wherein the estimation unit determines a second line segment that forms a linear figure representing the same lane marking as the first line segment among the plurality of line segments detected in the second area, based on a direction in which the first line segment extends toward the second area. (Supplementary Note 6) The information processing device according to Supplementary Note 2 or 3, wherein the estimation unit calculates a reliability of estimation for a linear figure representing the estimated lane marking of the road, and identifies linear figures representing two lane markings forming both ends of the road based on the calculated reliability. (Supplementary Note 7) The information processing device according to Supplementary Note 6, wherein the estimation unit calculates a width between the linear figures representing the estimated lane marking of the road, and calculates the reliability based on a comparison between the calculated width and a predetermined reference value representing the width of the lane of the road.(Supplementary Note 8) An information processing method comprising: recognizing an area of ​​a road's lane markings from a first image of a road around a vehicle; generating a second image in which the recognized area of ​​the road's lane markings is represented on a coordinate plane having two axes made up of the longitudinal and lateral directions of the vehicle; and estimating a linear figure representing the road's lane markings based on the second image. (Supplementary Note 9) A non-transitory computer-readable medium storing a program for causing a computer to execute processes of: recognizing an area of ​​the road's lane markings from a first image of a road around a vehicle; generating a second image in which the recognized area of ​​the road's lane markings is represented on a coordinate plane having two axes made up of the longitudinal and lateral directions of the vehicle; and estimating a linear figure representing the road's lane markings based on the second image.

[0041] REFERENCE SIGNS LIST 1 monitoring system 10 information processing device 11 recognition unit 12 generation unit 13 estimation unit

Claims

1. a recognition unit that recognizes an area of a lane marking on a road from a first image obtained by capturing a road around a vehicle; a generating unit that generates a second image in which the area of the recognized road lane markings is represented on a coordinate plane having two axes that are the longitudinal direction and the lateral direction of the vehicle; an estimation unit that estimates a linear figure representing a lane marking on the road based on the second image; An information processing device comprising:

2. the estimation unit detects a plurality of line segments based on an area of the lane markings of the road represented in the second image, and estimates a linear figure representing the lane markings of the road based on the detected plurality of line segments. The information processing device according to claim 1 .

3. the estimation unit divides the second image into a first area including a range less than a predetermined distance in the longitudinal direction of the vehicle and a second area including a range equal to or greater than the predetermined distance in the longitudinal direction of the vehicle, and estimates a linear figure representing a dividing line of the road based on a first line segment detected in the first area and a second line segment detected in the second area. The information processing device according to claim 2 .

4. The estimation unit estimates a linear figure representing a lane marking of the road based on a line segment that satisfies a predetermined angle condition among the detected line segments.

4. The information processing device according to claim 2 or 3.

5. the estimation unit determines, based on a direction in which the first line segment extends toward the second area, a second line segment that forms a linear figure representing the same lane marking as the first line segment, from among the plurality of line segments detected in the second area; The information processing device according to claim 3 .

6. the estimation unit calculates a reliability of estimation for a linear figure representing the estimated lane marking of the road, and identifies linear figures representing two lane markings forming both ends of the lane marking of the road based on the calculated reliability; 4. The information processing device according to claim 2 or 3.

7. the estimation unit calculates a width between linear figures representing the estimated lane markings of the road, and calculates the reliability based on a comparison between the calculated width and a predetermined reference value representing a width of a lane on the road. The information processing device according to claim 6 .

8. Recognizing an area of a lane marking on a road from a first image obtained by capturing a road around a vehicle; generating a second image in which the recognized lane marking area of the road is represented on a coordinate plane having two axes corresponding to the longitudinal direction and lateral direction of the vehicle; estimating a linear figure representing a dividing line of the road based on the second image; Information processing methods.

9. Recognizing an area of a lane marking on a road from a first image obtained by capturing a road around a vehicle; generating a second image in which the recognized lane marking area of the road is represented on a coordinate plane having two axes corresponding to the longitudinal direction and lateral direction of the vehicle; A program that causes a computer to execute a process of estimating a linear figure representing the road division line based on the second image.