Lane boundary detection device, lane boundary detection method, and computer program product for lane boundary detection

Abstract

An open lane boundary detection device, a lane boundary detection method, and a computer program for lane boundary detection. The lane boundary detection device includes: an identification unit (31) that inputs an image representing the surrounding area of the vehicle (10) obtained by the imaging unit (2) mounted on the vehicle (10) into an identifier learned to identify the category of the object represented in each pixel, and identifies the category of the object represented in each pixel of the image; and a detection unit (32) that, for each pixel column in the direction intersecting the own lane in the image, sequentially determines whether the position corresponding to the pixel group is within the own lane based on the arrangement order of the categories of the objects represented in a continuous predetermined number of pixel groups and the determination result of whether the position corresponding to the pixel group immediately before in the scanning direction is within the own lane along the scanning direction from one end of the pixel column to the other end, thereby detecting the boundary of the own lane.

Description

Technical Field

[0001] The present invention relates to a lane boundary detection device, a lane boundary detection method, and a computer program for lane boundary detection that detect boundaries of lanes represented in an image. Background Art

[0002] For autonomous driving control of a vehicle or for supporting driving of a driver of the vehicle, it is required to correctly detect the positional relationship between the vehicle and the lane in which the vehicle is traveling. For this purpose, a lane in which the vehicle is traveling and a boundary line of the lane are detected from an image around the vehicle obtained by a camera mounted on the vehicle (see Japanese Unexamined Patent Application Publication No. 2019-87134).

[0003] The driving support system disclosed in Japanese Unexamined Patent Application Publication No. 2019-87134 extracts edge points based on the luminance values of pixels in a captured image. Among the edge points, an edge point whose luminance of an inner pixel is smaller than that of an outer pixel is set as an upper edge point, and an edge point whose luminance of an outer pixel is smaller than that of an inner pixel is set as a lower edge point. The driving support system sets, as a lane boundary line, a line candidate obtained based on the positions of the edge points and closest to the vehicle position excluding line candidates that satisfy an exclusion condition. The exclusion condition includes that the number of edge points belonging to an upper edge line constituting the line candidate is equal to or more than a predetermined point threshold compared to the number of edge points belonging to a lower edge line. Summary of the Invention

[0004] Since hardware resources mounted on a vehicle are limited and are used for various processes for autonomous driving control or driving support, the less the computational load required for processing to detect boundaries of a lane (hereinafter sometimes referred to as the own lane) in which the vehicle is traveling from an image obtained by a camera mounted on the vehicle, the better.

[0005] Therefore, an object of the present invention is to provide a lane boundary detection device capable of reducing the computational load required for processing to detect a boundary line of a lane in which a vehicle is traveling from an image.

[0006] According to one embodiment, a lane boundary detection device is provided. The lane boundary detection device includes: an identification unit that, by inputting an image representing a peripheral area of the vehicle obtained by a camera unit mounted on the vehicle into an identifier learned to identify the category of an object represented in each pixel, identifies the category of the object represented in each pixel of the image, where the category of the object at least includes a category representing an object within the lane in which the vehicle is traveling and a category representing an object outside the lane; and a detection unit that, for each pixel column in the direction intersecting the lane in the image, along a scanning direction from one end of the pixel column toward the other end, sequentially determines whether a position corresponding to the pixel group is within the lane based on the arrangement order of the categories of the objects represented in a continuous predetermined number of pixel groups and the determination result of whether a position corresponding to the pixel group immediately preceding in the scanning direction is within the lane, thereby detecting the boundary of the lane.

[0007] In this lane boundary detection device, preferably, when there are multiple sets of consecutive pixels that are within the lane in any one of the pixel columns, the detection unit determines the set of pixels that is most definitely within the lane among the multiple sets of pixels as the lane area representing the lane, and detects the boundary of the lane as the boundary among the two boundaries of the lane area along the scanning direction where there are no pixels representing the lane on the image end side compared to this boundary.

[0008] Alternatively, in this lane boundary detection device, preferably, the category of the object further includes other markings outside the lane demarcation lines provided on the road, the category representing an object within the lane includes the lane itself, and the category representing an object outside the lane includes other road surfaces outside the lane. The detection unit determines that a position corresponding to a pixel group in which the categories of the objects sequentially identified from the side farther from the vehicle are arranged in the order of lane demarcation lines, other markings, and the lane represents the boundary of the lane for a pixel column in the image corresponding to a position that is more than a predetermined distance away from the vehicle in the image. On the other hand, for a pixel column in the image corresponding to a position within a predetermined distance from the vehicle, the detection unit determines that the position of the lane when the categories of the objects sequentially identified from the side farther from the vehicle are arranged in the order of other road surfaces or lane demarcation lines, and then the lane represents the boundary of the lane.

[0009] According to other embodiments, a lane boundary detection method is provided. The lane boundary detection method includes: by inputting an image representing a peripheral area of a vehicle obtained by an imaging unit mounted on the vehicle into an identifier learned in a manner of identifying the category of an object represented in each pixel, thereby identifying the category of an object represented in each pixel of the image, where the category of the object at least includes a category representing an object within the lane during vehicle travel and a category representing an object outside the lane; and for each pixel column in the direction intersecting the lane in the image, along a scanning direction from one end of the pixel column to the other end, successively determining whether a position corresponding to the pixel group is within the lane based on the arrangement order of the categories of the objects represented in a continuous predetermined number of pixel groups and the determination result of whether a position corresponding to the pixel group immediately preceding in the scanning direction is within the lane, thereby detecting the boundary of the lane.

[0010] Furthermore, according to other embodiments, a computer program for lane boundary detection is provided. The computer program for lane boundary detection includes commands for causing a processor to perform the following actions: by inputting an image representing a peripheral area of a vehicle obtained by an imaging unit mounted on the vehicle into an identifier learned in a manner of identifying the category of an object represented in each pixel, thereby identifying the category of an object represented in each pixel of the image, where the category of the object at least includes a category representing an object within the lane during vehicle travel and a category representing an object outside the lane; and for each pixel column in the direction intersecting the lane in the image, along a scanning direction from one end of the pixel column to the other end, successively determining whether a position corresponding to the pixel group is within the lane based on the arrangement order of the categories of the objects represented in a continuous predetermined number of pixel groups and the determination result of whether a position corresponding to the pixel group immediately preceding in the scanning direction is within the lane, thereby detecting the boundary of the lane.

[0011] The lane boundary detection device according to the present invention has an effect of being able to reduce the computational load required for processing to detect the boundary line of the lane during vehicle travel. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 is a schematic structural diagram of a vehicle control system equipped with a lane boundary detection device.

[0013] Figure 2 is a hardware structural diagram of an electronic control device as an embodiment of the lane boundary detection device.

[0014] Figure 3 is a functional block diagram of a processor of an electronic control device related to vehicle control processing including lane boundary detection processing.

[0015] Figure 4 This is a diagram showing an example of an image of the road surface in front of a vehicle and the recognition results of the categories of objects for each pixel in the image.

[0016] Figure 5 This is a state transition diagram showing the order of arrangement of the categories of objects based on each pixel group and indicating how the position corresponding to the pixel of interest during scanning changes.

[0017] Figure 6 This is a diagram showing an example of an image representing lane demarcation lines and other markings.

[0018] Figure 7 This is a diagram showing an example of an image of a road and the recognition results of the categories of objects for each pixel when the lane demarcation lines are unclear.

[0019] Figure 8 This is a flowchart of the operation of a vehicle control process including lane boundary detection processing. DETAILED DESCRIPTION

[0020] Hereinafter, with reference to the accompanying drawings, a lane boundary detection device, a lane boundary detection method, and a lane boundary detection computer program executed in the lane boundary detection device will be described. The lane boundary detection device inputs an image representing the surrounding area of the vehicle obtained by a camera mounted on the vehicle into a recognizer that has learned to recognize the category of the object represented in each pixel, thereby recognizing the category of the object represented in each pixel of the image. Further, for each pixel column in the direction intersecting the extending direction of the own lane in the image (hereinafter sometimes simply referred to as the direction intersecting the own lane), the lane boundary detection device sequentially determines whether the position corresponding to the pixel group is within the own lane based on the order of arrangement of the categories of objects represented in a continuous predetermined number of pixel groups and the determination result as to whether the position corresponding to the immediately preceding pixel group in the scanning direction is within the own lane. Thus, the lane boundary detection device detects the positions of the left and right boundaries of the own lane as observed from the vehicle. In this way, the lane boundary detection device detects the positions of the boundaries of the own lane by one scan for each pixel column, thereby reducing the computational load. In addition, since the lane boundary detection device only needs to scan sequentially in a specific direction (for example, from left to right), by making the arrangement order of the pixels when the image is stored in the memory consistent with its scanning direction, the efficiency of memory access can be improved.

[0021] Hereinafter, an example of applying the lane boundary detection device to a vehicle control system will be described. In this example, the lane boundary detection device detects the boundaries of the own lane by performing lane boundary detection processing on an image obtained by a camera mounted on the vehicle, and uses the detection results for the autonomous driving control of the vehicle.

[0022] Figure 1 It is a schematic structural diagram of a vehicle control system equipped with a lane boundary detection device. Figure 2 It is a hardware structural diagram of an electronic control unit which is an embodiment of a lane boundary detection device. In this embodiment, a vehicle control system 1 mounted on a vehicle 10 and controlling the vehicle 10 includes a camera 2 for photographing the road surface around the vehicle 10 and an electronic control unit (ECU) 3 which is an example of a lane boundary detection device. The camera 2 and the ECU 3 are communicably connected via an in-vehicle network conforming to a standard such as Controller Area Network. In addition, the vehicle control system 1 may further include a storage device (not shown) for storing a map used in the autonomous driving control of the vehicle 10. Furthermore, the vehicle control system 1 may include a distance sensor such as a LiDAR sensor or a radar (not shown), a receiver such as a GPS receiver for positioning the own position of the vehicle 10 in accordance with a satellite positioning system (not shown), a wireless communication terminal for wirelessly communicating with other devices (not shown), and a navigation device (not shown) for searching for a planned route of travel of the vehicle 10, etc.

[0023] The camera 2 is an example of an imaging unit that generates an image representing the peripheral area of the vehicle 10, and includes a two-dimensional detector composed of an array of photoelectric conversion elements sensitive to visible light or infrared light such as a CCD or a C-MOS, and an imaging optical system for imaging an image of an area to be photographed on the two-dimensional detector. The camera 2 is mounted in the vehicle interior of the vehicle 10, for example, in a direction facing the front of the vehicle 10. Moreover, the camera 2 photographs an area including the road surface in front of the vehicle 10 at every predetermined photographing cycle (for example, 1 / 30 second to 1 / 10 second), and generates an image capturing this area. The image obtained by the camera 2 may be a color image or a gray image. In addition, the vehicle control system 1 may include a plurality of cameras 2 with different photographing directions or fields of view.

[0024] Whenever the camera 2 generates an image, it outputs the generated image to the ECU 3 via the in-vehicle network.

[0025] The ECU 3 controls the vehicle 10. In this embodiment, the ECU 3 detects the boundary of the own lane detected from a series of time-series images obtained by the camera 2, and controls the vehicle 10 so that the vehicle 10 travels along the own lane determined by the detected boundary of the own lane. For this purpose, the ECU 3 includes a communication interface 21, a memory 22, and a processor 23.

[0026] The communication interface 21 is an example of a communication unit and has an interface circuit for connecting the ECU 3 to the in-vehicle network. That is, the communication interface 21 is connected to the camera 2 via the in-vehicle network. Moreover, whenever the communication interface 21 receives an image from the camera 2, it sends the received image to the processor 23. In addition, the communication interface 21 sends maps read from a storage device, positioning information from a GPS receiver, etc., received via the in-vehicle network, to the processor 23.

[0027] The memory 22 is an example of a storage unit and has, for example, a volatile semiconductor memory and a non-volatile semiconductor memory. Moreover, the memory 22 stores computer programs for implementing various processes executed by the processor 23 of the ECU 3, various data used in the lane boundary detection process, such as images received from the camera 2, various parameters for determining the recognizer used in the lane boundary detection process, etc. Furthermore, the memory 22 stores the calculation results during the lane boundary detection process.

[0028] The processor 23 is an example of a control unit and has one or more CPUs (Central Processing Unit) and their peripheral circuits. The processor 23 may also have other arithmetic circuits such as a logical arithmetic unit, a numerical arithmetic unit, or a graphics processing unit. Moreover, whenever the processor 23 receives an image from the camera 2 during the travel of the vehicle 10, it executes vehicle control processing including lane boundary detection processing on the received image. Then, the processor 23 controls the vehicle 10 in a manner to perform autonomous driving of the vehicle 10 or support the driving of the driver of the vehicle 10 based on the detected boundary of the own lane.

[0029] Figure 3 It is a functional block diagram of the processor 23 of the ECU 3 related to vehicle control processing including lane boundary detection processing. The processor 23 has an identification unit 31, a detection unit 32, and a vehicle control unit 33. These units of the processor 23 are, for example, functional modules implemented by a computer program operating on the processor 23. Or, these units of the processor 23 may also be dedicated arithmetic circuits provided in the processor 23. In addition, the identification unit 31 and the detection unit 32 among these units of the processor 23 execute lane boundary detection processing.

[0030] Whenever the identification unit 31 obtains an image from the camera 2, it inputs the image to a recognizer that has been pre-learned to identify the category of the object represented for each pixel, thereby identifying the category of the object represented in each pixel of the image.

[0031] In the present embodiment, the category of the object recognized by the recognizer is used to detect the boundary of the own lane represented in the image. Therefore, among the categories of the objects to be recognized, at least the category indicating being within the own lane and the category indicating being outside the own lane are included. In the present embodiment, among the categories of the objects indicating being within the own lane, the own lane itself is included. On the other hand, among the categories of the objects indicating being outside the own lane, stationary objects, the road surface outside the own lane (hereinafter sometimes referred to as other road surfaces), and lane demarcation lines are included. In addition, regarding the lane demarcation lines, it is also possible to individually recognize the identification portion provided on the road surface and the portion between the identifications in the lane demarcation lines represented by dotted lines (hereinafter sometimes referred to as the non-identification portion of the lane demarcation lines). In the following description, unless otherwise specifically stated in advance, the identification portion and the non-identification portion of the lane demarcation lines are collectively referred to as the lane demarcation lines. Furthermore, among the categories of the objects to be recognized by the recognizer, other markings such as moving objects, or guideway markings or deceleration markings, which are markings provided along the lane demarcation lines on the road surface, may also be included.

[0032] As the recognizer, for example, a neural network for semantic segmentation having a convolutional neural network type architecture including a plurality of convolutional layers, such as a fully convolution network (FCN) or U-Net, is used. By using such a neural network as the recognizer, the recognition unit 31 can recognize the category of the object represented in each pixel with relatively high accuracy. In addition, as the recognizer, a recognizer for semantic segmentation based on a random forest, which is a recognition system using a machine learning system other than a neural network, may also be used.

[0033] Figure 4 FIG. is an example showing an image of the road surface in front of the vehicle 10 and the recognition result of the category of the object in each pixel of the image. In Figure 4 In the example shown, in the image 400, on the left and right sides of the own lane 401, lane demarcation lines 402 represented by solid lines and guideway markings 403 represented by dot displays are respectively shown.

[0034] Therefore, in the recognition result 410 for each pixel of the image 400, each pixel is classified into any one of the own lane 401, lane demarcation line 402, other marking (guideway marking) 403, other road surface 404, stationary object 405, and moving object 406. In addition, in this example, regarding the lane in the direction opposite to the traveling direction of the vehicle 10, it is not recognized as the other road surface 404, but as the stationary object 405.

[0035] The recognition unit 31 notifies the detection unit 32 of the recognition result of each pixel.

[0036] Based on the recognition results of the objects of each pixel, the detection unit 32 obtains a set of pixels representing the left and right boundaries of the own lane. In the present embodiment, the detection unit 32 uses the end portions on the own-lane side of the lane demarcation lines as the boundaries of the own lane.

[0037] Here, assuming that a reference point is set within the own lane for each pixel column in the direction intersecting the own lane, and scanning is sequentially performed in the direction away from the reference point, referring to the recognition results of the objects of each pixel, the position where the lane demarcation line first appears is detected as the boundary of the own lane. In this case, for each pixel column, the pixel column is temporarily scanned to set the reference point, and after setting the reference point, the pixel column is scanned again, so the computational load becomes high. Therefore, it is not preferable to require such multiple scans for detecting the boundary of the own lane. Further, when the order of storing the values of each pixel in the cache memory of the memory 22 or the processor 23 is different from the scanning direction, the efficiency is reduced from the viewpoint of memory access.

[0038] Therefore, the detection unit 32 sets a scanning direction from one end to the other end of each pixel column in the direction intersecting the own lane in the image. Then, for each pixel column, along the scanning direction, the detection unit 32 sequentially determines whether the position corresponding to the pixel group is within the own lane based on the arrangement order of the categories of the objects represented in a continuous predetermined number of pixel groups and the determination result as to whether the position corresponding to the pixel group immediately preceding in the scanning direction is within the own lane, thereby detecting the left and right boundaries of the own lane.

[0039] In the present embodiment, since the extending direction of the own lane becomes the vertical direction in the image, the scanning direction is set to the horizontal direction so as to cross the extending direction of the own lane. Further, the values of the respective pixels of each pixel column in the horizontal direction of the image are sequentially stored from the left end to the right direction. Therefore, in order to improve the efficiency of memory access, the detection unit 32 sets the left end as the starting point of scanning for each pixel column in the horizontal direction, and sequentially scans from the starting point to the right direction. Further, in the present embodiment, in the scanning of each pixel column, the detection unit 32 sequentially scans the arrangement order of the categories of the objects represented in the pixel group including two pixels arranged in the horizontal direction.

[0040] Figure 5It is a graph that represents how the position corresponding to the pixel of interest during scanning changes as a state transition graph based on the arrangement order of the categories of objects for each pixel group. As described above, in the present embodiment, among the categories of objects to be recognized, there are 7 types of objects including the own lane 501, the identification part 502 of the lane division line, the non-identification part 503 of the lane division line, other markings 504, other road surfaces 505, moving objects 506, and stationary objects 507. Therefore, among the 49 arrangement orders of the categories of objects in each pixel group, the 27 arrangement orders that are concerned in the detection of the boundary of the own lane are classified into 15 cases from case 1 to case 15. For example, case 1 represents a case where the left pixel in the pixel group corresponds to the lane division line and the right pixel corresponds to other markings, and case 13 represents a case where the categories of objects are arranged in the reverse left-right order compared to case 1. In addition, case 2 represents a case where the left pixel in the pixel group corresponds to other markings and the right pixel corresponds to the own lane, and case 7 represents a case where the categories of objects are arranged in the reverse left-right order compared to case 2. Furthermore, case 3 represents a case where the left pixel in the pixel group corresponds to the lane division line or other road surface and the right pixel corresponds to the own lane, and case 8 represents a case where the categories of objects are arranged in the reverse left-right order compared to case 3. Additionally, case 4 represents a case where the left pixel in the pixel group corresponds to a moving object and the right pixel corresponds to the own lane, and case 9 represents a case where the categories of objects are arranged in the reverse left-right order compared to case 4. Additionally, case 6 represents a case where the left pixel in the pixel group corresponds to a stationary object and the right pixel corresponds to the own lane, and case 10 represents a case where the categories of objects are arranged in the reverse left-right order compared to case 6. Additionally, case 5, case 11, and case 12 respectively represent cases where any one pixel in the pixel group is the own lane, other markings, and a moving object. Additionally, case 14 represents a case where the left pixel in the pixel group corresponds to other markings and the right pixel corresponds to other road surfaces, moving objects, or stationary objects. Moreover, case 15 represents a case where the left pixel in the pixel group corresponds to a moving object and the right pixel corresponds to an object other than the moving object and the own lane.

[0041] In addition, in the state transition diagram 500, there are five states 511 to 515. Among them, state 511 indicates that the position of interest is outside the current lane. In addition, state 512 indicates that the position of interest is within the current lane. Furthermore, state 513 indicates that the position of interest may be within the current lane and represents the position of a moving object. Additionally, state 514 indicates that the position of interest may be within the current lane and represents the position of other markings. Moreover, state 515 represents the state at the start of scanning, indicating that the position of interest is not classified as either within or outside the current lane. Also, the arrows between the states represent the state transition from the root of the arrow to the tip of the arrow, and the case number attached to the arrow represents the case number of the pixel group to which the state transition is applied (any one of cases 1 to 15). Furthermore, when the process associated with each case number represents the arrangement of the categories of the objects in the pixel group of interest, it represents the content of the process executed together with the state transition. Additionally, "right end" and "left end" respectively represent the position of the right lane boundary line of the current lane and the position of the left lane boundary line of the current lane. Moreover, (k - 1) represents the position corresponding to the pixel on the left side of the pixel group of interest, and k represents the position corresponding to the pixel on the right side of the pixel group of interest.

[0042] After the start of scanning, the detection unit 32 sets the pixel group of interest such that the pixel at the left end, which is the start position of scanning, becomes the position of interest and the pixel on the left side of the pixel group of interest. Then, the detection unit 32 sets the state of the position of interest to one of states 511 to 514 according to the category of the object of the pixel at the left end. That is, when the category of the object of the pixel at the left end is a stationary object, other road surface, or lane marking, the detection unit 32 determines that the state of the position of interest is state 511, which is outside the current lane. On the other hand, when the category of the object of the pixel at the left end is the current lane, the detection unit 32 determines that the state of the position of interest is state 512, which is within the current lane. In addition, when the category of the pixel at the left end is a moving object, the detection unit 32 determines that the state of the position of interest is state 513, where the position of interest may be within the current lane and represents the position of a moving object. That is, the detection unit 32 sets the position of interest as a candidate position within the current lane. Then, when the category of the pixel at the left end is other markings, the detection unit 32 determines that the state of the position of interest is state 514, where the position of interest may be within the current lane and represents the position of other markings. In this case, the detection unit 32 also sets the position of interest as a candidate position within the current lane.

[0043] After the detection unit 32 determines the state of the initial position of interest, it determines which of the cases 1 to 15 the arrangement of the categories of the objects in the pixel group of interest conforms to. Then, the detection unit 32 refers to the state transition diagram 500 and changes the state in accordance with the arrangement order of the objects in the pixel group of interest in the state of the position of interest. For example, if the state of the position of interest is the state 511 outside the own lane and the arrangement order of the categories of the objects in the pixel group of interest conforms to case 3, the category of the object in the pixel on the left side of the pixel group of interest is a lane demarcation line or other road surface, and the category of the object in the pixel on the right side of the pixel group is the own lane. Therefore, the detection unit 32 sets the position of the pixel on the right side of the pixel group of interest as the position of the left boundary of the valid own lane and changes the state to the state 512 inside the own lane. Additionally, if the state of the position of interest is the state 511 outside the own lane and the arrangement order of the categories of the objects in the pixel group of interest conforms to case 6, the category of the object in the pixel on the left side of the pixel group of interest is a stationary object, and the category of the object in the pixel on the right side of the pixel group is the own lane. Therefore, in this pixel group of interest, there is no lane demarcation line or other road surface in between, and the stationary object is in contact with the own lane. Therefore, the detection unit 32 sets the position of the pixel on the right side of the pixel group of interest as the position of the left boundary of the invalid own lane and changes the state to the state 512 inside the own lane. Additionally, if the state of the position of interest is the state 513 that may be inside the own lane and the arrangement order of the categories of the objects in the pixel group of interest conforms to case 15 (the object in the right pixel is other than the own lane and a moving object), the position of interest should be outside the own lane rather than inside the own lane. Therefore, the detection unit 32 changes the state to the state 511 outside the own lane and discards the candidate of the own lane set at the position of interest. On the contrary, if the state of the position of interest is the state 513 that may be inside the own lane and the arrangement order of the categories of the objects in the pixel group of interest conforms to case 4 (the object in the right pixel is the own lane), the position of interest should be inside the own lane. Therefore, the detection unit 32 changes the state to the state 512 inside the own lane. Additionally, the detection unit 32 officially determines that the candidate of the own lane set at the position of interest is inside the own lane and updates the imaginary position of the right boundary of the own lane to the position of the pixel on the right side of the pixel group of interest. Similarly, if the state of the position of interest is the state 514 that may be inside the own lane and the arrangement order of the categories of the objects in the pixel group of interest conforms to case 13 or case 14, the position of interest should be outside the own lane rather than inside the own lane. Therefore, the detection unit 32 changes the state to the state 511 outside the own lane and discards the candidate of the own lane set at the position of interest. On the contrary, if the state of the position of interest is the state 514 that may be inside the own lane and the arrangement order of the categories of the objects in the pixel group of interest conforms to case 2 (the object in the right pixel is the own lane), the position of interest should be inside the own lane.Therefore, the detection unit 32 changes the state to the in-lane state 512. In addition, the detection unit 32 officially determines that the candidate of the own lane set at the position of interest is within the own lane, and updates the imaginary position of the right boundary of the own lane to the position of the pixel on the right side of the pixel group of interest. Furthermore, if the state of the position of interest is the in-lane state 512 and the arrangement order of the object categories of the pixel group of interest conforms to case 8 or case 13 (the object of the right pixel is a lane division line or other road surface), the position of the left pixel of the pixel group of interest is set as the position of the effective right boundary of the own lane, and the state is changed to the out-of-lane state 511. Furthermore, if the state of the position of interest is the in-lane state 512 and the arrangement order of the object categories of the pixel group of interest conforms to case 10, case 14, or case 15, the position of the left pixel of the pixel group of interest is set as the invalid right boundary of the own lane, and the state is changed to the out-of-lane state 511. In addition, for combinations of the states of other positions of interest and the arrangement order of the objects of the pixel group of interest, the state may be changed according to the state transition diagram 500.

[0044] Hereinafter, similarly, the detection unit 32 moves the position of interest one pixel at a time to the right while setting a pixel group to be observed with the position of interest as the left pixel, and changes the state of the position of interest according to the state of the current position of interest and the arrangement order of the object categories of the pixel group of interest in accordance with the state transition diagram 500. Then, when the position of interest reaches the right end of the scanned pixel column, the detection unit 32 detects the position registered as the effective left boundary position of the own lane as the left boundary position of the own lane in this pixel column. Similarly, if the detection unit 32 has a position registered as the effective right boundary position of the own lane, it detects the position as the right boundary position of the own lane in this pixel column. Furthermore, when the position of interest reaches the right end of the scanned pixel column, if there is no position registered as the effective left boundary position of the own lane and only an invalid left position is registered, the left boundary of the own lane is not detected in this pixel column. Similarly, when the position of interest reaches the right end of the scanned pixel column, if there is no position registered as the effective right boundary position of the own lane and only an invalid right position is registered, the right boundary of the own lane is not detected in this pixel column.

[0045] In this way, the detection unit 32 can detect the left and right boundaries of the own lane only by scanning one pixel column once.

[0046] In addition, even when other markings are provided parallel to the lane demarcation line, such as the guideway markings, since the lane demarcation line and the other markings do not meet, it is assumed that in the image, there are pixels representing the lane between the pixels representing the lane demarcation line and the pixels representing the other markings. However, the farther the position is from the vehicle 10, that is, the farther the position is from the camera 2, the smaller the object existing at that position is captured in the image. Therefore, in the area on the image corresponding to the road surface at a certain distance or more from the vehicle 10, it becomes difficult to identify the road surface portion of the lane between the lane demarcation line and the other markings, and the pixels representing the lane demarcation line and the pixels representing the other markings sometimes meet. Therefore, the detection unit 32 can also, for a pixel column corresponding to a position on the image that is a predetermined distance or more away from the vehicle 10, as Figure 5 shown in Case 1 or Case 13, detect the position of the other marking as the boundary of the lane when the categories of the objects identified for each pixel are arranged in the order of the lane demarcation line, the other marking, and the lane in sequence from the farther side relative to the vehicle 10. In addition, the predetermined distance can be set, for example, as the distance at which the road surface of the lane between the lane demarcation line and the other marking cannot be recognized on the image due to the resolution of the image obtained by the camera 2.

[0047] Figure 6 is a diagram showing an example of an image representing a lane demarcation line and other markings. As Figure 6 shown, in the image 600, the guideway markings 602 are marked along the lane demarcation line 601 and at a certain interval from the lane demarcation line 601. However, in the recognition result 610 of the categories of the objects of each pixel in the image 600, at the position corresponding to the area 620, since it is too far from the vehicle 10, the pixel group representing the lane demarcation line 601 and the pixel group representing the guideway markings 602 meet. However, as described above, the detection unit 32 detects the position of the other marking as the boundary position of the lane when the categories of the objects identified for each pixel are arranged in the order of the lane demarcation line, the other marking, and the lane, that is, when the lane demarcation line and the other marking meet. Thus, in such an area 620, the boundary of the lane can also be correctly detected. Therefore, the detection unit 32 can detect the boundary of the lane up to a position farther from the vehicle 10. Furthermore, for a position not so far from the vehicle 10, as described above, by taking the position of the lane in the pixel column arranged in the order of the lane demarcation line or other road surface, and then the lane in sequence from the farther side relative to the vehicle as the boundary of the lane, even when other markings such as guideway markings are provided along the lane demarcation line, it is possible to prevent the other marking and the boundary of the lane from being misdetected as the boundary of the original lane.

[0048] In addition, depending on the state of the road surface of the road on which the vehicle 10 travels, other markings such as lane demarcation lines or guideway markings may sometimes be unclear. In such a case, the recognition accuracy of the category of the object of each pixel by the recognizer may sometimes decrease.

[0049] Figure 7 FIG. is an example of a diagram showing an image of a road and the recognition result of the category of the object of each pixel in the case where the lane demarcation line is unclear. In the road shown in the image 700, the lane demarcation line 701 and other markings 702 become unclear. Therefore, in the recognition result 710 of the category of the object of each pixel of the image 700, there are pixels that represent other markings 702 but are misrecognized as representing the lane demarcation line, and pixels that represent other road surfaces but are misrecognized as representing the own lane. In such a case, there is a possibility that the position of the boundary line of the own lane is also misdetected. In particular, regarding the pixels included in the region adjacent to the own lane across the lane demarcation line, when they are misrecognized as representing the own lane, there is a possibility that a plurality of regions determined to be within the own lane (hereinafter referred to as own lane regions) are detected in one pixel column.

[0050] Therefore, for the pixel column in which the detection unit 32 detects a plurality of own lane regions as a result of scanning, the detection unit 32 selects the most accurate own lane region among the plurality of own lane regions. Then, among the left and right end points of the selected own lane region, the detection unit 32 detects the end point where there is no adjacent other own lane region on the image side compared to the selected own lane region as the effective boundary of the own lane, while regarding the end point where there is an adjacent other own lane region on the image side compared to the selected own lane region as an invalid boundary of the own lane. In addition, for example, the wider the width of the own lane region, the more accurate the detection unit 32 determines the own lane region to be. For example, in Figure 7 In the shown pixel column 720, two own lane regions 721 and 722 are detected, and among them, the width of the right own lane region 722 is wider than the width of the left own lane region 721. Therefore, the right own lane region 722 is selected. Moreover, there is no other own lane region on the right side of the own lane region 722, so the right end point of the own lane region 722 is detected as the position of the effective boundary of the own lane. On the other hand, there is an other own lane region 721 on the left side of the own lane region 722, so the left end point of the own lane region 722 is regarded as an invalid boundary. Thus, the detection unit 32 can suppress misdetection of the position of the boundary of the own lane even when the lane demarcation line or the like shown in the image is unclear.

[0051] According to the modified example, the detection unit 32 may also, for a pixel column in which a plurality of own-lane regions are detected in the latest image, use, as the most accurate own-lane region, a portion in the pixel column at the position closest to the pixel column among the plurality of own-lane regions in the immediately preceding image that has the smallest difference from the own-lane region. Alternatively, when there are a plurality of sets of consecutive pixels within the own lane, the detection unit 32 may also merge these plurality of sets of pixels into an own-lane region and use the left and right end points of the merged own-lane region as the positions of the boundaries of the own lane.

[0052] The detection unit 32 notifies the vehicle control unit 33 of the positions of the effective boundaries of the detected own lane for each pixel column.

[0053] The vehicle control unit 33 controls the travel of the vehicle 10 based on the positions of the boundaries of the own lane detected for each pixel column. For example, the vehicle control unit 33 generates a planned travel path of the vehicle 10 in a predetermined section from the current position of the vehicle 10 to a predetermined distance forward in such a manner that the vehicle 10 passes through the center of the own lane, based on the positions of the left and right boundaries of the own lane detected for each pixel column. Therefore, the vehicle control unit 33 uses, for example, a line approximated according to the least squares method as the left boundary line of the own lane for the set of pixels representing the left boundary of the own lane. Similarly, the vehicle control unit 33 uses, for example, a line approximated according to the least squares method as the right boundary line of the own lane for the set of pixels representing the right boundary of the own lane. In addition, the position of each pixel on the image obtained by the camera 2 corresponds to the azimuth of the object represented in that pixel as viewed from the camera 2. Therefore, the vehicle control unit 33 may set the planned travel path on the image such that the distances from the position corresponding to the center line along the traveling direction of the vehicle 10 to the left and right boundary lines of the own lane are equal. Then, the vehicle control unit 33 controls each part of the vehicle 10 such that the vehicle 10 travels along the planned travel path. For example, the vehicle control unit 33 calculates a steering angle for causing the vehicle 10 to travel along the planned travel path, and outputs a control signal corresponding to the steering angle to an actuator (not shown) that controls the steering wheel of the vehicle 10. Further, the vehicle control unit 33 calculates the acceleration of the vehicle 10 based on the current vehicle speed of the vehicle 10 measured by a vehicle speed sensor (not shown) or the acceleration of the vehicle 10 measured by an acceleration sensor (not shown), etc., in such a manner that the vehicle 10 travels at a speed specified by the driver, or at a speed set according to the legal speed of the road on which the vehicle 10 is currently traveling, or at a speed that maintains a certain inter-vehicle distance from other vehicles traveling ahead of the vehicle 10, and sets the accelerator opening or the braking amount so as to achieve this acceleration. Then, the vehicle control unit 33 calculates the fuel injection amount according to the set accelerator opening, and outputs a control signal corresponding to the fuel injection amount to the fuel injection device of the engine of the vehicle 10. Alternatively, the vehicle control unit 33 outputs a control signal corresponding to the set braking amount to the brakes of the vehicle 10.

[0054] Figure 8 is a flowchart of the operation of the vehicle control process including the lane boundary detection process executed by the processor 23. Whenever the processor 23 receives an image from the camera 2, it executes the vehicle control process according to the Figure 8 shown flowchart of the operation. In addition, in the flowchart of the operation shown below, the processes of steps S101 to S102 correspond to the lane boundary detection process.

[0055] The recognition unit 31 of the processor 23 inputs the image into the recognizer to recognize the category of the object represented in each pixel of the image (step S101).

[0056] The detection unit 32 of the processor 23, for each pixel column in the direction intersecting with the own lane in the image, along the scanning direction from one end of the pixel column to the other end, sequentially determines whether the position corresponding to the concerned pixel group is within the own lane based on the arrangement order of the categories of the objects represented in the concerned pixel group including a continuous predetermined number of pixels and the determination result of whether the position corresponding to the immediately preceding pixel group in the scanning direction is within the own lane, thereby detecting the left and right boundary positions of the own lane (step S102).

[0057] The vehicle control unit 33 of the processor 23 controls the vehicle 10 in such a manner that the vehicle 10 travels along the own lane according to the positions of the left and right boundaries of the own lane detected in each pixel column (step S103). Then, the processor 23 ends the vehicle control process.

[0058] As described above, the lane boundary detection device inputs the image into a recognizer that has been pre-learned to recognize the category of the object represented in each pixel, and recognizes the type of the object represented in each pixel. Then, the lane boundary detection device, for each pixel column in the direction intersecting with the own lane, along the scanning direction from one end of the pixel column to the other end, sequentially determines whether the position corresponding to the concerned pixel group is within the own lane based on the arrangement order of the categories of the objects represented in the concerned pixel group including a continuous predetermined number of pixels and the determination result of whether the position corresponding to the immediately preceding pixel group in the scanning direction is within the own lane, thereby detecting the left and right boundary positions of the own lane. Thus, the lane boundary detection device can detect the boundary of the own lane by one scan for each pixel column, so the arithmetic load can be reduced. In addition, the lane boundary detection device only needs to sequentially scan each pixel column in a specific direction, so the arrangement order of the pixels when the image is stored in the memory can be made consistent with its scanning direction, so the efficiency of memory access can be improved.

[0059] In addition, according to the modification example, the detection unit 32 may also invalidate the lane boundary position of any one pixel column when the lane boundary position between adjacent pixel columns on the same image is separated from a predetermined distance or more. For example, the detection unit 32 may also invalidate the lane boundary position of the pixel column corresponding to the position away from the vehicle 10 when detecting two pixel columns in which the lane boundary position is separated from a predetermined distance or more. Alternatively, the detection unit 32 may also invalidate the lane boundary position of the pixel column in the latest image among these images when the lane boundary positions of the pixel columns at the same position between two images obtained in time series are separated from a predetermined distance or more.

[0060] The computer program for implementing the functions of the respective parts of the processor 23 of the lane boundary detection device according to the above-described embodiment may also be provided in the form of a computer-readable removable recording medium such as a semiconductor memory, a magnetic recording medium, or an optical recording medium.

[0061] As described above, those skilled in the art can make various changes within the scope of the present invention in accordance with the implemented embodiments.

Claims

1. A lane boundary detection device, comprising: an identification unit that inputs an image representing a peripheral area of the vehicle obtained by an imaging unit mounted on the vehicle into an identifier learned in such a way as to identify the category of an object represented in each pixel, and thereby identifies the category of the object represented in each pixel of the image, where the category of the object at least includes a category representing an object within the current lane in which the vehicle is traveling and a category representing an object outside the current lane; and a detection unit that scans each pixel column in a direction intersecting the current lane in the image along a scanning direction from one end of the pixel column toward the other end, thereby detecting the boundary of the current lane, in the scanning of the pixel column, the detection unit sets a group of concerned pixels including a predetermined number of consecutive pixels of the pixel column in such a way that a pixel at one end as the scanning start position becomes a concerned position and a pixel on the one-end side of the group of concerned pixels, and determines whether the concerned position corresponding to the group of concerned pixels is within the current lane according to the arrangement order of the categories of the objects represented in the pixels included in the group of concerned pixels and the determination result as to whether a concerned position corresponding to the group of concerned pixels immediately preceding in the scanning direction is within the current lane, the detection unit moves the concerned position pixel by pixel toward the other end side while resetting the group of concerned pixels with the concerned position as the pixel on the one-end side, and performs the scanning of the pixel column until the concerned position reaches the other end of the pixel column, thereby detecting the boundary of the current lane.

2. The lane boundary detection device according to claim 1, wherein, when there are a plurality of sets of consecutive pixels that are within the current lane in any one of the pixel columns, the detection unit determines the set of pixels that are most definitely within the current lane among the plurality of sets of pixels as the lane area representing the current lane, and detects, as the boundary of the current lane, a boundary of the two boundaries of the lane area along the scanning direction where there are no pixels representing the current lane on the image end side compared with this boundary.

3. The lane boundary detection device according to claim 1 or 2, wherein, the category of the object further includes other markings outside the lane demarcation lines provided on the road, the category representing an object within the current lane includes the current lane itself, and the category representing an object outside the current lane includes other road surfaces outside the current lane. The detection unit determines that, for a pixel column in the image corresponding to a position where the distance from the vehicle is greater than a predetermined distance, the position of the other marking represents the boundary of the own lane when the categories of the objects are arranged in the order of the lane dividing line, the other marking, and the own lane from the side farther away from the vehicle. On the other hand, for a pixel column in the image corresponding to a position within the predetermined distance from the vehicle, the detection unit determines that the position of the own lane represents the boundary of the own lane when the categories of the objects are arranged in the order of the other road surface or the lane dividing line, and then the own lane from the side farther away from the vehicle.

4. A lane boundary detection method, comprising: identifying the category of an object represented in each pixel of the image by inputting an image representing a peripheral area of the vehicle obtained by an imaging unit mounted on the vehicle into an identifier learned in a manner of identifying the category of the object represented in each pixel, where the category of the object at least includes a category representing an object within the own lane in which the vehicle is traveling and a category representing an object outside the own lane; and detecting the boundary of the own lane by scanning each pixel column in a direction intersecting the own lane in the image along a scanning direction from one end of the pixel column to the other end. In the scanning of the pixel column, a group of attention pixels including a predetermined number of consecutive pixels of the pixel column is set such that a pixel at one end as the scanning start position becomes the attention position and becomes a pixel on the one-end side of the group of attention pixels. Based on the arrangement order of the categories of the objects represented in the pixels included in the group of attention pixels and the determination result as to whether the attention position corresponding to the immediately preceding group of attention pixels in the scanning direction is within the own lane, it is determined whether the attention position corresponding to the group of attention pixels is within the own lane. While moving the attention position pixel by pixel toward the other end side, the group of attention pixels with the attention position as a pixel on the one-end side is reset, and the pixel column is scanned until the attention position reaches the other end of the pixel column, thereby detecting the boundary of the own lane.

5. A computer program product for lane boundary detection, which causes a processor to execute: identifying the category of an object represented in each pixel of the image by inputting an image representing a peripheral area of the vehicle obtained by an imaging unit mounted on the vehicle into an identifier learned in a manner of identifying the category of the object represented in each pixel, where the category of the object at least includes a category representing an object within the own lane in which the vehicle is traveling and a category representing an object outside the own lane; and detecting the boundary of the own lane by scanning each pixel column in a direction intersecting the own lane in the image along a scanning direction from one end of the pixel column to the other end. In the scanning of the pixel column, a group of pixels of interest including a predetermined number of consecutive pixels of the pixel column is set in such a way that a pixel at one end as the scanning start position becomes the position of interest and a pixel on the one-end side of the group of pixels of interest, and it is determined whether the position of interest corresponding to the group of pixels of interest is within the own lane based on the arrangement order of the categories of the object represented by the pixels included in the group of pixels of interest and the determination result as to whether the position of interest corresponding to the immediately preceding group of pixels of interest in the scanning direction is within the own lane. While moving the position of interest pixel by pixel toward the other end side, the group of pixels of interest with the position of interest as the pixel on the one-end side is reset, and the pixel column is scanned until the position of interest reaches the other end of the pixel column, thereby detecting the boundary of the own lane.

Images