Driving lane estimation system

Abstract

To increase the reliability of travel lane estimation.SOLUTION: A travel lane estimation system comprises: an image recognition unit that identifies a compartment line by image recognition in a captured image of a road on which a vehicle travels, captured by an image capturing unit mounted on the vehicle; and a travel lane estimation unit that estimates a travel lane of the vehicle based on an image recognition result of the compartment line in each of a plurality of regions having different positions in a longitudinal direction of the road, which is a direction in which the road extends, in the captured image.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a driving lane estimation system. [Background technology]

[0002] Conventionally, there is known a technique for estimating the lane a vehicle is traveling in on a road. Patent Document 1 discloses a technique for identifying unit lines that make up dashed lane marks (dividing lines) based on an image captured by an on-board camera, and calculating the travel distance based on the number of unit lines that the vehicle has passed through. Here, the unit lines are the white line portions that make up the dashed lines. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2006-153565 Summary of the Invention [Problem to be solved by the invention]

[0004] However, there is a problem in that the lane markings cannot be identified due to reasons such as the vehicle being hidden by a vehicle ahead at the vehicle's position, making it impossible to estimate the driving lane.

[0005] The present invention has been made in view of the above-mentioned problems, and has an object to increase the reliability of driving lane estimation. [Means for solving the problem]

[0006] In order to achieve the above-mentioned object, the present invention is a driving lane estimation system comprising an image recognition unit that identifies dividing line marks by image recognition in an image of a road on which the vehicle is traveling, captured by an imaging unit mounted on the vehicle, and a driving lane estimation unit that estimates the driving lane of the vehicle based on the image recognition results of the dividing line marks in each of a plurality of partial sections in the captured image that are positioned differently in the road length direction, which is the direction in which the road extends.

[0007] In other words, the driving lane estimation system refers to the image recognition results of the dividing line for each of multiple partial sections, thereby increasing the reliability of identifying the dividing line, and therefore increasing the reliability of estimating the driving lane using the dividing line. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a configuration diagram of a driving lane estimation system. [Figure 2] FIG. 2A is a diagram showing a real space, and FIG. 2B is a diagram showing an example of a frame. [Figure 3] FIG. 2 is an explanatory diagram of information used in the driving lane estimation system. [Figure 4] FIG. 10 is a diagram illustrating an example of the data configuration of a frame information table. [Figure 5] FIG. 10 is a diagram illustrating an example of a data configuration of an in-link position information table. [Figure 6] 10 is a flowchart showing a driving lane estimation process. [Figure 7] FIG. 10 is an explanatory diagram of a driving lane estimation process. [Figure 8] 10 is a flowchart showing a recording process. [Figure 9] 10 is a flowchart showing a lane marking identification process. [Figure 10] FIG. 10 is a diagram illustrating an example of a data configuration of a vehicle range information table according to the second embodiment. [Figure 11] FIG. 11 is a diagram illustrating an example of a data configuration of an in-link position information table according to the second embodiment. [Figure 12]10 is a flowchart showing a recording process according to the second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0009] (First embodiment) FIG. 1 is a diagram showing the configuration of a driving lane estimation system 10 according to this embodiment. The driving lane estimation system 10 is mounted on a vehicle. Hereinafter, a vehicle on which the driving lane estimation system 10 is mounted will be referred to as the mounted vehicle. The driving lane estimation system 10 estimates the lane in which the mounted vehicle is traveling (hereinafter referred to as the driving lane) based on the image recognition results of an image of the road on which the mounted vehicle is traveling, which is captured by a capturing unit installed in the mounted vehicle.

[0010] The driving lane estimation system 10 includes a control unit 11 including a CPU, RAM, ROM, etc., and a recording medium 12. The control unit 11 executes various programs stored in the recording medium 12 and the ROM. The control unit 11 of this embodiment performs driving lane estimation by executing a driving lane estimation program.

[0011] The driving lane estimation system 10 is connected to an imaging unit 31, a GNSS receiving unit 32, a vehicle speed sensor 33, a gyro sensor 34, and a user I / F unit 35. The imaging unit 31, the GNSS receiving unit 32, the vehicle speed sensor 33, the gyro sensor 34, and the user I / F unit 35 are mounted on a vehicle.

[0012] The image capturing unit 31 is installed in the vehicle so as to capture images in the traveling direction (forward) of the vehicle, and the image capturing unit 31 captures images periodically. The image capturing unit 31 captures images, for example, every 33 ms. The frames (captured images) captured by the image capturing unit 31 are input to the driving lane estimation system 10 and used in the driving lane estimation process. The installation position, orientation, and angle of view of the image capturing unit 31 are adjusted so that the image capturing range of the image capturing unit 31 is within a predetermined range.

[0013] The imaging range will be described with reference to Figures 2A and 2B. Figure 2A is a diagram showing a road A3, an equipped vehicle C located on road A3, and an imaging range D in real space. Figure 2B is a diagram showing an example of a frame 210 captured by the imaging unit 31. Frame 210 is a captured image of the road on which the equipped vehicle is traveling. The driving lane estimation system 10 identifies lane markings, road boundaries, and road edges through image recognition of the frame, and estimates the driving lane of the equipped vehicle based on the image recognition results. Therefore, the imaging range must include lane markings, road boundaries, and road edges. Here, a road boundary is a line between a road and other areas. For example, if a sidewalk is provided beside the road, the boundary between the road and the sidewalk is the road boundary. The road edge is the edge of the lane marking closest to the road boundary.

[0014] As shown in FIG. 2A, a sidewalk A1 is provided beside the road, and a roadside tree area A2 is provided between the sidewalk A1 and road A3. In this case, boundary line B1 between road A3 and roadside tree area A2 is the road boundary line, and edge B2 of the lane marking closest to the road boundary line is the road edge. The shooting range is assumed to include at least one road boundary line closer to the vehicle. Furthermore, the shooting range is assumed to include at least one lane marking on each side of the vehicle.

[0015] Furthermore, as will be described in detail later, the driving lane estimation system 10 uses the image recognition results for a depth section 212 in the frame 210, which corresponds to a predetermined range 202 in the depth direction in real space, for driving lane estimation. The depth section 212 is, for example, a section corresponding to a range from a point 10 m to a point 30 m in the depth direction, based on the position of the imaging unit 31 (hereinafter referred to as the imaging position) in real space. Hereinafter, the distance in the depth direction in real space will be referred to as the depth distance.

[0016] Within the shooting range, there is a high possibility that the hood 220 of the vehicle equipped with the camera will be captured in a range corresponding to a depth distance shorter than 10 m. Therefore, the boundary on the front side of the depth section 212 is set to a depth distance of 10 m. Furthermore, in image recognition, the recognition accuracy decreases as the depth distance of the recognition target increases. Therefore, from the perspective of ensuring recognition accuracy, the boundary on the back side of the depth section 212 is set to a depth distance of 30 m. The depth section 212 may be set according to the accuracy of image recognition, and the depth distance and width of the depth section 212 are not limited to those in the embodiment.

[0017] Here, the relationship between the relative position of the road surface with respect to the vehicle and the position within the frame will be described. Since the shooting position (position of the shooting unit 31) is fixed with respect to the vehicle, the relative positional relationship between the shooting position and the road surface is kept constant. Therefore, the position of the depth section 212 in the frame 200, which corresponds to the range 202 on the road surface in real space, is constant regardless of the position of the vehicle (vehicle position).

[0018] In the driving lane estimation system 10 of this embodiment, positional relationship information associating each position in a frame with a position on the road surface in real space photographed at each position is assumed to be stored in advance in the recording medium 12. The control unit 11 can set a depth section 212 corresponding to the range 202 of real space in the photographed image by referring to the positional relationship information. Note that the position on the road surface indicated in the positional relationship information is a relative position with the photographing position as the reference. The position on the road surface is indicated by a depth distance and a lateral distance with the photographing position as the reference. Here, the lateral distance is the distance from the photographing position in the lateral direction perpendicular to the depth direction.

[0019] Next, with reference to FIG. 3, information used in the driving lane estimation system 10 will be described. The driving lane estimation system 10 divides the depth section 212 into a plurality of partial sections 214 in the depth direction and records the image recognition results of the lane markings for each partial section. In this embodiment, the partial sections 214 are sections with a width of 1 m in the depth direction. A section between a depth distance of 10 m and 11 m is referred to as a partial section with a depth distance of 10 m. The depth section 212 with a depth distance of 10 m to 30 m is divided into 30 partial sections with depth distances of 10 m, 11 m, ..., 29 m.

[0020] The depth distance of the partial section is converted into an in-link position. Here, the in-link position is the position within the link on which the loaded vehicle is traveling, and is expressed as the distance from the end of the link into which the loaded vehicle has entered (the link start point). For example, on a link extending from south to north, a position on the link that is 1 m north of the link start point is a point 1 m in-link position.

[0021] As shown in Figure 3, assume that mounted vehicle C captures frame 210 at a position 10 m within the link. The partial section with a depth distance of 10 m obtained from this frame 210 is a section from depth distance 10 m to 11 m, but because this partial section is based on the intra-link position 10 m, it is a section from 20 m to 21 m within the link. The section from 20 m to 21 m within the link is referred to as the intra-link section 20 m. By converting to an intra-link position in this way, it is possible to identify partial sections corresponding to the same range in real space even for frames captured at different capturing positions.

[0022] Assume that four lane markings are identified by image recognition, as shown in frame 210 of FIG. 2B. In this case, the image recognition results for each of the four lane markings are recorded for each partial section. In this embodiment, the image recognition results for the lane markings on the left side of the vehicle are designated EL and NL in order of proximity to the vehicle. The image recognition results for the lane markings on the left side of the vehicle are designated ER and NR in order of proximity to the vehicle. Here, EL, NL, ER, and NR are lane marking identification information. Furthermore, the image recognition result for the NL lane markings obtained in the partial section with a depth distance of 10 m is represented as NLd10. Here, d indicates the depth distance. Similarly, the image recognition result for the EL lane markings obtained in the partial section with a depth distance of 10 m is represented as ELd10, and the image recognition result for the NL lane markings obtained in the partial section with a depth distance of 30 m is represented as NLd30. Similarly, the image recognition result for the NL lane markings obtained in the partial section with a depth distance of 10 m within the link is represented as NLk10, and the image recognition result for the NL lane markings obtained in the partial section with a depth distance of 30 m within the link is represented as NLk30. Here, k indicates the position within the link.

[0023] The image recognition results for each lane marking also include the type, color, and lateral distance of the lane marking. Here, lane marking types include solid lines and dashed lines. Lane marking colors include white and yellow. The lateral distance is the lateral distance from the photographing position to the lane marking, and is a distance in real space. For example, in the example shown in FIG. 3, the lateral distance to the NL is indicated by arrow 230.

[0024] The GNSS receiver 32 shown in FIG. 1 is a device that receives GNSS (Global Navigation Satellite System) signals. It receives radio waves from navigation satellites and outputs a signal for calculating the vehicle position via an interface (not shown). The control unit 11 acquires this signal and determines the vehicle position based on this signal. The vehicle speed sensor 33 outputs a signal corresponding to the rotation speed of wheels provided on the vehicle. The control unit 11 acquires this signal via an interface (not shown) and obtains the vehicle speed of the vehicle. The gyro sensor 34 detects angular acceleration of the vehicle turning in a horizontal plane and outputs a signal corresponding to the orientation of the vehicle. The control unit 11 acquires this signal and obtains the traveling direction of the vehicle. The vehicle speed sensor 33, gyro sensor 34, etc. are used to determine the traveling path of the vehicle. In this embodiment, the vehicle position is determined based on the departure point and traveling path of the vehicle, and the vehicle position determined based on the departure point and traveling path is corrected based on the output signal of the GNSS receiver 41.

[0025] The user I / F unit 35 is an interface unit for receiving instructions from the user and providing various information to the user, and includes input units such as a display unit and switches, and an audio output unit such as a speaker. The user I / F unit 35 receives control signals from the control unit 20 and displays images for providing various guidance, such as a map including the current location of the vehicle and a planned driving route, on the display unit. Examples of the display unit include a center display and a head-up display.

[0026] Map information 121 is stored in advance in the recording medium 12. The map information 121 is information used to identify the vehicle position and destination facilities, provide route guidance, etc., and includes node data indicating the positions of nodes set on the road on which the loaded vehicle travels, link data indicating the connections between nodes, and feature data indicating the positions of features existing on the road and its surroundings.

[0027] As another example, the driving lane estimation system 10 may receive map information of the surrounding area including the vehicle position via a network from a map server device that manages map information 121, and refer to this map information. In this case, the driving lane estimation system 10 may store map information of the surrounding area in a storage unit such as the recording medium 12, acquire map information of a new area according to the vehicle position after the vehicle has traveled, and update the map information stored in the storage unit with the newly acquired map information. In this case, it is not necessary for map information 121 of a wide range to be stored in advance on the recording medium 12. In this case, the driving lane estimation system 10 further includes a communication unit for communicating with the map server device via the network.

[0028] The recording medium 12 further stores an image recognition result information table 122, a frame information table 123, and an in-link position information table 124. The image recognition result information table 122 records image recognition result information. The image recognition result information is information indicating the image recognition result for each frame, and includes the frame number and the image recognition result obtained from the frame. The frame number is information for identifying the frame. In this embodiment, the frame numbers are consecutive numbers according to the order in which the images were captured. The image recognition result is information indicating the type and position of objects identified from the frame by image recognition, such as lane lines, road edges, road boundaries, and surrounding vehicles. Here, the position of an object is information indicating the position in the depth direction and lateral direction within the frame. The image recognition result information is stored in the recording medium when image recognition is performed by the control unit 11.

[0029] FIG. 4 is a diagram showing an example of the data configuration of the frame information table 123. The frame information table 123 stores frame information as records. The frame information is information indicating the image recognition results of the lane markings obtained in one frame. The frame information includes a frame number, elapsed time, vehicle range, and frame lane marking information. Here, the elapsed time is the time elapsed since the first frame was captured by the image capture unit 31. The vehicle range is a range that includes the vehicle position at the time the frame was captured. The vehicle range is a range along the link on which the vehicle is traveling, and is expressed as an intra-link position. If the vehicle position is located within the intra-link position range of 0 m to 1 m, the vehicle range is recorded as 1 m. Similarly, if the vehicle position is located within the range of (x-1) m to x m, the vehicle range is recorded as x m.

[0030] The frame demarcation line information indicates the image recognition results of the demarcation lines identified by image recognition in the depth range, for each partial section. If multiple demarcation lines are identified, the frame demarcation line information includes the image recognition results for each partial section of the multiple demarcation lines. In the example shown in FIG. 4, the frame demarcation line information includes 30 image recognition results, NLd10, NLd11, ..., NLd29, ELd10, ELd11, ..., ELd29, corresponding to depth distances of 10 m to 29 m for each of the two demarcation lines (NL and EL) on the left side of the vehicle. Furthermore, the frame demarcation line information includes 30 image recognition results, ERd10, ERd11...ERd29, NRd10, NRd11...NRd29, corresponding to depth distances of 10 m to 29 m for each of the two demarcation lines (ER and NR) on the right side of the vehicle. In other words, one frame demarcation line information includes 120 image recognition results.

[0031] Furthermore, the image recognition results for each of the multiple partial sections of the multiple section lines include the type, color, and horizontal distance of the lane marking. Here, the type is set to one of four values: solid line, dashed line, no lane marking, or unknown presence or absence of lane marking. In this embodiment, the lane marking type is indicated by "0" for solid line, "1" for dashed line, "2" for no lane marking, and "-1" for unknown presence or absence of lane marking. The color of the lane marking is set to one of two values: white or yellow. The horizontal distance of the lane marking is the horizontal distance from the shooting position to the lane marking.

[0032] FIG. 5 is a diagram showing an example of the data configuration of the intra-link position information table 124. In the intra-link position information table 124, intra-link position information is recorded as a record. In the intra-link position information table 124, recognition results of lane markings for each partial section indicated by an intra-link position are recorded in association with a vehicle range. The vehicle range is the same as the vehicle range in the frame information table 123. For example, the depth section in a frame captured in a vehicle range of an intra-link position of 1 m (referred to as a vehicle range 1 m) is the section from intra-link positions 11 m to 31 m. Therefore, for a vehicle range of 1 m, the image recognition results for the partial section from intra-link positions 11 m to 30 m obtained from this frame are recorded in association with the vehicle range 1 m in the intra-link position information table 124. Similarly, for a vehicle range of 5 m, image recognition results for the partial section from intra-link positions 15 m to 34 m are recorded in association with the vehicle range 5 m.

[0033] The image recognition result information table 122, frame information table 123, and in-link position information table 124 are information created for each link while the equipped vehicle is traveling. That is, while the equipped vehicle is traveling on link A, the image recognition result information table 122, frame information table 123, and in-link position information table 124 for link A are created. Then, when the equipped vehicle exits link A and enters link B, the image recognition result information table 122, frame information table 123, and in-link position information table 124 for link B are created.

[0034] The driving lane estimation program 110 executed by the control unit 11 is a program for causing the computer of the driving lane estimation system 10 to function as a vehicle position identification unit 111, a route guidance unit 112, an image recognition unit 113, a road edge identification unit 114, a driving lane estimation unit 115, and a lane guidance unit 116. In the following, the processes described as being performed by the vehicle position identification unit 111, the route guidance unit 112, the image recognition unit 113, the road edge identification unit 114, the driving lane estimation unit 115, and the lane guidance unit 116 are processes realized by the control unit 11 (CPU) executing the driving lane estimation program 110, i.e., processes executed by the control unit 11.

[0035] The vehicle position identifying unit 111 identifies the vehicle position. Specifically, the vehicle position identifying unit 111 obtains the vehicle position based on the output signal of the GNSS receiver 32, the output signal of the vehicle speed sensor 33, and the output signal of the gyro sensor 34, and further corrects the vehicle position by map matching processing using map information 121. The vehicle position identifying unit 111 further identifies the link that includes the vehicle position, and identifies the in-link position.

[0036] When the route guidance unit 112 receives an input of a destination via the user I / F unit 35, it searches for a route from the vehicle position to the destination by referring to the map information 121. Furthermore, when a travel route is set in response to a user operation, the route guidance unit 112 provides guidance on the travel route.

[0037] The image recognition unit 113 acquires frames captured by the photographing unit 31 and performs image recognition on the frames to identify lane lines, road boundaries, surrounding vehicles, etc. In image recognition, for example, images including objects to be identified, such as lane lines and road boundaries, are used as learning data, and a model obtained by machine learning such as deep learning is used. As another example, image features of lane lines, road boundaries, etc. may be stored in advance on the recording medium 12, and the image recognition unit 113 may identify lane lines and road boundaries using these features and features extracted from the frames.

[0038] The road edge identification unit 114 refers to the image recognition results of the road boundary line and the image recognition results of the lane markings, identifies the lane marking closest to the road boundary line, and identifies the end of the identified lane marking as the road edge.

[0039] The driving lane estimation unit 115 estimates the driving lane of the equipped vehicle on this link based on the image recognition results of the lane markings obtained in each of multiple frames captured while the vehicle is traveling on the same link. At this time, the driving lane estimation unit 115 estimates the number of the lane from the road edge based on the road edge identified by the road edge identification unit 114. The processing by the driving lane estimation unit 115 will be described in detail later with reference to FIG. 6 etc.

[0040] The lane guidance unit 116 provides guidance on a recommended lane in which the equipped vehicle should travel. For example, when the distance to an intersection where a right turn should be made is equal to or shorter than a predetermined distance, the lane guidance unit 116 provides guidance on the right lane as the recommended lane. For example, the lane guidance unit 116 may display a captured image on the display unit and cause the display unit to display AR content that superimposes the recommended lane in color on the captured image. Furthermore, the lane guidance unit 116 may also cause the display unit to display the driving lane estimated by the driving lane estimation unit 115 superimposed on a map image.

[0041] FIG. 6 is a flowchart showing the driving lane estimation process performed by the driving lane estimation system 10. FIG. 7 is an explanatory diagram of the driving lane estimation process. The driving lane estimation process is executed when the driving lane estimation system 10 is started. As shown in FIG. 6, in the driving lane estimation process, the control unit 11 of the driving lane estimation system 10 first starts capturing images using the imaging unit 31 and identifying the vehicle position using the vehicle position identification unit 111 (step S100). Thereafter, the imaging unit 31 captures images every 33 ms, and the vehicle position identification unit 111 identifies the vehicle position in accordance with the timing of the capture. The frames obtained by capturing images are input to the driving lane estimation system 10.

[0042] Next, the control unit 11 refers to the map information 121 and determines whether the vehicle position is on a link (step S102). For example, if the vehicle is located in a parking lot, it is determined that the vehicle is not on a link. If the vehicle position is not on a link, the control unit 11 waits (N in step S102). If the vehicle position is on a link (Y in step S102), the control unit 11 creates a frame information table 123 and an in-link position information table 124 for the link on which the vehicle is located (step S104). Note that, here, only a recording area for recording each table is reserved, and no records are recorded in the frame information table 123 and the in-link position information table 124. Next, the control unit 11 waits until a new frame is acquired by the photographing unit 31 (N in step S106), and when a new frame is acquired (Y in step S106), it performs a recording process (step S108).

[0043] 8 is a flowchart showing the recording process. The recording process is a process of recording the image recognition results for the target frame, with the frame acquired in step S106 being the target frame to be processed. In the recording process, first, the image recognition unit 113 performs image recognition on the target frame (step S200). This identifies lane lines, road boundaries, surrounding vehicles, etc. Furthermore, the road edge identification unit 114 identifies the road edges. Information on the identified lane lines, road boundaries, and road edges is recorded as the image recognition results in the image recognition result information table 122, associated with the frame number of the target frame. For lane lines, information indicating the type, color, and position of the lane line is included in the image recognition results.

[0044] Next, the travelling lane estimation unit 115 sets a depth section in the target frame (step S202). Specifically, the travelling lane estimation unit 115 sets a vertical range in the target frame corresponding to a depth distance of 10 m to 30 m as the depth section by referring to the positional relationship information. Next, the travelling lane estimation unit 115 divides the depth section into partial sections each having a length of 1 m in the depth direction by referring to the positional relationship information (step S204). Next, the travelling lane estimation unit 115 determines the lateral distance of the lane markings for each partial section based on the image recognition results for the target frame recorded in the image recognition result information table 122 (step S206). If multiple lane marks have been identified, the travelling lane estimation unit 115 determines the lateral distance of each of the multiple lane marks. As described above, the lateral distance of a lane marking is the lateral distance from the shooting position to the lane marking, and is a distance in real space. When determining the lateral distance, the travelling lane estimation unit 115 determines the distance in real space by referring to the positional relationship information.

[0045] Next, the driving lane estimation unit 115 records frame information about the target frame in the frame information table 123 (step S208). Specifically, the driving lane estimation unit 115 records the frame number, elapsed time, vehicle range, and frame dividing line information. Here, the frame number is assigned to the frame by the control unit 11 each time a frame is acquired. The elapsed time is the time elapsed since the image capturing unit 31 started capturing images, and is measured by a timer (not shown) under the control of the control unit 11. The vehicle range is identified from the vehicle position identified by the vehicle position identification unit 111 when the target frame was captured.

[0046] Next, the driving lane estimation unit 115 converts the depth distance of the partial section in the frame information into an intra-link position (step S210). Note that the value of the vehicle range is used as a reference value in the conversion to the intra-link position. As another example, the actually identified vehicle position may be used as a reference value instead of the value of the vehicle range in the conversion to the intra-link position. In this case, the intra-frame information includes the vehicle position in addition to the vehicle range. Furthermore, the converted value is expressed in units of 1 meter by rounding off fractions. As a result, each partial section is expressed as an intra-link position. For example, a partial section with a depth distance of 10 meters in a target frame captured at a point 1 meter in the intra-link position is converted into a partial section with an intra-link position of 11 meters.

[0047] Next, the driving lane estimation unit 115 performs a loop process of steps S212 to S218, processing all partial sections included in the depth section of the target frame. Hereinafter, the partial section to be processed is referred to as the target partial section. In the loop process, the driving lane estimation unit 115 first references the frame information of the target frame stored in the frame information table 123, and checks whether or not a lane marking has been identified in the target partial section (step S212). If a lane marking has not been identified (N in step S212), the driving lane estimation unit 115 ends the loop process for the target partial section.

[0048] If at least one lane marking has been identified (Y in step S212), the driving lane estimation unit 115 records the image recognition result of the lane marking in the target partial section in the in-link position information table 124 (step S214). For example, if an image recognition result of NL is obtained in a partial section with a depth distance of 1 m (intra-link position 11 m) in the target frame obtained at a point with a vehicle range of 1 m, the image recognition result is recorded in NLk11 in the first row of the in-link position information table 124 shown in Fig. 5. If multiple lane marks have been identified in one partial section, the recognition result of each lane marking is recorded.

[0049] Next, the driving lane estimation unit 115 compares the number of records corresponding to the same vehicle range recorded in the in-link position information table 124 with a preset record threshold (step S216). If the number of records corresponding to the same vehicle range is equal to or greater than the record threshold (Y in step S216), the driving lane estimation unit 115 deletes the oldest record (step S218). This ends the loop processing. On the other hand, if the number of records corresponding to the same vehicle is less than the record threshold (N in step S216), the driving lane estimation unit 115 ends the loop processing without performing step S218.

[0050] Here, the processing of steps S216 and S218 will be described. As will be described later, the recording processing is executed each time the driving lane estimation system 10 acquires a frame. Therefore, as more frames are acquired, the number of records recorded in the in-link position information table 124 increases. For example, in the example of the frame information table 123 shown in FIG. 4, six frames, frame numbers 1 to 6, are associated with a vehicle range of 1 m. If lane markings are identified in all of these frames, six image recognition results associated with a vehicle range of 1 m are recorded in the in-link position information table 124. When the equipped vehicle is stopped or traveling at a low speed due to traffic congestion, the number of records indicating the same vehicle range increases, which may result in excessive memory capacity. Therefore, in this embodiment, as described above, the number of image recognition results corresponding to the same vehicle range that can be stored is limited. This prevents the memory capacity used by the in-link position information table 124 from becoming too large.

[0051] In FIG. 6, after the recording process (step S108), the driving lane estimation unit 115 determines whether the travel distance of the equipped vehicle is equal to or greater than a first distance (step S110). Here, the travel distance is the distance traveled by the equipped vehicle from a first reference timing to the time of processing step S110. Here, the first reference timing is the timing at which the driving lane estimation process starts, if the processing of step S110 is performed for the first time. Furthermore, if the processing of step S110 has already been performed after the start of the driving lane estimation process and it has been determined that the distance is equal to or greater than the first distance, the first reference timing is the timing at which it was determined that the distance was equal to or greater than the first distance immediately before in step S110. The travel distance can be obtained from the vehicle position history. The first distance is a preset distance, and in this embodiment, it is set to 10 m.

[0052] If the driving lane estimation unit 115 has not traveled the first distance or more (N in step S110), it proceeds to step S116 without performing processing related to driving lane estimation. In step S116, the driving lane estimation unit 115 determines whether the vehicle position is the next link. Here, the next link is a link connected to the link on which the vehicle was traveling when the target frame was acquired. For example, assume that the vehicle was traveling on link A at the time of processing the immediately preceding step S106. In this case, if the vehicle has already left link A and entered link B at the time of processing step S116, it is determined that the vehicle position is the next link. On the other hand, if the vehicle is traveling on link A at the time of processing step S116, it is determined that the vehicle position is not the next link.

[0053] If the vehicle position is not the next link (N in step S116), the driving lane estimation unit 115 proceeds to step S106. In this case, a recording process (step S108) is performed on the newly acquired frame. By repeating the recording process in this manner, frame information is accumulated in the frame information table 123, and image recognition results indicated by intra-link positions are accumulated in the intra-link position information table 124.

[0054] For example, as shown in Figure 7 (1), image recognition results for lane markings are obtained at a position of 0 m within the link. Then, as shown in (2), image recognition results for lane markings are obtained from a frame captured when the vehicle has advanced a little further within the link. In this way, image recognition results for different positions within the link are accumulated as the vehicle travels.

[0055] In step S116, if the vehicle position is the next link (Y in step S116), the driving lane estimation unit 115 proceeds to step S104. In this case, a new frame information table 123 and in-link position information table 124 are created, and in the subsequent processing, data is recorded in the newly created tables.

[0056] In step S110, if the equipped vehicle has traveled a first distance or more (Y in step S110), the driving lane estimation unit 115 proceeds to step S112. As shown in (3) of FIG. 7, when the equipped vehicle has traveled a first distance or more, the position within the link corresponding to the depth section obtained from the captured frame also moves by 10 m. Therefore, as shown in (1) to (3) of FIG. 7, when the equipped vehicle moves from a position 0 m to 10 m within the link, image recognition results of the lane lines from a position 10 m to 40 m within the link are obtained. In step S112, the driving lane estimation unit 115 performs lane line identification processing based on these image recognition results. The lane line identification processing is processing for identifying lane lines based on image recognition results at each of a plurality of different positions within the link.

[0057] 9 is a flowchart showing detailed processing in the lane marking identification process. In the lane marking identification process, the driving lane estimation unit 115 first sets a target section based on the vehicle range (step S300). Here, the target section is a section with a preset distance in the depth direction. In this embodiment, the target section is set to a section with a depth distance of 1 m to 10 m based on the vehicle range. As a result, for example, as shown in (3) in FIG. 7, if the vehicle range is within a range of 10 m within the link, a section 300 from 11 m to 20 m within the link is set as the target section.

[0058] Next, the driving lane estimation unit 115 references the intra-link position information table 124 and finds the most frequent value of the image recognition results (type, color, and lateral distance) of the lane markings in each partial section included in the target section (step S302). This obtains the most frequent value of the image recognition results of the lane markings in each of the 10 partial sections from intra-link positions 11 m to 20 m. In the intra-link position information table 124, the image recognition results obtained from multiple frames captured in the same vehicle range and multiple frames captured in different vehicle ranges are converted into intra-link positions and recorded. Therefore, the driving lane estimation unit 115 can find the most frequent value of the image recognition results for frames captured in different vehicle ranges and frames captured in the same vehicle range.

[0059] Next, the travelling lane estimation unit 115 determines the most frequent value of the image recognition results of the lane markings in the target section as the image recognition result of the lane markings at the vehicle position at this time (step S304). Specifically, the travelling lane estimation unit 115 determines the most frequent value of the multiple most frequent values obtained in each subsection within the target section as the most frequent value for the target section. In this way, the travelling lane estimation unit 115 determines the most frequent value in two stages.

[0060] For example, in the example of (3) in Figure 7, four lane markings, NL, EL, ER, and NR, are identified, and their image recognition results (modes) are obtained. In the example of Figure 7, the types of lane markings NL, EL, ER, and NR are solid line (0), dashed line (1), dashed line (1), and dashed line (1), respectively. All are white in color. The lateral distances of the lane markings NL, EL, ER, and NR are "-4.5 (m)," "-1.4 (m)," "1.4 (m)," and "4.5 (m)," respectively. Note that the lateral distances are expressed as values with the vehicle position as the origin, with the right side being the positive direction and the left side being the negative direction. These image recognition results are identified as the image recognition results for lane markings at a vehicle position 10 m within the link.

[0061] As shown in FIG. 6, upon completion of the lane marking identification process (step S112), the driving lane estimation unit 115 estimates the driving lane based on the image recognition results of the lane markings obtained by the lane marking identification process (step S114). Specifically, the driving lane estimation unit 115 determines the number of the driving lane from the road edge based on the lateral position of the lane markings identified by image recognition and the lateral position of the road edge obtained by image recognition. Note that, in estimating the driving lane, the driving lane estimation unit 115 only needs to identify which lane on the road the driving lane is, and the lane identification method is not limited to that described in the embodiment. Next, the driving lane estimation unit 115 proceeds to step S116 and repeats the process. Note that the driving lane estimation process ends when the driving lane estimation system 10 is powered off. Alternatively, the process may end in response to a user operation.

[0062] As described above, the driving lane estimation system 10 of this embodiment estimates the driving lane of a vehicle (equipped vehicle) based on the image recognition results of the lane markings in each of a plurality of areas in a captured image. This increases the reliability of the driving lane estimation.

[0063] As described above, the driving lane estimation unit 115 identifies the type, color, and lateral distance of the lane markings at the vehicle position based on the image recognition results of the lane markings in multiple partial sections with different depth distances. Here, the multiple partial sections with different depth distances correspond to areas with different positions in the road length direction, which is the direction in which the road on which the vehicle is traveling extends. In other words, the driving lane estimation unit 115 estimates the driving lane of the vehicle (equipped vehicle) based on the image recognition results of the lane markings in each of multiple areas with different positions in the road length direction. In this way, by referencing the image recognition results in multiple areas in the road length direction, the reliability of driving lane estimation can be increased.

[0064] As described above, the driving lane estimation unit 115 estimates the driving lane based on the image recognition results recorded in the intra-link position information table 124. Here, the intra-link position information table 124 stores image recognition results for each of a plurality of captured images (frames) taken at each of a plurality of different intra-link positions (a plurality of different positions in the road length direction). The driving lane estimation unit 115 estimates the driving lane based on the image recognition results for each of a plurality of frames taken at each of a plurality of different positions in the road length direction. In this way, the driving lane estimation unit 115 references frames taken at different capture positions. Therefore, even if a lane marking cannot be identified in a frame taken at a certain capture position, the driving lane estimation unit 15 can estimate the driving lane by using information about the lane marking identified in a frame taken at another capture position.

[0065] As described above, the intra-link position information table 124 stores image recognition results from multiple frames corresponding to one vehicle range that is 1 meter wide in the depth direction. The driving lane estimation unit 115 then performs driving lane estimation based on the image recognition results for each of the multiple frames captured within one vehicle range. In this way, by referencing multiple frames captured within one vehicle range, even if a lane marking cannot be identified in a frame captured at a certain time due to an obstruction caused by a vehicle ahead, the lane marking may be identified from a frame captured at another time. In this way, the probability of identifying a lane marking at a certain point can be increased. Note that the vehicle range may be a range smaller than the depth range and is not limited to a range of 1 meter.

[0066] As described above, the driving lane estimation unit 115 estimates the driving lane based on the recognition results of each partial section included in the target section. This increases the likelihood of estimating the correct driving lane, even when, for example, the vehicle is stopped for a long time at a point where lane markings cannot be identified. In conventional technology, an onboard camera captures multiple frames at regular intervals, and the driving lane is estimated based on the image recognition results of the multiple frames. However, in this case, lane markings may not be included in the captured image due to reasons such as being obscured by a vehicle ahead. If the vehicle remains stopped for a long time in this state, frames in which lane markings cannot be identified accumulate. In this case, only frames in which lane markings cannot be identified are referenced in the driving lane estimation, resulting in the determination that the driving lane is unknown and an estimation result cannot be obtained. Similarly, when traveling at a low speed in a section where lane markings are not included in the captured image due to congestion, etc., frames in which lane markings cannot be identified accumulate, making it impossible to obtain an estimation result for the driving lane. In contrast, as described above, the present embodiment increases the reliability of driving lane estimation by using the recognition results of a specified section (target section).

[0067] (Second embodiment) Next, a description will be given of a driving lane estimation system 10 according to a second embodiment, focusing on differences from the driving lane estimation system 10 according to the first embodiment. The driving lane estimation unit 115 of the driving lane estimation system 10 according to the second embodiment creates vehicle range information for each vehicle range from frame delimiting line information recorded in the frame information table 123 as intermediate data. Here, the vehicle range information is information indicating a representative value of the frame delimiting line information for the vehicle range. In this embodiment, the driving lane estimation unit 115 uses the most frequent value of each value of the frame delimiting line information associated with the same vehicle range as the representative value. The driving lane estimation unit 115 estimates the driving lane using the vehicle range information. This process will be described in detail later with reference to FIG. 12, etc.

[0068] In the driving lane estimation system 10 according to the second embodiment, a vehicle range information table is stored in the recording medium 12 in addition to map information 121, an image recognition result information table 122, and a frame information table 123. In the second embodiment, the data configuration of the in-link position information table is different from that of the in-link position information table 124 according to the first embodiment.

[0069] FIG. 10 is a diagram showing an example of the data configuration of a vehicle range information table 500 according to the second embodiment. Vehicle range information is stored as records in the vehicle range information table 500. The vehicle range information is information indicating the most frequent value in frame lane marking information corresponding to the same vehicle range. The vehicle range information includes a vehicle range and vehicle range lane marking information. For example, as shown in FIG. 4, assume that six pieces of frame lane marking information are stored for a vehicle range at a position 1 m within a link. In this case, the most frequent value of the six pieces of frame lane marking information is created as vehicle range lane marking information for the vehicle range at a position 1 m within a link and recorded in the vehicle range information table 500. For example, the most frequent value of NL10 for each of frame numbers 1 to 6 is recorded as NLd10 in the vehicle range lane marking information.

[0070] FIG. 11 is a diagram showing an example of the data configuration of an in-link position information table 510 according to the second embodiment. In-link position information is recorded in the in-link position information table 510 according to the second embodiment. Note that the in-link position information in this embodiment is information that associates a vehicle range with in-link lane line information. The in-link vehicle position information does not include a frame number. In this embodiment, one piece of in-link lane line information is recorded for one vehicle position. The in-link lane line information is similar to the in-link lane line information in the first embodiment, and is information that indicates the image recognition results of the lane lines in each partial section.

[0071] FIG. 12 is a flowchart showing the recording process executed by the driving lane estimation system 10 according to the second embodiment. The processes of steps S400 to S408 of the recording process are the same as the processes of steps S200 to S208 of the recording process according to the first embodiment shown in FIG. 6. In the second embodiment, after the process of step S408, the driving lane estimation unit 115 determines whether the travel distance of the equipped vehicle is equal to or greater than a second distance (step S410). Here, the travel distance is the distance traveled by the equipped vehicle from the second reference timing to the time of the process of step S410. Here, the second reference timing is the timing at which the driving lane estimation process starts if the process of step S410 is performed for the first time. Furthermore, if the process of step S410 has already been performed after the start of the driving lane estimation process and it has been determined that the distance is equal to or greater than the second distance, the second reference timing is the timing at which it was determined that the distance was equal to or greater than the second distance in step S410 immediately before. The second distance is a preset distance, and is set to 1 m in this embodiment.

[0072] If the vehicle has not traveled the second distance or more (N in step S410), the driving lane estimation unit 115 ends the recording process without recording a record in the vehicle range information table 500 or the intra-link position information table 510. If the vehicle has traveled the second distance or more (Y in step S410), the driving lane estimation unit 115 creates vehicle range information (step S412). If the vehicle has traveled the second distance or more, frames captured at intra-link positions within the second distance (1 m in the depth direction) during this period are recorded in the frame information table 123. The driving lane estimation unit 115 determines the most frequent occurrence of the frame lane marking information obtained from these frames as a representative value, and creates vehicle range information that associates the vehicle range lane marking line information indicating the most frequent value with the vehicle range.

[0073] Next, the driving lane estimation unit 115 records the vehicle range information created in step S412 in the vehicle range information table 500 (step S414). For example, when the equipped vehicle moves from an in-link position of 0 m to 1 m, it is determined in step S410 that it has traveled a second distance. If frame information for frame numbers 1 to 6 shown in FIG. 4 is recorded in the frame information table 123 during this time, vehicle range information for the vehicle range at an in-link position of 1 m is created from this frame information, as shown in FIG. 10. As described above, the vehicle range information is information that indicates the most frequent value in the frame lane line information. Thus, in this embodiment, the driving lane estimation unit 115 calculates the most frequent value of the recognition results of the lane line in a 1-m section every time the vehicle range at the time of image capture moves 1 m, and records this as intermediate data in the vehicle range information table 500.

[0074] Next, the driving lane estimation unit 115 converts the depth distance of the partial section in the vehicle range information into an intra-link position (step S416). Next, the driving lane estimation unit 115 checks whether the vehicle range information stored in the vehicle range information table 500 includes a recognition result indicating the presence of lane markings (step S418). If the recognition result indicating the presence of lane markings is not included (N in step S418), the driving lane estimation unit 115 ends the recording process. If the recognition result indicating the presence of lane markings is included (Y in step S418), the driving lane estimation unit 115 records intra-link position information corresponding to the vehicle range information in the intra-link position information table 510 (step S420). The intra-link position information is information corresponding to the vehicle range information. However, in the intra-link position information, the partial section is indicated by an intra-link position.

[0075] In the second embodiment, one piece of vehicle range information is generated for a predetermined vehicle range (a section of 1 m within a link), and the in-link position information corresponding to this one piece of vehicle range information is stored in the in-link position information table 510. Therefore, for one vehicle range, only one piece of in-link position information indicating the most frequent value of the recognition results of the lane markings in that vehicle range is recorded in the in-link position information table 510.

[0076] Other configurations and processes of the driving lane estimating system 10 according to the second embodiment are similar to those of the driving lane estimating system 10 according to the first embodiment.

[0077] As described above, in the second embodiment, the most frequent value of the recognition results of the lane markings for each vehicle range is calculated as intermediate data, and the driving lane is estimated based on this. In this way, by using the most frequent value, the reliability of the driving lane estimation can be improved.

[0078] (Other embodiments) The above embodiment is an example for implementing the present invention, and various other embodiments are also possible. For example, at least some of the components constituting the driving lane estimation system 10 may be separated into multiple devices or systems. That is, at least some of the components constituting the driving lane estimation system 10, including the vehicle position identification unit 111, route guidance unit 112, image recognition unit 113, road edge identification unit 114, driving lane estimation unit 115, and lane guidance unit 116, may be separated into multiple devices. For example, the image recognition unit 113 may be provided in another device. Furthermore, some components of the above embodiment may be omitted, and the order of processing may be changed or omitted.

[0079] In this embodiment, the frame (captured image) used for driving lane estimation is an image with a capture range in front of the vehicle, but the captured image used for driving lane estimation may be an image with the direction of the road on which the vehicle is traveling as the depth direction. In other words, the captured image used for driving lane estimation may be an image with a capture range behind the vehicle. In this case, the image capturing unit 31 is fixed so as to capture an image behind the vehicle.

[0080] The driving lane estimation unit only needs to refer to the image recognition results of the lane markings in each of a plurality of regions that are located at different positions in the road length direction, and the positions and depth widths of the plurality of regions in this case are not limited to those described in the embodiment. In other words, the regions referred to here do not need to be partial sections divided at regular intervals.

[0081] The driving lane estimation unit estimates the driving lane based on the most frequent value of the lane marking recognition results, but the value is not limited to the most frequent value and can use a statistical value of multiple recognition results. Here, the statistical value is a value that takes into account multiple recognition results, and for lateral distance, it can include the average value, median value, etc.

[0082] The driving lane estimation unit may estimate the driving lane based on image recognition results of lane markings in multiple captured images taken within a predetermined range in the road length direction, such as a vehicle range of 1 meter within the link. In other words, the driving lane estimation unit does not need to refer to image recognition results in captured images of different ranges when estimating the driving lane. In this case, referring to multiple captured images within the same range can improve the performance of driving lane estimation compared to referring to only one captured image.

[0083] Furthermore, the present invention can also be applied as a program or method. The above-described systems, programs, and methods may be realized as standalone devices or may be realized using components shared with various vehicle components, and thus encompass a variety of embodiments. For example, it is possible to provide a method or program realized by the above-described system. Furthermore, the invention can be implemented as a recording medium for a program that controls the device. Of course, the software recording medium may be a magnetic recording medium or a semiconductor memory, and any recording medium developed in the future can be considered in the same manner. [Explanation of symbols]

[0084] 10...driving lane estimation system, 11...control unit, 12...recording medium, 31...photographing unit, 32...GNSS receiving unit, 33...vehicle speed sensor, 34...gyro sensor, 35...user I / F unit, 111...vehicle position identification unit, 112...route guidance unit, 113...image recognition unit, 114...road edge identification unit, 11...5 driving lane estimation unit, 116...lane guidance unit, 121...map information, 122...image recognition result information table, 123...frame information table, 124, 510...intra-link position information table, 500...vehicle range information table

Claims

1. an image recognition unit that identifies lane markings by image recognition in an image of a road on which the vehicle is traveling, the image being captured by an image capture unit mounted on the vehicle; a driving lane estimation unit that estimates a driving lane of the vehicle based on image recognition results of the lane markings in each of a plurality of regions in the captured image that are positioned differently in a road length direction, which is the direction in which the road extends; Equipped with The driving lane estimation unit setting a depth section in the captured image corresponding to a predetermined range in a depth direction in real space; Dividing the depth section into a plurality of subsections in a depth direction; a depth distance in a depth direction based on the position of the image capturing unit in real space, and converting the depth distance of the partial section into an intra-link position that is a position within a link of map information during which the vehicle is traveling; the image recognition result for each of the partial sections is associated with the position within the link of the partial section, and recorded on a recording medium; estimating a driving lane of the vehicle based on a plurality of the image recognition results for the partial sections at the same intra-link position and the image recognition results for a plurality of the partial sections at different intra-link positions, which are recorded on the recording medium; Driving lane estimation system.

2. the image recognition and the recording of the image recognition results for each of the partial sections corresponding to the intra-link positions are performed for each of a plurality of the photographed images taken at a plurality of different positions in the road length direction, 2. The driving lane estimation system according to claim 1, wherein the driving lane estimation unit estimates the driving lane of the vehicle based on a plurality of image recognition results for the partial section at the same intra-link position, the image recognition results being taken at different positions in the road length direction.

3. A predetermined range along the link that includes the position of the vehicle at the time the photographed image was taken is defined as a vehicle range, the image recognition and the recording of the image recognition results for each of the partial sections corresponding to the in-link positions are performed for each of the plurality of captured images taken within the same vehicle range; 3. The driving lane estimation system according to claim 1, wherein the driving lane estimation unit estimates the driving lane of the vehicle based on a plurality of image recognition results for the partial section at the same intra-link position, the image recognition results being for the same vehicle range.

4. The driving lane estimation unit calculating a statistical value of the image recognition result for each of the partial sections corresponding to the intra-link position by referring to the recording medium; The driving lane estimation system according to claim 1 , wherein the driving lane of the vehicle is estimated based on the statistical value for each of the partial sections.

5. The driving lane estimation system according to claim 1 , wherein the image recognition result includes a type of the lane marking.

6. The driving lane estimation unit A target section is set as a section of a predetermined distance in the depth direction, determining a mode of the image recognition results in each of the partial sections of each of the intra-link positions included in the target section; The most frequent value of the plurality of most frequent values obtained in each of the partial intervals within the target interval is determined as the most frequent value in the target interval; estimating a driving lane of the vehicle based on the mode value of the target section; The driving lane estimation system according to any one of claims 1 to 5.

Images