A lane line tracking method based on template matching, medium and device

Abstract

An embodiment of the present application discloses a lane line tracking method based on template matching, a medium and equipment, the method comprises the following steps: S1: acquiring each frame image output by a vehicle-mounted front camera, and initializing a lane line template according to a current vehicle speed; S2: matching a plurality of lane lines detected according to a current frame image with a lane line template constructed in advance; and S3: classifying the matching result and performing corresponding processing. The present application can improve the robustness of overall lane line detection when the lane line detection result is unstable, that is, when the lane line is interrupted, shielded, in a large-angle curve, and complex.

Description

Technical Field

[0001] This invention relates to the field of image processing. More specifically, it relates to a lane line tracking method, medium, and apparatus based on template matching. Background Technology

[0002] Lane tracking is a crucial function of the perception module in autonomous driving technology, playing a vital role in the operation of autonomous vehicles. Functional modules such as LDW (Lane Departure Warning) and LKA (Lane Keeping Assist) rely on continuous and stable lane detection and tracking. Lane tracking utilizes the continuity of image acquisition time, predicting the lane line in the current frame based on lane lines detected in previous frames, then fusing the lane line detection results of the current frame to finally output the most probable lane line for the current frame.

[0003] Currently, common lane tracking methods include those based on Kalman filtering and its variations. These methods first extract lane lines from image features using traditional image processing algorithms, then match the lane lines from the previous and current frames, and then activate Kalman filters to track the lane line parameters. Finally, they output the new tracked lane line parameters. However, directly tracking lane line parameters presents problems. Kalman filtering is based on the assumption that noise follows a Gaussian distribution, but the noise distribution of lane line parameters usually does not conform to this. Therefore, existing lane tracking algorithms have some shortcomings, especially in non-ideal situations, such as when lane line images are interrupted, occluded, or blurred, or when lane lines are complex, such as at sharp curves, entrances / exits, and intersections. Existing lane tracking algorithms have poor robustness and struggle to consistently provide good lane tracking results, leading to lane line misses and false detections, which pose a danger to subsequent autonomous driving functions. Summary of the Invention

[0004] In view of this, the first embodiment of the present invention provides a lane line tracking method based on template matching, comprising:

[0005] S1: Acquire each frame of image output by the vehicle's forward-facing camera and initialize the lane line template based on the current vehicle speed;

[0006] S2: Match the multiple lane lines detected based on the current frame image with the pre-built lane line template;

[0007] S3: Classify the matching results and perform corresponding processing.

[0008] In one specific embodiment, initializing the lane line template based on the current vehicle speed includes:

[0009] Determine whether the current vehicle speed exceeds a preset vehicle speed threshold. If it does not exceed the threshold, clear the lane line template. Otherwise, process each frame of the image output by the vehicle-mounted forward camera to obtain multiple lane lines detected based on the current frame image.

[0010] Determine if the pre-built lane line template is zero. If it is, build the lane line template. Otherwise, match the multiple lane lines detected in the current frame image with the pre-built lane line template.

[0011] In one specific embodiment, matching multiple lane lines detected based on the current frame image with a pre-built lane line template includes:

[0012] Each lane line is matched with a sub-template in a pre-built lane line template to obtain the sub-template with the highest score for each lane line.

[0013] Determine whether the score of the sub-template exceeds the matching score threshold. If it does, the matching is considered successful; otherwise, the matching fails.

[0014] In one specific embodiment, the matching results are divided into lane lines and their sub-templates that are successfully matched, sub-templates of lane lines that are not matched, and lane lines that are not matched with sub-templates.

[0015] In one specific embodiment, when the matching result is a successfully matched lane line and its sub-template, the tracking time corresponding to the sub-template is set to time = time + 1, and the content of its corresponding other sub-templates is updated.

[0016] The line with the longest successful template tracking time and the highest lane line point probability score is added to the candidate lane line list.

[0017] The distance between the remaining successfully matched lane lines and the lane lines in the candidate lane line list is calculated sequentially. If the distance is greater than a preset distance threshold, the remaining successfully matched lane lines are also added to the candidate lane line list. Otherwise, the tracking time difference between the lane lines in the candidate lane line list and the remaining successfully matched lane lines is calculated. Based on the tracking time difference and the comparison between the lane lines in the candidate lane line list and the remaining successfully matched lane lines, it is determined which lane line to add or retain to the candidate lane line list.

[0018] In one specific embodiment, when the matching result is that no lane line sub-template is matched, if the probability value of the semantic segmentation result in the current frame sub-template exceeds the preset score threshold, and lane lines have appeared at this position in the past period of time, then the lane line in the current frame is considered to be missed, and a lane line is added to the candidate lane line.

[0019] In one specific embodiment, when the matching result is that no lane line is matched with a sub-template, the distance between the unmatched lane line and all sub-templates is calculated. If the distance is greater than a preset distance threshold, it is considered a newly detected lane line and added to the candidate lane lines. Otherwise, the tracking time corresponding to the sub-template closest to the unmatched lane line is determined to be greater than a preset time threshold. If it is, the lane line is determined to be a false detection lane line; otherwise, the sub-template is determined to be incorrect.

[0020] In one specific embodiment, processing each frame of image output by the vehicle-mounted forward-facing camera includes:

[0021] Each pixel of each frame of image output by the vehicle-mounted forward-facing camera is classified into two categories, and each pixel is given the probability of whether it is a point on the lane line.

[0022] Clustering or fitting these points yields a parametric representation of the lane lines.

[0023] A second embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon, characterized in that the program, when executed by a processor, implements the method described in the first embodiment.

[0024] A third embodiment of the present invention provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor executes the program to implement the method described in the first embodiment.

[0025] The beneficial effects of this invention are as follows:

[0026] This invention provides a lane line tracking method, medium, and device based on template matching. The proposed lane line tracking method improves the robustness of overall lane line detection when lane line detection results are unstable, i.e., when lane lines are interrupted, occluded, or there are sharp curves or complex lane lines. Even if there are several frames of unstable output during lane line processing, i.e., some lane lines are missed or misidentified, the proposed tracking method can compensate for these errors and still output relatively continuous lane lines in time and space through template-based tracking prediction. Actual road testing shows that the proposed lane line tracking method can improve the overall lane line detection accuracy by 10%. Attached Figure Description

[0027] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0028] Figure 1 A schematic flowchart of a lane line tracking method based on template matching according to an embodiment of the present invention is shown;

[0029] Figure 2 A flowchart illustrating a successful match of a lane line and its sub-templates according to an embodiment of the present invention is shown.

[0030] Figure 3 This diagram illustrates a process when the matching result is that no sub-template for the lane line is matched, according to an embodiment of the present invention.

[0031] Figure 4 This diagram illustrates a process when the matching result is that no lane line of the sub-template is matched, according to an embodiment of the present invention.

[0032] Figure 5 A schematic diagram of the structure of a computer device according to another embodiment of the present invention is shown. Detailed Implementation

[0033] To make the present invention, its technical solutions, and advantages clearer, the embodiments of the present invention will be described in further detail below with reference to the accompanying drawings.

[0034] like Figure 1 As shown, one embodiment of the present invention provides a lane line tracking method based on template matching, including:

[0035] S1: Acquire each frame of image output by the vehicle's forward-facing camera and initialize the lane line template based on the current vehicle speed;

[0036] In one specific embodiment, initializing the lane line template based on the current vehicle speed includes:

[0037] Determine whether the current vehicle speed exceeds a preset vehicle speed threshold. If it does not exceed the threshold, clear the lane line template. Otherwise, process each frame of the image output by the vehicle-mounted forward camera to obtain multiple lane lines detected based on the current frame image.

[0038] Determine if the pre-built lane line template is zero. If it is, build the lane line template. Otherwise, match the multiple lane lines detected in the current frame image with the pre-built lane line template.

[0039] In this embodiment, the system first determines whether lane lines need to be output based on the current vehicle speed. When the vehicle speed is low, there is no need to output lane lines, and accumulating lane line templates is of little value, so the lane line templates are cleared. When the vehicle speed meets the threshold and the number of templates is zero, lane line templates are started. The lane line templates contain the coordinates, width, curvature, and other features of multiple lane lines on the road surface, which are used for subsequent matching and tracking.

[0040] The lane line template differs significantly from existing technologies, containing not only lane line parameter information but also a wealth of data for evaluating lane line quality. Real-world roads typically contain two or more lane lines, and a single lane line template can contain multiple sub-lane line templates. Each sub-template includes: the coordinates of the lane line points in the forward image, the coordinates of the lane line points after inverse perspective transformation, the semantic segmentation probability value of the lane line points, the second-order fitting parameters of the lane line in the forward image, the first-order fitting parameters after inverse perspective transformation, the second-order fitting parameters after inverse perspective transformation, whether it is a left lane marker, whether it is a straight lane marker, the tracking time, the maximum curvature of the lane line, the template width, the longest tracking time threshold, and the lane line-template matching score threshold, among other things.

[0041] In one specific embodiment, processing each frame of image output by the vehicle-mounted forward-facing camera includes:

[0042] Each pixel of each frame of image output by the vehicle-mounted forward-facing camera is classified into two categories, and each pixel is given the probability of whether it is a point on the lane line.

[0043] Clustering or fitting these points yields a parametric representation of the lane lines.

[0044] In this embodiment, each pixel of each frame of image output by the vehicle-mounted forward-facing camera is classified into two categories. Each pixel is given the probability of whether it is a point on the lane line. These points are clustered or fitted to obtain a parameterized representation of the lane line, such as a quadratic curve, cubic curve, spline curve, etc., so as to facilitate subsequent matching with a pre-constructed lane line template.

[0045] S2: Match the multiple lane lines detected based on the current frame image with the pre-built lane line template;

[0046] In one specific embodiment, matching multiple lane lines detected based on the current frame image with a pre-built lane line template includes:

[0047] Each lane line is matched with a sub-template in a pre-built lane line template to obtain the sub-template with the highest score for each lane line.

[0048] Determine whether the score of the sub-template exceeds the matching score threshold. If it does, the matching is considered successful; otherwise, the matching fails.

[0049] In this embodiment, lane lines after semantic segmentation and post-processing may experience false positives and false negatives. For example, a lane line might be missed, misidentified as two, or a damaged section of the road surface might be mistaken for a lane line. In such cases, relying solely on the detection results of the current frame can easily lead to incorrect lane line results. It is necessary to consider historical lane lines from a short period of time. Even with continuous vehicle movement, lane lines maintain a certain spatial continuity; the interval between two consecutive frames is typically less than 0.05 seconds, and the position of the lane lines does not change significantly. The lane line template represents historical lane line information. Therefore, template matching is performed based on the positional relationship between lane line points and the template. By finding the correspondence between the current frame's lane lines and historical lane lines through template matching, the error in lane line results caused by detection errors in a few frames can be effectively eliminated.

[0050] The specific implementation process of the lane line matching module with the pre-built lane line template is as follows: If a new frame image is acquired, the vehicle speed meets the threshold, and the template is not empty, the lane line matching operation is initiated. Multiple lane lines are detected in each frame image. Each lane line is matched pairwise with a sub-template in the lane line template. The sub-template with the highest score for each lane line is identified. If the score exceeds the matching score threshold, the match is considered successful. The matching score is the ratio of the number of lane line points detected in the current frame that are within the sub-template range to the total number of points on that lane line. The distance from the point to the sub-template line and the template width can be calculated to determine whether a lane line point is within the sub-template range.

[0051] Therefore, in a specific embodiment, through the above matching process, the matching result is divided into lane lines and their sub-templates that are successfully matched, sub-templates of lane lines that are not matched, and lane lines that are not matched with sub-templates.

[0052] S3: Classify the matching results and perform corresponding processing.

[0053] In one specific embodiment, such as Figure 2 As shown, when the matching result is a successfully matched lane line and its sub-template, the tracking time corresponding to the sub-template is set to time = time + 1, and the content of its other corresponding sub-templates is updated.

[0054] The line with the longest successful template tracking time and the highest lane line point probability score is added to the candidate lane line list.

[0055] The distance between each of the remaining successfully matched lane lines (denoted as line2) and the lane lines in the candidate lane line list (denoted as line1) is calculated sequentially. If the distance is greater than a preset distance threshold K_dis, the remaining successfully matched lane lines are also added to the candidate lane line list; otherwise, the tracking time difference line1.time - line2.time between the lane lines in the candidate lane line list and the remaining successfully matched lane lines is calculated. Based on the tracking time difference and a comparison between the lane lines in the candidate lane line list and the remaining successfully matched lane lines, a decision is made on which lane line to add or retain to the candidate lane line list. The tracking time difference is then used to determine the appropriate lane line to add or retain to the candidate lane line list, and the tracking time difference is calculated for the sub-template corresponding to the other lane line. The tracking time is set to time = time - 4. For example, the tracking time difference line1.time - line2.time between the lane lines in the candidate lane line list and the other successfully matched lane lines is calculated, and it is determined whether the tracking time difference exceeds the preset time threshold K_time. If it does not exceed the threshold, the lane lines in the candidate lane line list are saved, and the other successfully matched lane lines and their corresponding sub-templates are deleted. Otherwise, the lane lines in the candidate lane line list are compared with the other successfully matched lane lines, the line with the higher score is selected and added to the candidate lane line list, and the tracking time corresponding to the sub-template of the other lane line is set to time = time - 4.

[0056] like Figure 3 As shown, when the matching result is that no lane line is matched in the sub-template, if the probability value of the semantic segmentation result in the current frame sub-template exceeds the preset score threshold K_prob, and lane lines have appeared at this position in the past period of time, then the lane line in the current frame is considered to be missed, and a lane line is added to the candidate lane line.

[0057] In this embodiment, it is necessary to determine whether to add candidate lane lines based on the semantic segmentation result of the current frame and the attribute values ​​of the sub-template. If the probability value of the semantic segmentation result within the current frame sub-template exceeds the threshold K_prob, and lane lines have appeared at this position consistently over a period of time, then the lane line in the current frame is considered a missed detection, and a lane line is added to the candidate lane lines based on the sub-template. However, it is still necessary to penalize the lane line template to reduce the attribute value tracking time. The specific reduction value depends on the probability value to prevent lane lines from disappearing due to changes in road conditions or other reasons.

[0058] like Figure 4As shown, when the matching result is a lane line that does not match a sub-template, the distance between the unmatched lane line and all sub-templates is calculated. If the distance is greater than the preset distance threshold K_dis, it is considered a newly detected lane line and added to the candidate lane lines. Otherwise, the tracking time of the sub-template closest to the unmatched lane line is determined to be greater than the preset time threshold. If it is, the lane line is judged to be a false detection lane line; otherwise, the sub-template is judged to be incorrect.

[0059] After processing using the lane line tracking method described above, the lane line template needs to be updated as a whole based on the intermediate processing results. New lane line sub-templates need to be added, and lane line sub-templates that do not meet the requirements need to be deleted. Finally, the lane lines output from the candidate lane line list are used as the lane line output for this frame.

[0060] Another embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, performs the following: acquiring each frame of image output by a vehicle-mounted forward-facing camera; initializing a lane line template based on the current vehicle speed; matching multiple lane lines detected based on the current frame image with the pre-constructed lane line template; classifying the matching results and performing corresponding processing.

[0061] In practical applications, the computer-readable storage medium can be any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. For example, a computer-readable storage medium can be, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this embodiment, the computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0062] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, capable of sending, propagating, or transmitting programs for use by or in connection with an instruction execution system, apparatus, or device.

[0063] Program code contained on a computer-readable medium may be transmitted using any suitable medium, including but not limited to wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0064] Computer program code for performing the operations of this invention can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as "C" or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0065] like Figure 5 As shown, another embodiment of the present invention provides a structural schematic diagram of a computer device. Figure 5 The computer device 12 shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of the present invention.

[0066] like Figure 5 As shown, the computer device 12 is represented in the form of a general-purpose computing device. The components of the computer device 12 may include, but are not limited to: one or more processors or processing units 16, system memory 28, and a bus 18 connecting different system components (including system memory 28 and processing unit 16).

[0067] Bus 18 represents one or more of several bus architectures, including a memory bus or memory controller, a peripheral bus, a graphics acceleration port, a processor, or a local bus using any of the various bus architectures. For example, these architectures include, but are not limited to, the Industry Standard Architecture (ISA) bus, the Micro Channel Architecture (MAC) bus, the Enhanced ISA bus, the Video Electronics Standards Association (VESA) local bus, and the Peripheral Component Interconnect (PCI) bus.

[0068] Computer device 12 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by computer device 12, including volatile and non-volatile media, removable and non-removable media.

[0069] System memory 28 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 30 and / or cache memory 32. Computer device 12 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, storage system 34 may be used to read and write non-removable, non-volatile magnetic media (…). Figure 5 Not shown; usually referred to as a "hard drive"). Although Figure 5 Not shown, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk") and an optical disk drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to bus 18 via one or more data media interfaces. Memory 28 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of the embodiments of the present invention.

[0070] A program / utility 40 having a set (at least one) of program modules 42 may be stored, for example, in memory 28. Such program modules 42 include, but are not limited to, an operating system, one or more application programs, other program modules, and program data. Each or some combination of these examples may include an implementation of a network environment. Program modules 42 typically perform the functions and / or methods described in the embodiments of the present invention.

[0071] Computer device 12 can also communicate with one or more external devices 14 (e.g., keyboard, pointing device, display 24, etc.), and with one or more devices that enable a user to interact with the computer device 12, and / or with any device that enables the computer device 12 to communicate with one or more other computing devices (e.g., network card, modem, etc.). This communication can be performed through input / output (I / O) interface 22. Furthermore, computer device 12 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) through network adapter 20. Figure 5 As shown, network adapter 20 communicates with other modules of computer device 12 via bus 18. It should be understood that, although... Figure 5 As not shown, it can be used in conjunction with computer device 12 with other hardware and / or software modules, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0072] The processor unit 16 executes various functional applications and data processing by running programs stored in the system memory 28, such as implementing a lane line tracking method based on template matching provided in the embodiments of the present invention.

[0073] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the implementation of the present invention. For those skilled in the art, other variations or modifications can be made based on the above description. It is impossible to exhaustively list all the implementation methods here. All obvious variations or modifications derived from the technical solutions of the present invention are still within the protection scope of the present invention.

Claims

1. A lane line tracking method based on template matching, characterized in that, include: S1: Acquire each frame of image output by the vehicle's forward-facing camera and initialize the lane line template based on the current vehicle speed; S2: Match the multiple lane lines detected based on the current frame image with the pre-built lane line template; S3: Classify the matching results and perform corresponding processing; Initializing the lane line template based on the current vehicle speed includes: Determine whether the current vehicle speed exceeds a preset vehicle speed threshold. If it does not exceed the threshold, clear the lane line template. Otherwise, process each frame of the image output by the vehicle-mounted forward camera to obtain multiple lane lines detected based on the current frame image. Determine if the pre-built lane line template is zero. If it is, build the lane line template. Otherwise, match the multiple lane lines detected in the current frame image with the pre-built lane line template. Matching multiple lane lines detected from the current frame image with pre-built lane line templates includes: Each lane line is matched with a sub-template in a pre-built lane line template to obtain the sub-template with the highest score for each lane line. Determine whether the score of the sub-template exceeds the matching score threshold. If it does, the matching is considered successful; otherwise, the matching fails. The matching results are divided into lane lines and their sub-templates that are successfully matched, sub-templates of lane lines that are not matched, and lane lines that are not matched with sub-templates. When the matching result is a successfully matched lane line and its sub-template, the tracking time corresponding to the sub-template is set to time = time + 1, and the content of its other corresponding sub-templates is updated. The line with the longest successful template tracking time and the highest lane line point probability score is added to the candidate lane line list. The distance between the remaining successfully matched lane lines and the lane lines in the candidate lane line list is calculated sequentially. If the distance is greater than a preset distance threshold, the remaining successfully matched lane lines are also added to the candidate lane line list. Otherwise, the tracking time difference between the lane lines in the candidate lane line list and the remaining successfully matched lane lines is calculated. Based on the tracking time difference and the comparison between the lane lines in the candidate lane line list and the remaining successfully matched lane lines, it is determined which lane line to add or retain to the candidate lane line list. When the matching result is that no sub-template is matched, the distance between the unmatched lane line and all sub-templates is calculated. If the distance is greater than the preset distance threshold, it is considered a newly detected lane line and added to the candidate lane line. Otherwise, the tracking time of the sub-template closest to the unmatched lane line is greater than the preset time threshold. If it is, the lane line is judged as a false detection lane line. Otherwise, the sub-template is judged to be incorrect.

2. The method according to claim 1, characterized in that, When the matching result is that no lane line is matched in the sub-template, if the probability value of the semantic segmentation result in the current frame sub-template exceeds the preset score threshold, and lane lines have appeared at this position in the past period, then the lane line in the current frame is considered to be missed, and a lane line is added to the candidate lane line.

3. The method according to claim 1, characterized in that, Processing each frame of the image output by the vehicle-mounted forward-facing camera includes: Each pixel of each frame of image output by the vehicle-mounted forward-facing camera is classified into two categories, and each pixel is given the probability of whether it is a point on the lane line. Clustering or fitting these points yields a parametric representation of the lane lines.

4. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1-3.

5. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the method as described in any one of claims 1-3.

Images