Lane line recognition method, device, system and vehicle

Abstract

The present invention discloses a lane line recognition method, device, system and vehicle. The lane line recognition method comprises the steps of: extracting feature data representing the lane line from an image; generating a lane line prediction area of ​​the current lane; and fitting the lane line of the current lane according to the feature data and the lane line prediction area. The present invention also discloses a corresponding computing device.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a lane line recognition method, device, system and vehicle. Background Art

[0002] With the continuous development of the automotive industry and the increasing demand for driving experience, more and more in-vehicle functions are becoming popular, such as lane departure warning, front vehicle collision warning, etc. The popularization of functions inevitably requires the support of sensors. Take the vehicle's front-view camera as an example. Nowadays, the front-view camera has gradually become a standard feature of vehicles, which can assist drivers to directly see obstacles in front of the vehicle.

[0003] On the other hand, the most important carrier between the vehicle and the road is the lane line. Taking the recognition of lane lines as an example, the lane lines on the road are identified during the driving process of the vehicle to ensure that the vehicle is within the lane limit and reduce the chance of the vehicle colliding with other vehicles due to crossing the lane. This is of great significance to improving driving safety. Therefore, the recognition of lane lines is generally considered to be a basic capability in vision-based applications, and how to quickly and accurately identify and extract lane lines is particularly important.

[0004] There are many existing lane line recognition solutions, some of which focus on execution efficiency, some focus on accuracy, or only focus on extracting the lane line of the current lane. However, there is currently no lane line recognition solution that can take all of the above into account. In the application scenarios of lane line recognition, it is often necessary to quickly identify the lane line to assist the driver in making decisions to avoid traffic accidents.

[0005] In view of this, a solution is needed that can identify lane lines quickly and accurately. Summary of the invention

[0006] To this end, the present invention provides a lane line recognition method, device, system and vehicle, in an effort to solve or at least alleviate at least one of the above problems.

[0007] According to one aspect of the present invention, a lane line recognition method is provided, comprising the steps of: extracting feature data representing the lane line from an image; generating a lane line prediction area for the current lane; and fitting the lane line of the current lane based on the feature data and the lane line prediction area.

[0008] Optionally, the method according to the present invention further comprises the step of: identifying lane lines of other lanes on the current road based on the lane lines of the current lane.

[0009] Optionally, in the method according to the present invention, the step of extracting feature data representing the lane line from the image further includes: obtaining an initial lane line of the current lane using the extracted feature data.

[0010] Optionally, in the method according to the present invention, the step of generating a lane line prediction area for the current lane includes: generating a lane line prediction area for the current lane based on vehicle driving data and attribute data of the current road.

[0011] Optionally, in the method according to the present invention, the step of fitting the lane line of the current lane based on the feature data and the lane line prediction area includes: selecting at least one feature data characterizing the lane line from the lane line prediction area; and fitting the lane line of the current lane based on the selected feature data.

[0012] Optionally, in the method according to the present invention, the step of identifying lane lines of other lanes on the current road based on the lane lines of the current lane includes: generating lane line detection areas for other lanes on the current road based on the lane lines of the current lane; and identifying at least one lane line from the lane line detection area.

[0013] Optionally, the method according to the present invention further comprises the step of: verifying the identified lane line.

[0014] According to another aspect of the present invention, a lane line recognition method is also provided, comprising the steps of: determining whether there is lane line information on the current road; if there is no lane line information, executing the lane line recognition method as described above to identify the lane line on the current image frame; if there is lane line information, identifying the lane line on the current image frame based on the lane line information.

[0015] Optionally, the method according to the present invention also includes the steps of: calculating the error between the identified lane line of the current image frame and the corresponding predicted lane line; when the error exceeds a threshold, executing the lane line recognition method as described above to re-identify the lane line on the current image frame.

[0016] According to another aspect of the present invention, a lane line recognition device is also provided, including: a preliminary extraction unit, suitable for extracting feature data representing the lane line from an image; a first recognition unit, suitable for generating a lane line prediction area of ​​the current lane based on vehicle driving data and attribute data of the current road, and fitting the lane line of the current lane based on the feature data and the lane line prediction area.

[0017] Optionally, the lane line recognition device according to the present invention further includes a second recognition unit, which is adapted to recognize lane lines of other lanes on the current road based on the lane line of the current lane.

[0018] According to another aspect of the present invention, a lane line recognition system is also provided, including: a judgment module, suitable for judging whether there is lane line information on the current road; a lane line recognition device, suitable for identifying the lane line on the current image frame when it is confirmed after judgment that there is no lane line information; a tracking and recognition module, suitable for identifying the lane line on the current image frame based on the lane line information when it is confirmed after judgment that there is lane line information.

[0019] Optionally, the system according to the present invention also includes: a tracking anomaly detection module, suitable for calculating the error between the identified lane line of the current image frame and the corresponding predicted lane line; the lane line recognition device is also suitable for re-identifying the lane line on the current image frame when the error exceeds a threshold.

[0020] According to yet another aspect of the present invention, there is provided a vehicle, comprising: the lane line recognition system as described above; and a camera unit coupled to the lane line recognition system, suitable for capturing images of the vehicle during driving.

[0021] Optionally, the vehicle according to the present invention further comprises: a status acquisition system adapted to acquire vehicle driving data of the vehicle; and a map information acquisition system adapted to acquire attribute data of the road on which the vehicle is driving.

[0022] According to another aspect of the present invention, a computing device is provided, comprising: at least one processor; and a memory storing program instructions, wherein the program instructions are configured to be suitable for execution by the at least one processor, and the program instructions include instructions for executing the lane line recognition method as described above.

[0023] According to another aspect of the present invention, a readable storage medium storing program instructions is provided. When the program instructions are read and executed by a computing device, the computing device executes the lane line recognition method as described above.

[0024] According to the solution of the present invention, the area where the current lane line is located (i.e., the lane line prediction area) is first predicted, and the lane line of the current lane can be obtained by curve fitting the feature data in the prediction area. At the same time, the lane line detection area of ​​other lanes is generated based on the lane line of the current lane, and the lane line recognition is performed only in the detection area. The above processing greatly reduces the search space of image processing, improves the accuracy, and improves the processing efficiency, meeting the real-time requirements in vehicle applications.

[0025] In addition, considering the complex environment in the lane line recognition process (such as the diversity of lane line shapes and the complexity of the road environment), as well as the fact that the camera unit is greatly affected by conditions such as lighting, weather, and road environment, the vehicle driving data, road attribute data and other information are integrated into the lane line recognition scheme, thereby improving the reliability of the recognition method through the fusion of multi-source data.

[0026] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented according to the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] To achieve the above and related purposes, certain illustrative aspects are described herein in conjunction with the following description and accompanying drawings, which indicate various ways in which the principles disclosed herein can be practiced, and all aspects and their equivalents are intended to fall within the scope of the claimed subject matter. The above and other purposes, features and advantages of the present disclosure will become more apparent by reading the following detailed description in conjunction with the accompanying drawings. Throughout the present disclosure, the same reference numerals generally refer to the same parts or elements.

[0028] Figure 1 shows a schematic diagram of a vehicle 100 according to some embodiments of the present invention;

[0029] Figure 2 A schematic diagram of a computing device 200 according to an embodiment of the present invention is shown;

[0030] Figure 3 A schematic flow chart of a lane line recognition method 300 according to some embodiments of the present invention is shown;

[0031] Figure 4 A schematic diagram showing the shape of lane lines according to some embodiments of the present invention is shown;

[0032] Figure 5A and Figure 5B A schematic diagram of a lane line prediction area and a lane line detection area according to an embodiment of the present invention is shown;

[0033] Figure 6 A schematic flow chart of a lane line recognition method 600 according to some other embodiments of the present invention is shown;

[0034] Figure 7 A schematic diagram showing a lane line recognition device 700 according to some embodiments of the present invention; and

[0035] Figure 8 A schematic diagram of a lane recognition system 120 according to some embodiments of the present invention is shown. DETAILED DESCRIPTION

[0036] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0037] According to an embodiment of the present invention, a vehicle for implementing a lane line recognition solution is provided. Figure 1 A schematic diagram of a vehicle 100 is shown according to some embodiments of the present invention.

[0038] like Figure 1 As shown, in addition to the basic configuration, the vehicle 100 is also provided with a camera unit 110 and a lane line recognition system 120 (further, the system 120 also includes a lane line recognition device 700). According to one embodiment, the camera unit 110 is arranged at the front end of the vehicle 100, especially at the middle position of the front end of the vehicle 100. For example, the camera unit 110 is fixed on the center console of the vehicle 100, but it is not limited thereto. As long as it is ensured that during the driving process of the vehicle 100, the camera unit 110 can clearly collect the environmental data in front of the vehicle 100 (especially the relevant data including the lane line), and generate an image of the vehicle during the driving process. When arranging the camera unit 110, those skilled in the art can perform multiple debugging based on experience until a position that meets the above conditions is found. At the same time, the camera unit 110 is also coupled to the lane line recognition system 120 to transmit the collected image to the system 120, and the system 120 processes the image and finally identifies the lane line on the driving road.

[0039] In addition, according to an embodiment of the present invention, a status acquisition system 130 and a map information acquisition system 140 are also arranged on the vehicle 100, and the status acquisition system 130 and the map information acquisition system 140 are respectively coupled to the lane line recognition system 120 to send the data collected by each to the lane line recognition system 120.

[0040] More specifically, the state acquisition system 130 can collect the driving data of the vehicle 100, such as the vehicle's location information, driving speed, posture information, etc. Among them, the location information can be combined with the map information acquisition system 140 to obtain the road information of the current road; the posture information is mainly used to determine the posture of the camera unit 110, so as to perform more accurate projection transformation and visual positioning. In some embodiments, the state acquisition system 130 may include one or more sensors arranged on the vehicle 100, such as a global navigation positioning system GNSS (for obtaining the vehicle's location), an inertial measurement unit IMU (for obtaining the vehicle's three-axis angular rate and three-axis acceleration, etc.) and an odometer ODOM (for obtaining the vehicle's driving speed, driving distance, etc.), etc., and the above-mentioned vehicle driving data is obtained based on these sensors.

[0041] The map information collection system 140 includes real-time road attribute data, such as road type, road name, road direction, road width, number of lanes on the road, etc. Furthermore, the map information collection system 140 can also provide road attribute data for the next road segment based on the current driving direction.

[0042] Meanwhile, the lane line recognition system 120 may be a separate module disposed in the vehicle 100, or may be implemented as a part of an onboard device or a central control unit of the vehicle 100, or may be a part disposed in the camera unit 110, and the embodiments of the present invention do not limit this. In an embodiment according to the present invention, the system 120 may be implemented by a computing device.

[0043] Figure 2 is a schematic block diagram of an example computing device 200 .

[0044] like Figure 2 As shown, in a basic configuration 202, computing device 200 typically includes a system memory 206 and one or more processors 204. A memory bus 208 may be used for communication between processor 204 and system memory 206.

[0045] Depending on the desired configuration, the processor 204 can be any type of processor, including but not limited to: a microprocessor (μP), a microcontroller (μC), a digital information processor (DSP), or any combination thereof. The processor 204 can include one or more levels of cache such as a primary cache 210 and a secondary cache 212, a processor core 214, and registers 216. An example processor core 214 can include an arithmetic logic unit (ALU), a floating point unit (FPU), a digital signal processing core (DSP core), or any combination thereof. An example memory controller 218 can be used with the processor 204, or in some implementations, the memory controller 218 can be an internal part of the processor 204.

[0046] Depending on the desired configuration, system memory 206 may be any type of memory, including but not limited to: volatile memory (such as RAM), non-volatile memory (such as ROM, flash memory, etc.), or any combination thereof. System memory 206 may include an operating system 220, one or more applications 222, and program data 224. In some implementations, application 222 may be arranged to execute instructions on the operating system by one or more processors 204 using program data 224.

[0047] The computing device 200 may also include an interface bus 240 that facilitates communication from various interface devices (e.g., output devices 242, peripheral interfaces 244, and communication devices 246) to the basic configuration 202 via the bus / interface controller 230. Example output devices 242 include a graphics processing unit 248 and an audio processing unit 250. They can be configured to facilitate communication with various external devices such as a display or speakers via one or more A / V ports 252. Example peripheral interfaces 244 may include a serial interface controller 254 and a parallel interface controller 256, which may be configured to facilitate communication with external devices such as input devices (e.g., keyboards, mice, pens, voice input devices, touch input devices) or other peripherals (e.g., printers, scanners, etc.) via one or more I / O ports 258. Example communication devices 246 may include a network controller 260, which may be arranged to facilitate communication with one or more other computing devices 262 via a network communication link via one or more communication ports 264.

[0048] A network communication link can be an example of a communication medium. Communication media can generally be embodied as computer-readable instructions, data structures, program modules in a modulated data signal such as a carrier wave or other transmission mechanism, and can include any information delivery medium. A "modulated data signal" can be a signal in which one or more of its data sets or its changes can be carried out in a manner that encodes information in the signal. As non-limiting examples, communication media can include wired media such as a wired network or a dedicated line network, and various wireless media such as sound, radio frequency (RF), microwave, infrared (IR) or other wireless media. The term computer-readable medium used herein can include both storage media and communication media.

[0049] The computing device 200 can be implemented as a personal computer including a desktop computer and a notebook computer configuration. Of course, the computing device 200 can also be implemented as a part of a small-sized portable (or mobile) electronic device, which can be a cellular phone, a digital camera, a personal digital assistant (PDA), a personal media player device, a wireless network browsing device, a personal head-mounted device, an application-specific device, or a hybrid device that can include any of the above functions. It can even be implemented as a server, such as a file server, a database server, an application server, and a WEB server. The embodiments of the present invention are not limited to this.

[0050] In an embodiment of the present invention, the computing device 200 is configured to execute the lane line recognition method 300 and the method 600 according to the present invention. The application 222 of the computing device 200 includes a plurality of program instructions for executing the method 300 and / or the method 600 according to the present invention.

[0051] Figure 3 FIG. 3 is a flow chart of a lane line recognition method 300 according to some embodiments of the present invention. The method 300 is suitable for being executed in a lane line recognition device 700. Figure 3 As shown, the method 300 starts at step S310 .

[0052] In step S310, feature data representing lane lines are extracted from the image.

[0053] Further, in step S310, only the feature data of the current lane is extracted. In some embodiments, based on the color, shape and other features of the lane line, a number of segmented straight lines are extracted from the image through grayscale processing, edge detection, color thresholding processing and other methods. These segmented straight lines are the feature data that characterize the lane line. Then, the initial lane lines of the current lane are formed using these extracted feature data. Generally, there are two initial lane lines of the current lane, which are located on the left and right sides of the current lane respectively. In the process of executing the lane line recognition scheme of the present invention, the same steps are performed on the lane lines on the left and right sides. To simplify the description, the left and right lane lines are collectively referred to as lane lines.

[0054] It should be understood that the method of extracting feature data representing lane lines from images using some traditional image processing algorithms is a common technical means in the field and will not be described in detail here. Any known or future known method for extracting feature data can be combined with the embodiments of the present invention to implement the lane line recognition solution according to the present invention.

[0055] According to some other embodiments of the present invention, considering that there may be some interference information, such as noise, in the image captured by the camera unit 110, the image is preprocessed before executing step S310 to filter out the interference in the image. The preprocessing may be Gaussian filtering, median filtering, etc., and the embodiments of the present invention do not impose too many restrictions on this.

[0056] After processing in step S310, the extracted feature data may still include some non-lane line data, such as straight lines included in the outline of the vehicle 100. Thus, the initial lane lines need to be further extracted to filter out the non-lane line data.

[0057] Then in step S320, a lane line prediction area of ​​the current lane is generated.

[0058] According to an embodiment, a lane line prediction area of ​​the current lane is generated based on the vehicle driving data from the state acquisition system 130 and the attribute data of the current road from the map information acquisition system 140. Specifically, the process of generating the lane line prediction area can be performed in the following two steps (it should be understood that although the initial lane lines on the left and right sides are not distinguished in the description here, during the execution process, the initial lane lines on the left and right sides of the current lane are both processed as follows).

[0059] Step 1), based on the vehicle driving data, calculate the position relationship between the photographic reference point and the initial lane line.

[0060] In one embodiment, the position of the photographic reference point is first calculated based on the vehicle driving data. As mentioned above, the vehicle driving data includes: the position of the vehicle, the driving speed of the vehicle and the posture information. The photographic reference point here can be the center position of the camera unit 110 on the image; of course, the photographic reference point can also be off-center or at a certain position of the image (such as the lower left corner of the image, etc.), which can be selected according to the actual situation. For example, the photogrammetry method is used to locate its position coordinates in the image, but it is not limited to this. Then, the distance between the photographic reference point and the initial lane line and the angle between the straight line where the photographic reference point is located and the initial lane line are calculated in the image. In an embodiment of the present invention, assuming that the photographic reference point is A, the distance from A to the initial lane lines on the left and right sides is calculated respectively, which are recorded as D1 and D2 respectively. Then, the angle between the straight line where the photographic reference point is located and the initial lane lines on the left and right sides is calculated respectively, which are recorded as R1 and R2 respectively. With D1 and D2, R1 and R2, the positional relationship between the photographic reference point and the initial lane line is obtained, and the positional relationship represents the positional relationship between the photographic reference point and the initial lane line in the image.

[0061] Step 2) Acquire the attribute data of the current road, and generate the lane line prediction area of ​​the current lane based on the position relationship and the attribute data.

[0062] Considering that the direction of the vehicle head (that is, the direction of the photographic reference point) may not be completely consistent with the road direction during the vehicle's driving. For example, when the vehicle changes lanes, the direction of the vehicle head is different from the road direction. Therefore, in an embodiment according to the present invention, the direction of the current lane is predicted based on the direction of the current road.

[0063] Specifically, firstly, based on the positioning of the current vehicle 100, the attribute data of the road where the current vehicle 100 is located is obtained from the map information acquisition system 140. The attribute data may include the travel direction, road width, etc. of the current road. Then, the lane line prediction area of ​​the current lane is generated based on the positional relationship and attribute data calculated previously. Optionally, the shape of the lane line is first predicted based on the attribute data of the current road. For example, the general shape of the current lane line can be predicted based on the travel direction of the current road, where the lane line shape includes a straight line, a curve that bends to the left, a curve that bends to the right, etc. In one embodiment, the map information acquisition system 140 segments the roads on the road surface (especially curved sections) to show the shape of the lane lines. As Figure 4 A schematic diagram of the shape of a lane line according to an embodiment of the present invention is shown. Figure 4In the figure, nodes A, B, C, D, E, and F segment the curved road section, and the road attribute data fed back by the map information acquisition system 140 includes all the nodes contained in the area where the current position is located. Through these nodes, the shape of the lane line ahead can be roughly predicted. Then, the predicted lane line shape is projected and transformed onto the image coordinate system to generate a lane line prediction area. In one embodiment, the shape of the predicted current lane line is projected and transformed onto the image coordinate system according to the positional relationship, and a certain degree of magnification is performed to obtain the lane line prediction area of ​​the current lane. In an embodiment of the present invention, the degree of magnification can refer to factors such as lane width and image width to ensure that the lane line is within the lane line prediction area.

[0064] like Figure 5A , shows a schematic diagram of a lane line prediction area according to an embodiment of the present invention. Figure 5A As can be seen from the figure, the road contains more than 4 lanes. Figure 5A In FIG. 4 , lane lines are represented by rectangular segments, and lane line prediction regions 410 and 420 are drawn with straight lines as lane line prediction regions on the left and right sides of the current lane (i.e., the second lane in the figure). It should be noted that Figure 5A This is only for illustration to illustrate the lane line prediction area and does not limit the width and range of the prediction area.

[0065] Then in step S330, the lane line of the current lane is fitted based on the feature data and the lane line prediction area.

[0066] As mentioned above, the feature data extracted by step S310 will also include some non-lane line data, so in one embodiment of the present invention, at least one feature data representing the lane line is selected from the lane line prediction area. In other words, the feature data that is not in the lane line prediction area is eliminated, and the feature data after elimination is used as the selected feature data. Then, based on the selected feature data, the lane line of the current lane is fitted. In an embodiment of the present invention, a curve fitting method is used to fit the feature data, that is, the endpoints of the straight line elements, to obtain the lane line of the current lane. Optionally, a straight line is fitted by linear fitting as the lane line. It should be noted that the embodiment of the present invention does not limit the specific curve fitting algorithm used to fit the feature data.

[0067] According to an embodiment of the present invention, considering that there is more than one lane on the driving road, after the lane line of the current lane is determined, in the subsequent step S340, the lane lines of other lanes on the current road are identified based on the lane line of the current lane.

[0068] According to an embodiment of the present invention, the process of identifying lane lines of other lanes can be performed in the following two steps.

[0069] Step a), based on the lane line of the current lane, generate lane line detection areas for other lanes on the current road.

[0070] According to an embodiment, the lane width (i.e., D1+D2) on the current road can be calculated based on the distances D1 and D2 from the photographic reference point to the initial lane lines on the left and right sides. Then, based on the lane line of the current lane, the lane line detection area of ​​at least one other lane is expanded to both sides, where the expanded distance is the lane width.

[0071] like Figure 5B , shows a schematic diagram of a lane line detection area according to an embodiment of the present invention. Figure 5B In the lane detection area, a straight line is used. For comparison, Figure 5A The lane line prediction areas 410 and 420 in FIG. Figure 5B In the example, region 430, region 440 and region 450 are lane line detection regions obtained according to step a). It should be noted that Figure 5B This is only for illustration to illustrate the lane detection area and does not limit the width and range of the detection area.

[0072] by Figure 5B As an example, the process of generating a lane line detection area is described. Taking the lane line in area 410 as a reference, expand it to the left by a lane width to obtain a predicted lane line, and then amplify the predicted lane line to a certain extent to obtain a lane line detection area 430. The degree of amplification here can refer to step S320, as long as it is ensured that area 430 can cover the lane line. Similarly, taking the lane line in area 420 as a reference, expand it to the right by a lane width, and after the same amplification process, obtain lane line detection area 440; and so on, expand two lane widths to the right of the lane line in area 420 to obtain lane line detection area 450. The specific process will not be repeated one by one.

[0073] It should be noted that Figure 5B The example of expanding to both sides to generate three lane line detection areas is only given as an example, and the embodiments of the present invention are not limited to this. Generally, in order to avoid missed detection, the number of expanded lane line detection areas is preferably greater than the number of lane lines. The number of lane lines can be obtained from the map information collection system 140, or a value can be set based on experience, and the embodiments of the present invention do not impose too many restrictions on this.

[0074] Step b), identifying at least one lane line from these lane line detection areas.

[0075] According to an embodiment, feature data representing the lane lines are directly extracted from each lane line detection area. Optionally, based on features such as the color and shape of the lane lines, a number of segmented straight lines are extracted from each lane line detection area through grayscale processing, edge detection, color threshold processing, and other methods. These segmented straight lines are the feature data representing the corresponding lane lines.

[0076] Then, the lane line of the corresponding lane is obtained using the extracted feature data. Optionally, the feature data extracted in each detection area is fitted by curve fitting to obtain the lane line corresponding to each detection area.

[0077] According to another embodiment, if a sufficient amount of feature data is not extracted within the detection area, it is considered that the detection area may have reached outside the road, or the lane lines within the detection area are too blurred, so the detection area is directly eliminated and the feature data within the detection area is no longer fitted.

[0078] At this point, the lane lines of the current lane and other lanes in the image have been identified. According to other embodiments of the present invention, there may be occlusions (such as the vehicle in front) in the image, which may cause the lane lines to be blocked, or the lane lines themselves may be stained or missing, which may affect the recognition results. Therefore, after all the lane lines in the image are identified, all the recognition results must be verified. Optionally, the identified lane lines are verified using the attribute data of the road. For example, if the attribute data of the road shows that there are 5 lane lines on the road, but 4 lane lines are actually identified, it means that the lane lines are missed. At this time, interpolation can be used to supplement the missing lane line. Alternatively, if 6 lane lines are actually identified, it means that 1 lane line is identified more, and the redundant lane lines can be removed in combination with information such as lane width. In short, the redundant lane lines are removed, and the unextracted lane lines or incomplete lane lines are reconstructed or completed, that is, the recognition and extraction of all lane lines are completed.

[0079] After extracting the lane lines on the driving road, according to an embodiment of the present invention, the driving state of the current vehicle can also be controlled based on the identified lane lines. For example, based on the identified lane lines, the driving path of the vehicle 100 is monitored. Once it is detected that the vehicle 100 is driving outside the lane lines, an early warning is issued to assist the driver in driving safely. For another example, based on the identified lane lines, it is also possible to monitor whether there are other vehicles driving on the lane lines of the current lane within a close distance of the vehicle 100. If so, an early warning is issued to remind the driver that there may be a vehicle ahead that is about to merge.

[0080] According to the lane line recognition method 300 of the present invention, taking into account the complex environment in the lane line recognition process (such as the diversity of lane line shapes and the complexity of the road environment), as well as the fact that the camera unit 110 is greatly affected by conditions such as lighting, weather, and road environment, information such as vehicle driving data and road attribute data are integrated into the lane line recognition scheme, thereby improving the reliability of the recognition method through the fusion of multi-source data.

[0081] At the same time, combined with the attribute data of the current road, the shape of the lane line and the area where the lane line is located (i.e., the lane line prediction area) are predicted, and accurate recognition results can be obtained by curve fitting the feature data in the prediction area. In addition, based on the lane line of the current lane, the lane line detection area of ​​other lanes is generated, and lane line recognition is performed only in the detection area. The above processing greatly reduces the search space of image processing, improves accuracy, and improves processing efficiency, meeting the real-time requirements of vehicle applications.

[0082] According to the lane line recognition scheme of the present invention, the camera unit 110 continuously captures corresponding images containing lane lines during the driving process of the vehicle 100. According to other embodiments, the lane lines in the image frame at the current moment that have been recognized are used to quickly recognize the lane lines in the image frame at the next moment, so as to achieve accurate and rapid recognition of the lane lines during the entire driving process.

[0083] Figure 6 FIG. 6 is a flow chart of a lane line recognition method 600 according to some other embodiments of the present invention. The method 600 is suitable for being executed in the lane line recognition system 120. Figure 6 As shown, the method 600 starts at step S610 .

[0084] In step S610, it is determined whether there is lane line information on the current road. In one embodiment of the present invention, the lane line information includes at least one lane line identified from the previous image frame. In other words, if the lane line on the road has been identified during the current driving process, it is determined that there is lane line information on the current road; if the lane line has not been identified or the lane line has not been identified, it is determined that there is no lane line information.

[0085] If there is no lane line information, then in the subsequent step S620, a lane line recognition method is executed to recognize the lane line of the current image frame. Then, the process re-enters step S610 for determination, and the determination result at this time is: there is lane line information.

[0086] In the solution according to the present invention, any lane line recognition method can be used to identify the lane lines of the current image frame. In a preferred embodiment, the lane line recognition method 300 described above is used to identify the lane lines on the current image frame.

[0087] If lane line information exists, then in the subsequent step S630, the lane line on the current image frame is identified based on the lane line information.

[0088] According to an embodiment of the present invention, at least one initial lane line is extracted from the current image frame. Referring to the foregoing, based on the color, shape and other features of the lane line, grayscale processing, edge detection, color thresholding and other methods can be used to extract at least one lane line from the current image frame as the initial lane line. Next, at least one predicted lane line of the current image frame is generated based on the lane line information. Optionally, based on the lane line information contained in the lane line information, combined with the vehicle's driving speed and the frequency of image acquisition, the lane line of the current image frame can be estimated as the predicted lane line. Finally, the lane line of the current image frame is identified based on the initial lane line and the predicted lane line. Optionally, based on the initial lane line and the predicted lane line, Kalman filtering is used to obtain the lane line of the current image frame.

[0089] Through the tracking mechanism described above, the lane lines in the next image frame can be quickly and accurately identified based on the previously identified lane lines.

[0090] In another embodiment, in order to improve the robustness of method 600, after executing step S630 to obtain the final recognition result, it is necessary to calculate the recognition error. If the error converges, it means that the current tracking and recognition mechanism is correct. If the error does not converge, it means that the current tracking and recognition mechanism has an error and needs to be restarted to ensure the stability of tracking.

[0091] Therefore, according to another embodiment of the present invention, the step of identifying the lane line of the current image frame based on the initial lane line and the predicted lane line also includes: calculating the error between the identified lane line of the current image frame and the corresponding predicted lane line, if the error is within the threshold range (i.e., the error converges), it is determined that the identification is accurate; if the error exceeds the threshold (i.e., the error does not converge), it is determined that the identification is incorrect, and the process proceeds to step S620 to execute the lane line identification method to re-identify the lane line on the current image frame. In a preferred embodiment, when it is determined that the error exceeds the threshold, the lane line identification method 300 as described above is executed to re-identify the lane line on the current image frame.

[0092] According to the lane line recognition method 600 of the present invention, if the initial recognition of the current road has been completed, the current lane line can be directly recognized based on the initial recognition result (ie, lane line information), which greatly reduces the amount of calculation for recognizing the lane line and improves the recognition efficiency.

[0093] In summary, the solution of the present invention provides a fast, accurate, and highly reliable lane line recognition solution, and can identify all lane lines on the road. Based on the lane line recognition solution of the present invention, it is possible to quickly and accurately assist drivers in making decisions to avoid traffic accidents. In addition, this solution does not rely on high-precision positioning and high-precision maps, but can be achieved by combining sensors on existing vehicles. Through multi-source data fusion, they complement each other and solve the most basic functional support for vehicle vision applications.

[0094] Figure 7 A schematic diagram of a lane line recognition device 700 according to some embodiments of the present invention is shown. The device 700 is used to execute the lane line recognition method 300 according to the present invention.

[0095] like Figure 7 As shown, the lane line recognition device 700 includes a preliminary extraction unit 710 and a first recognition unit 720 coupled to each other. The preliminary extraction unit 710 extracts feature data representing the lane line from the image. The first recognition unit 720 generates a lane line prediction area of ​​the current lane based on the vehicle driving data and the attribute data of the current road, and fits the lane line of the current lane according to the feature data and the lane line prediction area.

[0096] In addition, the lane line recognition device 700 may further include a second recognition unit 730. The second recognition unit 730 recognizes lane lines of other lanes on the current road based on the lane line of the current lane.

[0097] It should be understood that the specific description of the device 700 can refer to the previous description of the method 300, and due to space limitations, it will not be repeated here.

[0098] Likewise, Figure 8 A schematic diagram of a lane line recognition system 120 according to some embodiments of the present invention is shown. The system 120 is used to execute a lane line recognition method 600 according to the present invention.

[0099] like Figure 8As shown, the lane line recognition system 120 includes a judgment module 122, a lane line recognition device 700 and a tracking and recognition module 124 that are coupled to each other. Among them, the judgment module 122 judges whether there is lane line information on the current road. When the lane line recognition device 700 determines that there is no lane line information after judgment, it identifies the lane line on the current image frame. When the tracking and recognition module 124 determines that there is lane line information after judgment, it identifies the lane line on the current image frame based on the lane line information. In one embodiment, the tracking and recognition module 124 can also extract at least one initial lane line from the current image frame, and at the same time, generate at least one predicted lane line of the current image frame according to the lane line information, and then identify the lane line of the current image frame according to the initial lane line and the predicted lane line.

[0100] According to another embodiment of the present invention, the system 120 may further include a tracking anomaly detection module 126. The tracking anomaly detection module 126 is used to calculate the error between the identified lane line of the current image frame and the corresponding predicted lane line, and when the error exceeds a threshold, the lane line recognition device 700 is used to re-recognize the lane line on the current image frame.

[0101] It should be understood that the specific description of system 120 can refer to the previous description of method 300 and method 600, and due to space limitations, it will not be repeated here.

[0102] It should be understood that the specific description of each step in method 500 and method 600 can refer to the previous description of system 100, and due to space limitations, it will not be repeated here.

[0103] The various techniques described herein may be implemented in combination with hardware or software, or a combination thereof. Thus, the method and apparatus of the present invention, or some aspects or parts of the method and apparatus of the present invention may be in the form of program codes (i.e., instructions) embedded in a tangible medium, such as a removable hard disk, a USB flash drive, a floppy disk, a CD-ROM, or any other machine-readable storage medium, wherein when the program is loaded into a machine such as a computer and executed by the machine, the machine becomes a device for practicing the present invention.

[0104] In the case where the program code is executed on a programmable computer, the computing device generally includes a processor, a storage medium readable by the processor (including volatile and non-volatile memory and / or storage elements), at least one input device, and at least one output device. The memory is configured to store the program code; the processor is configured to execute the data storage method and / or data query method of the present invention according to the instructions in the program code stored in the memory.

[0105] By way of example and not limitation, readable media include readable storage media and communication media. Readable storage media stores information such as computer readable instructions, data structures, program modules or other data. Communication media generally embody computer readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transmission mechanism, and include any information transfer medium. Any combination of the above is also included within the scope of readable media.

[0106] In the description provided herein, algorithms and displays are not inherently related to any particular computer, virtual system or other device. Various general purpose systems can also be used together with the examples of the present invention. According to the above description, it is obvious that the structure required for constructing such systems. In addition, the present invention is not directed to any specific programming language either. It should be understood that various programming languages ​​can be utilized to implement the content of the present invention described herein, and the above description of specific languages ​​is for the purpose of disclosing the best mode of the present invention.

[0107] In the description provided herein, a large number of specific details are described. However, it is understood that embodiments of the present invention can be practiced without these specific details. In some instances, well-known methods, structures and techniques are not shown in detail so as not to obscure the understanding of this description.

[0108] Similarly, it should be understood that in order to streamline the present disclosure and aid in understanding one or more of the various inventive aspects, in the above description of exemplary embodiments of the present invention, the various features of the present invention are sometimes grouped together into a single embodiment, figure, or description thereof. However, this disclosed method should not be interpreted as reflecting the following intention: that the claimed invention requires more features than the features explicitly recited in each claim. More specifically, as reflected in the claims below, inventive aspects lie in less than all the features of the individual embodiments disclosed above. Therefore, the claims that follow the specific embodiment are hereby expressly incorporated into the specific embodiment, with each claim itself serving as a separate embodiment of the present invention.

[0109] Those skilled in the art will appreciate that the modules or units or components of the devices in the examples disclosed herein may be arranged in the devices described in the embodiment, or alternatively may be located in one or more devices different from the devices in the examples. The modules in the foregoing examples may be combined into one module or may be divided into multiple submodules.

[0110] Those skilled in the art will appreciate that the modules in the devices in the embodiments may be adaptively changed and arranged in one or more devices different from the embodiments. The modules or units or components in the embodiments may be combined into one module or unit or component, and in addition they may be divided into a plurality of submodules or subunits or subcomponents. Except that at least some of such features and / or processes or units are mutually exclusive, all features disclosed in this specification (including the accompanying claims, abstracts and drawings) and all processes or units of any method or device disclosed in this manner may be combined in any combination. Unless otherwise expressly stated, each feature disclosed in this specification (including the accompanying claims, abstracts and drawings) may be replaced by an alternative feature providing the same, equivalent or similar purpose.

[0111] In addition, those skilled in the art will appreciate that, although some embodiments described herein include certain features included in other embodiments but not other features, the combination of features of different embodiments is meant to be within the scope of the present invention and form different embodiments. For example, in the claims below, any one of the claimed embodiments may be used in any combination.

[0112] In addition, some of the embodiments are described herein as methods or combinations of method elements that can be implemented by a processor of a computer system or by other devices that perform the functions. Therefore, a processor with necessary instructions for implementing the method or method elements forms a device for implementing the method or method elements. In addition, the elements described herein of the device embodiments are examples of devices for implementing the functions performed by the elements for the purpose of implementing the invention.

[0113] As used herein, unless otherwise specified, the use of ordinal numbers "first," "second," "third," etc. to describe common objects merely indicates that different instances of similar objects are involved, and is not intended to imply that the objects so described must have a given order in time, space, order, or in any other manner.

[0114] Although the present invention has been described according to a limited number of embodiments, it will be apparent to those skilled in the art, with the benefit of the above description, that other embodiments may be envisioned within the scope of the invention thus described. In addition, it should be noted that the language used in this specification is selected primarily for readability and teaching purposes, rather than for explaining or defining the subject matter of the present invention. Therefore, many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the appended claims. The disclosure of the present invention is illustrative and not restrictive with respect to the scope of the present invention, and the scope of the present invention is defined by the appended claims.

Claims

1. A lane line recognition method, comprising the steps of: Extract feature data representing lane lines from the image; Based on the vehicle driving data, the positional relationship between the photographic reference point and the initial lane line is calculated; Get the attribute data of the current road; as well as Generate a lane line prediction area for the current lane based on the positional relationship and attribute data; Fitting the lane line of the current lane according to the feature data and the lane line prediction area; The step of generating a lane line prediction area of ​​the current lane according to the position relationship and the attribute data includes: Predict the shape of lane lines based on the current road direction; The predicted lane line shape is projected and transformed into the image coordinate system to generate the lane line prediction area.

2. The method according to claim 1, further comprising the steps of: Based on the lane markings of the current lane, the lane markings of other lanes on the current road are identified.

3. The method of claim 2, wherein: The step of extracting feature data representing lane lines from the image also includes: The extracted feature data is used to obtain the initial lane line of the current lane.

4. The method of claim 1, wherein: The steps of calculating the positional relationship between the photographic reference point and the initial lane line based on the vehicle driving data include: Based on the vehicle driving data, the position of the photographic reference point is calculated; Calculate the distance between the photographic reference point and the initial lane line; and Calculate the angle between the straight line where the photographic reference point is located and the initial lane line.

5. The method of claim 4, wherein: The step of fitting the lane line of the current lane according to the feature data and the lane line prediction area comprises: Selecting at least one feature data representing the lane line from the lane line prediction area; Based on the selected feature data, the lane line of the current lane is fitted.

6. The method of claim 2, wherein: Based on the lane line of the current lane, the steps of identifying the lane lines of other lanes on the current road include: Based on the lane line of the current lane, generate the lane line detection area of ​​other lanes on the current road; At least one lane line is identified from the lane line detection area.

7. The method of claim 6, wherein: The step of generating lane line detection areas of other lanes on the current road based on the lane line of the current lane includes: Calculate lane width on the current road; Taking the lane line of the current lane as a reference, the lane line detection area of ​​at least one other lane is expanded to both sides, wherein the expanded distance is the lane width.

8. The method of claim 7, wherein: The step of identifying at least one lane line from the lane line detection area comprises: Extracting feature data representing the lane line from each lane line detection area; The lane line of the corresponding lane is obtained using the extracted feature data.

9. The method according to any one of claims 1 to 8, wherein: The vehicle driving data includes at least: the vehicle's position, driving speed and posture.

10. The method according to any one of claims 1 to 8, wherein: The attribute data of the road includes at least: the direction of travel and the width of the road.

11. The method of claim 10, wherein: Before the step of extracting the characteristic data representing the lane from the image, the method further includes the following steps: The image is preprocessed to filter out noise in the image.

12. The method of claim 11, wherein: After the step of identifying lane lines of other lanes on the current road based on the lane line of the current lane, the step further includes: Verify the identified lane lines.

13. The method according to claim 1 or 2, further comprising the steps of: Based on the identified lane lines, the driving state of the current vehicle is controlled.

14. The method of claim 13, wherein: The step of controlling the driving state of the current vehicle based on the identified lane line includes: When a vehicle is detected traveling outside the lane line, an early warning is issued to assist the driver in driving safely.

15. A lane line recognition method, comprising the steps of: Determine whether there is lane line information on the current road; If the lane line information does not exist, executing the lane line recognition method according to any one of claims 1 to 14 to recognize the lane line on the current image frame; If the lane line information exists, the lane line on the current image frame is identified based on the lane line information.

16. The method of claim 15, wherein: Based on the lane line information, the step of identifying the lane line on the current image frame includes: Extract at least one initial lane line from the current image frame; Generate at least one predicted lane line of the current image frame according to the lane line information; The lane line of the current image frame is identified based on the initial lane line and the predicted lane line.

17. The method of claim 16, wherein: The step of identifying the lane line of the current image frame according to the initial lane line and the predicted lane line comprises: Based on the initial lane line and the predicted lane line, the lane line of the current image frame is obtained using Kalman filtering.

18. The method of claim 17, wherein: The step of identifying the lane line of the current image frame according to the initial lane line and the predicted lane line also includes: Calculate the error between the identified lane line of the current image frame and the corresponding predicted lane line; When the error exceeds a threshold, the lane line recognition method according to any one of claims 1 to 9 is executed to re-recognize the lane line on the current image frame.

19. The method according to any one of claims 15 to 18, wherein: The lane line information includes at least one lane line identified from a previous image frame.

20. A lane line recognition device, comprising: A preliminary extraction unit, adapted to extract feature data representing lane lines from the image; The first recognition unit is adapted to calculate the positional relationship between the photographic reference point and the initial lane line based on the vehicle driving data; obtain the attribute data of the current road; and generate the lane line prediction area of ​​the current lane according to the positional relationship and the attribute data, and fit the lane line of the current lane according to the feature data and the lane line prediction area; The step of generating a lane line prediction area of ​​the current lane according to the position relationship and the attribute data includes: Predict the shape of lane lines based on the current road direction; The predicted lane line shape is projected and transformed into the image coordinate system to generate the lane line prediction area.

21. The apparatus of claim 20, further comprising: The second recognition unit is adapted to recognize lane lines of other lanes on the current road based on the lane line of the current lane.

22. A lane recognition system, comprising: A judgment module, adapted to judge whether there is lane line information on the current road; The lane line recognition device as claimed in claim 20, adapted to recognize the lane line on the current image frame when it is determined after judgment that no lane line information exists; The tracking and identification module is adapted to identify the lane line on the current image frame based on the lane line information when it is determined that there is lane line information after judgment.

23. The system of claim 22, wherein: The tracking and identification module is also suitable for: Extract at least one initial lane line from the current image frame; Generate at least one predicted lane line of the current image frame according to the lane line information; The lane line of the current image frame is identified based on the initial lane line and the predicted lane line.

24. The system of claim 23, further comprising: A tracking anomaly detection module adapted to calculate an error between the identified lane line of the current image frame and the corresponding predicted lane line; The lane line recognition device is also suitable for re-recognizing the lane line on the current image frame when the error exceeds a threshold.

25. A vehicle comprising: The lane recognition system as claimed in any one of claims 22 to 24; as well as The camera unit coupled to the lane line recognition system is suitable for collecting images of the vehicle during driving.

26. The vehicle of claim 25, further comprising: a state acquisition system adapted to acquire vehicle travel data of the vehicle; as well as The map information collection system is suitable for collecting attribute data of the road on which the vehicle travels.

27. A computing device comprising: at least one processor; and A memory storing program instructions, wherein the program instructions are configured to be executed by the at least one processor, and the program instructions include instructions for executing the method according to any one of claims 1-19.

28. A readable storage medium storing program instructions, when the program instructions are read and executed by a computing device, the computing device executes the method according to any one of claims 1 to 19.

Images