Vehicle lane line detection method and apparatus, autonomous mobile apparatus, and storage medium

Abstract

Provided in the embodiments of the present application are a vehicle lane line detection method and apparatus, an autonomous mobile apparatus, and a storage medium, relating to the technical field of autonomous mobile apparatuses. The method comprises: obtaining an original image captured by a camera of an autonomous mobile apparatus, and according to the original image, computing a gradient image based on the a channel and b channel; extracting line segments located on an edge of a vehicle lane line from the gradient image based on the a channel and b channel to obtain a plurality of original line segments; using an iterative method, cascading the collinear line segments among the plurality of original line segments to obtain a plurality of cascaded line segments; generating a plurality of candidate vehicle lane lines from the plurality of cascaded line segments; and merging the plurality of collinear candidate vehicle lane lines to obtain a vehicle lane line in the original image. Thus, a more accurate original line segment can be extracted with respect to a challenging image having a large amount of reflected light and low contrast, and the accuracy, robustness, and practicality of vehicle lane line detection are improved.

Description

Lane line detection method and device, autonomous mobile device, and storage medium Technical Field

[0001] The present application relates to the technical field of autonomous mobile devices, and more specifically, to a lane line detection method and device, an autonomous mobile device, and a storage medium. Background Art

[0002] Lane line detection is the core of lane keeping technology. By detecting lane lines, real-time feedback can be provided on the position of the autonomous mobile device, allowing the autonomous mobile device to promptly adjust its movement direction and speed based on the real-time position, thereby ensuring that the vehicle body remains in the lane (the area between two lane lines is the lane) during stable driving. Or, when the autonomous mobile device switches lanes, it assists the autonomous mobile device in safely and smoothly transitioning from the starting lane to another lane.

[0003] Current lane detection technologies include computer vision methods. These methods typically involve the following steps: extracting low-level features from images captured by cameras on autonomous mobile devices; classifying and distinguishing the extracted features; performing curve fitting on the features that meet the classification criteria to obtain candidate lane lines; grouping the candidate lane lines and using vanishing points to remove erroneous lane lines from the candidate lane lines to obtain the final lane lines. This computer vision method places high demands on the quality of the camera-captured images and is generally only applicable to images with a wide viewing angle and low reflections. When the camera is mounted at a low height, the lane line's appearance in the image decreases at a rate inversely proportional to the square of the distance. Furthermore, the image of the lane line will contain a large amount of reflected light, causing fragmentation of the lane lines, resulting in less robust and accurate lane detection. Furthermore, images with a wide viewing angle and low reflections require the camera to be mounted at a high height, which places a high demand on the height of the autonomous mobile device itself. This makes it unsuitable for autonomous mobile devices with low heights, resulting in poor practicality.

[0004] In other words, it is currently only applicable to images with a large viewing angle and less reflection. When the camera is installed at a low height, there are problems with lane line detection, such as low robustness, low accuracy, and poor practicality.

[0005] Summary of the Invention

[0006] The embodiments of the present application provide a lane line detection method and device, an autonomous mobile device, and a storage medium to improve the above-mentioned problems.

[0007] In a first aspect, an embodiment of the present application provides a lane line detection method, which includes: obtaining an original image captured by a camera of an autonomous mobile device, and calculating a gradient image based on an ab channel based on the original image; extracting line segments located on the edges of the lane lines from the gradient image based on the ab channel to obtain multiple original line segments; using an iterative method to cascade collinear line segments in the multiple original line segments to obtain multiple cascaded line segments; generating multiple candidate lane lines from the multiple cascaded line segments; and merging the multiple collinear candidate lane lines to obtain the lane lines in the original image.

[0008] In a second aspect, an embodiment of the present application provides a lane line detection device, which includes: an image acquisition module for acquiring an original image captured by a camera of an autonomous mobile device, and calculating a gradient image based on an ab channel based on the original image; a line segment extraction module for extracting line segments located on the edge of the lane line from the gradient image based on the ab channel to obtain multiple original line segments; an iterative cascade module for cascading collinear line segments in multiple original line segments using an iterative method to obtain multiple cascaded line segments; a lane line generation module for generating multiple candidate lane lines from multiple cascaded line segments; and a lane line merging module for merging multiple collinear candidate lane lines to obtain lane lines in the original image.

[0009] In a third aspect, an embodiment of the present application provides an autonomous mobile device, which includes a memory and a processor. The memory stores an application program, which is used to execute the method provided by the embodiment of the present application when called by the processor.

[0010] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having program code stored thereon, and the program code is used to enable the processor to execute the method provided by the embodiment of the present application when called by the processor.

[0011] The lane line detection method and device, autonomous mobile device, and storage medium provided in the embodiments of the present application perform illumination-invariant edge detection on images by calculating a gradient image based on the ab channel from the original image, extracting the original line segments on the lane line edges from the gradient image based on the ab channel, and detecting the lane lines in the original image based on the original line segments. For challenging images with large amounts of reflected light and low contrast, this lane line detection method can extract relatively accurate original line segments, thereby improving the accuracy and robustness of lane line detection. Furthermore, due to its low image quality requirements, this lane line detection method can achieve accurate and reliable lane line detection using a camera with a low mounting height, thereby reducing the height requirements for the autonomous mobile device itself and having broad practical application. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] To more clearly illustrate the technical solutions in the embodiments of this application, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments and drawings obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0013] FIG1 shows a structural block diagram of an autonomous mobile device provided by an embodiment of the present application;

[0014] FIG2 is a schematic diagram showing a flow chart of a lane line detection method provided in an embodiment of the present application;

[0015] FIG3 is a schematic diagram showing a flow chart of a lane detection method provided by another embodiment of the present application;

[0016] FIG4 shows a grayscale image of an original image provided by an exemplary embodiment of the present application;

[0017] FIG5 shows an image on the L channel corresponding to the original image corresponding to the grayscale image of FIG4 ;

[0018] FIG6 shows an image on a color channel corresponding to the original image corresponding to the grayscale image of FIG4 ;

[0019] FIG7 shows an image on the b color channel corresponding to the original image corresponding to the grayscale image of FIG4 ;

[0020] FIG8 shows a schematic diagram of lane lines provided by an exemplary embodiment of the present application;

[0021] FIG9 is a schematic diagram showing the result of adding the original image corresponding to the grayscale image of FIG4 after the original line segment extraction provided by an exemplary embodiment of the present application;

[0022] FIG10 shows a schematic diagram of a histogram provided by an exemplary embodiment of the present application;

[0023] FIG11 shows a schematic flow chart of step S130 in FIG3 of the present application;

[0024] FIG12 is a schematic diagram showing the process of step S140 in FIG3 of the present application;

[0025] FIG13 is a schematic diagram showing a color verification process provided by an exemplary embodiment of the present application;

[0026] FIG14 is a schematic diagram showing a color verification process provided by another exemplary embodiment of the present application;

[0027] FIG15 is a schematic diagram showing the process of step S150 in FIG3 of the present application;

[0028] FIG16 is a schematic flow chart of a lane detection method provided by another embodiment of the present application;

[0029] FIG17 is a schematic diagram showing a lane keeping process provided by an exemplary embodiment of the present application;

[0030] FIG18 is a schematic diagram showing a lane keeping process provided by another exemplary embodiment of the present application;

[0031] FIG19 shows a schematic diagram of a lane keeping process provided by yet another exemplary embodiment of the present application;

[0032] FIG20 shows a schematic diagram of a lane keeping process provided by yet another exemplary embodiment of the present application;

[0033] FIG21 shows a structural block diagram of a lane detection device provided in an embodiment of the present application;

[0034] FIG22 shows a structural block diagram of an autonomous mobile device provided in another embodiment of the present application. DETAILED DESCRIPTION

[0035] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.

[0036] The lane line detection method provided in the embodiment of the present application can be applied to the autonomous mobile device or lane line detection device provided in the embodiment of the present application. Next, the autonomous mobile device provided in an embodiment of the present application will be introduced.

[0037] Please refer to Figure 1, which shows a block diagram of the structure of an autonomous mobile device provided by an embodiment of the present application. The autonomous mobile device 100 may include, but is not limited to, an autonomous mobile robot (AMR) or an automated guided vehicle (AGV). The autonomous mobile device 100 includes an autonomous mobile device body 110, a camera 120, an illumination invariant line segment extraction module 130, an iterative collinear line segment detection module 140, a candidate lane line search module 150, a candidate lane line merging module 160, a lane line tracking module 170, and a lane keeping monitoring module 180.

[0038] The camera 120 is mounted on the autonomous mobile device body 110. The camera 120 can be a camera with a low mounting height and a small viewing angle. The image captured by the camera 120 usually has a large amount of reflected light and has a low contrast. The use of a camera with a low mounting height can achieve accurate and reliable lane line detection according to the lane line detection method provided in the embodiment of the present application, thereby reducing the height requirement for the autonomous mobile device body, making the lane line detection method provided in the embodiment of the present application have wide practicality. For example, when the autonomous mobile device 100 is located on the ground, the distance between the camera 120 and the ground is less than a specified height, and the orientation of the camera 120 forms an angle with the ground that is less than a specified angle. The specified height and the specified angle can be set according to the actual height of the autonomous mobile device body 110.

[0039] 2 , the autonomous mobile device 100 may sequentially perform an illumination invariant line segment extraction operation, an iterative cascaded collinear line segment operation, a candidate lane line search operation, a candidate lane line merging operation, a lane line tracking operation, and a lane keeping monitoring operation. For example, for each frame of image acquired by the camera 120 during the movement of the autonomous mobile device 100, the illumination invariant line segment extraction module 130 can be used to perform an illumination invariant line segment extraction operation, specifically, to use an illumination invariant line segment extraction method to extract line segments from the current frame image to obtain a large number of original line segments corresponding to the image, wherein the illumination invariant line segment extraction method refers to a line segment extraction method that uses an edge detection algorithm to extract line segments located on the lane line edge in the gradient image calculated from the a color channel and the b color channel corresponding to the original image, without relying on the brightness of the image; the iterative collinear line segment detection module 140 is used to perform an iterative cascade collinear line segment operation, specifically, to iteratively cascade collinear line segments in a large number of original line segments to obtain multiple cascaded line segments; the candidate lane line search module 150 is used to perform a candidate lane line search operation, specifically, to generate multiple candidate lane lines from multiple cascaded line segments. The candidate lane line merging module 160 is used to perform a collinear candidate lane line merging operation, specifically, to merge multiple collinear candidate lane lines to obtain the lane line corresponding to the current frame image. The lane line tracking module 170 is used to perform a lane line tracking operation, specifically, to obtain the lane line corresponding to the continuous multiple frames of images obtained according to the above operation, and to determine the reference lane line based on the lane line corresponding to the continuous multiple frames of images. The lane keeping monitoring module 180 is used to perform a lane keeping monitoring operation, specifically, to calculate the position and posture of the autonomous mobile device 100 relative to the reference lane line, and when the position and posture meet the warning conditions, to issue a warning message to prompt the autonomous mobile device 100 to deviate from the lane. In particular, for the working processes not described in detail in the above modules 130-180, please refer to the relevant parts of the method embodiment below.

[0040] Please refer to Figure 3, which shows a flow chart of a lane line detection method provided by an embodiment of the present application. The lane line detection method may include the following steps S110 to S150.

[0041] Step S110 , obtaining an original image captured by a camera of the autonomous mobile device, and calculating a gradient image based on ab channels according to the original image.

[0042] The original image captured by the camera is a true color image (RGB image), and the color of the RGB image is a combination of the three basic colors of red (Red), green (Green), and blue (Blue). In some embodiments, the camera is installed at a low angle and has a small viewing angle. The original image captured by the camera has a large amount of reflected light and has low contrast. In order to deal with situations where lighting conditions vary greatly and reflections are strong, and to prevent brightness from affecting the accuracy of subsequent extraction of original line segments, it is necessary to implement illumination invariant edge detection. In this embodiment of the present application, the original image is converted into a Lab image, and then based on the gradient image corresponding to the a color channel and the b color channel in the Lab image, the line segments on the lane line edge are extracted to improve the accuracy and robustness of the original line segment extraction, because the color difference of the lane line is the most prominent feature. The Lab image is composed of the brightness (Light, L for short) channel, the a color channel, and the b color channel. When operating the image, the brightness channel, a color channel, and b color channel of the Lab image can be operated separately. The channels do not affect each other, which can improve the robustness of the original line segment extraction. Moreover, by not considering the L channel, the influence of the image's light intensity changes on the detection results can be avoided, which can improve the accuracy of the original line segment extraction.

[0043] For example, the algorithm for converting RGB color space to XYZ color space in the Open Source Computer Vision Library (OpenCV) can be used to convert the original image from RGB color space to XYZ color space; the algorithm for converting XYZ color space to Lab color space in OpenCV can be used to convert the XYZ color space to Lab color space to obtain a Lab image.

[0044] After obtaining the Lab image, the graphics processing unit (GPU) acceleration method of the DiZenzo operator can be used to construct a gradient image based on the a color channel and the b color channel of the Lab image. The GPU acceleration method of the DiZenzo operator refers to adding the existing DiZenzo operator to the Compute Unified Device Architecture (CUDA) module in OpenCV, using CUDA to implement GPU acceleration of the DiZenzo operator, and calculating the gradient image of the a color channel and the b color channel. At present, the DiZenzo operator in OpenCV is usually implemented using a central processing unit (CPU). Since the amount of calculation required to construct the gradient image is large and the processing efficiency of the CPU is low, the GPU acceleration of the DiZenzo operator used in the embodiment of the present application can significantly improve the speed of constructing the gradient image compared to the existing CPU method using the DiZenzo operator, thereby improving the efficiency of the entire lane line detection.

[0045] For example, please refer to Figures 4-7. Figure 4 is a grayscale image of the original image. The original image corresponding to the grayscale image of Figure 4 is converted from RGB color space to Lab color space to obtain a Lab image. The L channel image in the Lab image is shown in Figure 5, the a color channel image in the Lab image is shown in Figure 6, and the b color channel image in the Lab image is shown in Figure 7.

[0046] Step S120 : extracting line segments on lane line edges from the gradient image based on the ab channel to obtain a plurality of original line segments.

[0047] Please refer to Figure 8, which shows three lane lines 1-3. The area between lane line 1 and lane line 2 forms a lane 4, and the area between lane line 2 and lane line 3 forms a lane 5. The black bold parts in lane lines 1-3 are the edges of lane lines 1-3 respectively.

[0048] A related lane line edge extraction operator can be used to extract line segments located on the lane line edge from the gradient image based on the ab channel to obtain multiple original line segments. The lane line edge extraction operator may include but is not limited to the Marr-Hildreth operator, the Canny operator, and the Laplacian operator, etc., and is not specifically limited here. For ease of understanding, please refer to Figures 4 and 9 for examples. For the original image corresponding to the grayscale image shown in Figure 4, the operations of step S110 and step S120 are performed to obtain multiple original line segments. The multiple original line segments are added to the original image to obtain the original image corresponding to the grayscale image shown in Figure 9, where the line segments indicated by the arrows are the original line segments.

[0049] Step S130 , using an iterative method, cascading the collinear line segments in the plurality of original line segments to obtain a plurality of cascaded line segments.

[0050] The lane lines in the original image captured by the camera have smaller widths the farther away from the autonomous mobile device, and larger widths the closer to the autonomous mobile device. To ensure that the widths of lane lines at far and near locations are consistent, the embodiment of the present application converts multiple original line segments into a ground coordinate system to obtain multiple original line segments in the ground coordinate system. The angles between the multiple original line segments in the ground coordinate system and the horizontal axis of the ground coordinate system can be calculated separately as the direction angles of each of the multiple original line segments.

[0051] The priority direction of the lane line can be determined based on the direction angles of multiple original line segments. For example, all original line segments can be scanned and a histogram as shown in Figure 10 can be constructed based on the direction angles of each of the multiple original line segments. The horizontal axis of the histogram represents the direction angle of the original line segment, and the vertical axis represents the number of original line segments. Please refer to Figure 9. After the original line segment extraction operation, the multiple original line segments actually contain short noise lines, such as the short noise lines in the lane indicated by the dotted line in Figure 9. The direction angles of the original line segments on the lane line are usually consistent with the direction angle of the actual lane line (the angle between the actual lane line direction and the horizontal axis of the ground coordinate system) and are more numerous, while the direction angles of the short noise lines are very messy and less. Therefore, the direction corresponding to the original line segment corresponding to the peak value in the histogram (usually the length direction of the original line segment) can be used as the priority direction of the lane line.

[0052] After determining the lane priority direction, a line segment with the lane priority direction or a line segment with a length greater than a length threshold can be selected from multiple original line segments as a priority line segment, and the priority flag (e.g., bprefered flag) of the priority line segment can be set to true. The length threshold is used to filter out the aforementioned short noise lines and can be determined based on the actual operating scenario of the autonomous mobile device.

[0053] The original line segments may be sorted according to their direction angles to obtain a sequence of the original line segments. For example, the original line segments may be sorted in ascending order of direction angles or in descending order of direction angles.

[0054] According to the sequence of original line segments, an iterative method can be used to sequentially cascade the collinear line segments in the multiple original line segments to obtain multiple cascaded line segments, wherein each iteration involves the cascade of two line segments, the two line segments include a first line segment and a second line segment, the bprefered tag of the first line segment must be true and not merged, and the first line segment involved in the first iteration is the first priority line segment in the sequence of original line segments. The directional angle difference between the second line segment and the first line segment is less than a preset angle, and the lateral distance is less than a preset lateral distance error, and the front and rear distances in the line segment direction are less than a preset interval. The preset angle, the preset lateral distance error, and the preset interval can all be determined based on the actual requirements for accuracy and the actual operation scenario, and are not specifically limited here. In the embodiment of the present application, the purpose of sorting is to facilitate the rapid finding of the first line segment and the second line segment required for each iteration during iteration, thereby improving the iteration efficiency. The bprefered tag is used to quickly find the first line segment involved in each iteration, which can significantly improve the iteration efficiency. It should be noted that the lateral distance in the embodiment of the present application refers to the distance in the normal direction of the line segment.

[0055] Specifically, during the first iteration, the first priority line segment is obtained from the sequence of original line segments as the first line segment involved in the first iteration. The bprefered flag of the first line segment must be true and not merged. A second line segment that meets the original line segment cascade condition is searched. For example, the next line segment of the first line segment can be obtained as the second line segment, and the direction angle difference between the second line segment and the first line segment is detected to be less than a preset angle. If the direction angle between the second line segment and the first line segment is less than the preset angle, the lateral distance between the second line segment and the first line segment is detected to be less than a preset lateral distance error. If the lateral distance between the second line segment and the first line segment is less than the preset lateral distance error, the front-to-back distance between the second line segment and the first line segment in the line segment direction is detected to be less than a preset interval. If the front-to-back distance between the second line segment and the first line segment in the line segment direction is less than the preset interval, it is determined that the second line segment that meets the original line segment cascade condition has been found. If any of the three judgments about the direction angle difference, lateral distance, and front-to-back distance between the second line segment and the first line segment is "no", it is determined that the second line segment has not been found.

[0056] If a second line segment is found, the first line segment and the second line segment are merged according to the number of original line segments respectively included in the first line segment and the second line segment to obtain the cascaded line segment obtained in this iteration. For example, if the number of original line segments included in the first line segment is greater than or equal to the number of original line segments included in the second line segment, the second line segment is merged into the first line segment to obtain the cascaded line segment obtained in this iteration; if the number of original line segments included in the first line segment is less than the number of original line segments included in the second line segment, the first line segment is merged into the second line segment to obtain the cascaded line segment obtained in this iteration. By setting the merging order of the first line segment and the second line segment, the efficiency of line segment merging can be improved; the bprefered flag of the cascaded line segment obtained in this iteration is set to true, and the cascaded line segment obtained in this iteration is used as the first line segment of the next iteration. Entering the next iteration, the step of "finding the second line segment that meets the cascade condition of the original line segment" is repeated until the iteration end condition is met and multiple cascaded line segments are obtained. The iteration end condition is that no new cascaded line segments are generated in the iteration loop.

[0057] If the second segment is not found, the bprefered flag of the first segment is set to false, and the next iteration is started. The original segment sequence is continued, and the first segment involved in the next iteration is re-determined from the priority segments until the iteration end condition is met, resulting in multiple cascaded segments. The sorting operation can implement backtracking traversal (look back). Backtracking traversal means that since the first segment in the iterative cascade may not be in front of the second segment in the sequence, after traversing the last segment in the sequence, it is possible to return to the first segment in the sequence and continue searching for the second segment until the iteration end condition is met.

[0058] For ease of understanding, an example shown in FIG. 11 is provided here to illustrate the iterative cascade logic of step S130 . Step S130 may include steps S131 to S1327 shown in FIG. 11 .

[0059] Step S131, determine the lane priority direction.

[0060] Step S132 : setting the bprefered flag of the original line segment with the lane priority direction and the long original line segment (the line segment length is greater than the length threshold) to true.

[0061] Step S133: sort the line segments according to their direction angles to form a sequence.

[0062] Step S134 : setting a new concatenation line segment flag (bnewConcatenation flag) to true to initialize the bnewConcatenation flag, indicating that a new concatenation line segment is generated, for starting the first round of iteration.

[0063] Step S135, determine whether the bnewConcatenation flag is true.

[0064] If the bnewConcatenation flag is true, indicating that a new concatenated line segment is generated, the process proceeds to step S136. If the bnewConcatenation flag is false, indicating that no new concatenated line segment is generated, the iterative process ends.

[0065] Step S136, set the bnewConcatenation flag to false, and set i=0.

[0066] Where i is the number in the sequence, which starts at 0. When i = 0, segment 0 actually refers to the first segment in the sequence. i is an integer zero or greater. The total number of segments in the sequence (also known as the length of the sequence) is the maximum value of i plus 1.

[0067] In the embodiments of this application, i, k (described below), and j (described below) are all integers zero or greater than zero, and are used to obtain the i+1, k+1, and j+1th original line segments (or cascaded line segments or candidate lane lines) in a sequence of original line segments (or a sequence of cascaded line segments or a sequence of candidate lane lines, respectively). For example, if i is 0, the first line segment in the sequence is obtained, and the same applies to j and k.

[0068] Step S137: Obtain the i-th line segment in the sequence as the first line segment.

[0069] Step S138: Determine whether the priority flag (bprefered flag) of the first line segment is true.

[0070] If the bprefered flag of the first line segment is true, proceed to step S139. If the bprefered flag of the first line segment is false, proceed to step S1321.

[0071] Step S139: determine whether the merge flag (bmerged flag) of the first line segment is true.

[0072] The bmerged flag indicates whether the first line segment has been merged. If the bmerged flag is true, the first line segment has been merged; if the bmerged flag is false, the first line segment has not been merged. If the bmerged flag of the first line segment is true, the process proceeds to step S1321. If the bmerged flag of the first line segment is false, the process proceeds to step S1310.

[0073] Step S1310, set the bprefered flag of the first line segment to false, set k=i+1, and set the forward flag (bforward flag) to true.

[0074] The bforward flag indicates that a segment has been traversed backwards along the sequence. k is the number in the sequence, starting at 0. When k = 0, segment 0 actually refers to the first segment in the sequence. k is an integer zero or greater. The total number of segments in the sequence (also known as the length of the sequence) is the maximum value of k plus 1.

[0075] Step S1311, determine whether k is less than the sequence length.

[0076] If k is less than the sequence length, the process proceeds to step S1323. If k is not greater than the sequence length, the process proceeds to step S1312.

[0077] Step S1312: Use the kth line segment as the second line segment.

[0078] Step S1313: Determine whether the bmerged flag of the first line segment is true.

[0079] If the bmerged flag of the first line segment is false, the process proceeds to step S1314. If the bmerged flag of the first line segment is true, the process proceeds to step S1325.

[0080] Step S1314: Calculate the direction angle difference between the first line segment and the second line segment, and determine whether the direction angle difference is greater than a preset angle.

[0081] If the direction angle difference is greater than the preset angle, the process proceeds to step S1325. If the direction angle difference is not greater than the preset angle, the process proceeds to step S1315.

[0082] Step S1315 : Calculate the transverse (ie, normal) distance between the first line segment and the second line segment, and determine whether the transverse distance between the first line segment and the second line segment is less than a preset transverse distance error.

[0083] If the lateral distance between the first line segment and the second line segment is less than the preset lateral distance error, the process proceeds to step S1316. If the lateral distance between the first line segment and the second line segment is not less than the preset lateral distance error, the process proceeds to step S1326.

[0084] Step S1316, calculate the interval distance between the first line segment and the second line segment in the sequence (i.e., the distance along the line segment direction), calculate the ratio of the interval distance and the total length, obtain the interval ratio, and determine whether the interval ratio is less than the preset ratio (the preset ratio can be determined based on the actual operating scenario of the autonomous mobile device).

[0085] If the spacing ratio is less than the preset ratio, the process proceeds to step S1317. If the spacing ratio is not less than the preset ratio, the process proceeds to step S1325.

[0086] Step S1317, determine whether the number of original line segments included in the first line segment (which can also be understood as the number of merged original line segments included in the first line segment) is greater than or equal to the number of original line segments included in the second line segment (which can also be understood as the number of merged original line segments included in the second line segment).

[0087] If the number of original line segments included in the first line segment is greater than or equal to the number of original line segments included in the second line segment, the process proceeds to step S1318. If the number of original line segments included in the first line segment is not greater than the number of original line segments included in the second line segment, the process proceeds to step S1319.

[0088] Step S1318: After merging the second line segment into the first line segment, set the bprefered flag of the first line segment to true, and set the bmerged flag of the second line segment to true.

[0089] Step S1319: After merging the first line segment into the second line segment, set the bprefered flag of the second line segment to true, and set the bmerged flag of the first line segment to true.

[0090] Step S1320: Set the bnewConcatenation flag to true. After executing step S1320, proceed to step S1325.

[0091] Step S1321, i=i+1.

[0092] Step S1322, determine whether i is less than the sequence length.

[0093] If i is less than the sequence length, the process returns to step S137. If i is not less than the sequence length, the process returns to step S135.

[0094] Step S1323, set k=0, and set the bforward flag to false.

[0095] Step S1324: Determine whether k is less than i. If k is less than i, return to step S1312. If k is not less than i, return to step S1321.

[0096] Step S1325, k=k+1.

[0097] Step S1326, determine whether the bforward flag is true.

[0098] If the advance flag is true, the process proceeds to step S1327. If the advance flag is false, the process returns to step S1324.

[0099] Step S1327, determine whether k is less than the sequence length.

[0100] If k is less than the sequence length, the process returns to step S1312. If k is not less than the sequence length, the process returns to step S1321.

[0101] Step S140: Generate multiple candidate lane lines from the multiple cascaded line segments.

[0102] Obtain the preset lane line width specification and the preset lane line color specification. The preset lane line width specification refers to the width specification determined from the standard lane line width specification based on the actual operating scenario of the autonomous mobile device, for example, 8-10 cm. The preset lane line width specification may include one or more width specifications. The preset lane line color specification refers to the color specification determined from the standard lane line color specification based on the actual operating scenario of the autonomous mobile device. The preset lane line color specification may include one or more color specifications, for example, red, yellow, white, and black. Both the preset lane line width specification and the preset lane line color specification are prior information pre-set based on the actual operating scenario of the autonomous mobile device. The candidate lane line generated based on the prior information is closer to the actual lane line, which can improve the lane line detection accuracy.

[0103] According to the preset lane line width specification, multiple line segment pairs are selected from multiple cascaded line segments. For example, the angles between the multiple cascaded line segments and the horizontal axis of the ground coordinate system can be calculated respectively as the direction angles of each of the multiple cascaded line segments. According to the direction angles of the multiple cascaded line segments, the multiple cascaded line segments are sorted to obtain a sequence of cascaded line segments, wherein the sorting order can be from small to large direction angles, or from large to small direction angles, and no specific limitation is made here. According to the sequence of cascaded line segments, multiple line segment pairs are selected from the multiple cascaded line segments in turn, and the direction angle difference of the two line segments included in each line segment pair is less than the preset angle and the lateral distance meets the preset lane line width specification. Wherein, the lateral distance meeting the preset lane line width specification refers to meeting one of the preset lane line width specifications.

[0104] The method for selecting multiple line segment pairs from multiple cascade line segments also uses an iterative approach. Each iteration involves matching two cascade line segments, and the two successfully matched cascade line segments form a line segment pair. Specifically, the two cascade line segments involved in each iteration are respectively referred to as the first cascade line segment and the second cascade line segment. The priority flag of the first cascade line segment must be true. If the first cascade line segment and the second cascade line segment successfully match, the first cascade line segment and the second cascade line segment in this iteration form a line segment pair.

[0105] For example, for each iteration, the first cascade line segment involved in the current iteration process is determined according to the sequence of cascade line segments, and the priority mark of the first cascade line segment is true. The second cascade line segment is searched in the sequence of cascade line segments. It is detected whether the first cascade line segment and the second cascade line segment meet the matching conditions. If the first cascade line segment and the second cascade line segment meet the matching conditions (that is, a matching second cascade line segment is found in the sequence of cascade line segments), the first cascade line segment and the second cascade line segment are matched successfully, and the first cascade line segment and the second cascade line segment form a line segment pair. If a matching second cascade line segment is not found in the sequence of cascade line segments, the priority mark of the first cascade line segment is set to false, and the first cascade line segment and the second cascade line segment of the next iteration are re-determined in the sequence of cascade line segments until all line segments in the cascade line segments are traversed.

[0106] Among them, the angular difference between the first cascade line segment and the second cascade line segment can be calculated to determine whether the angular difference between the first cascade line segment and the second cascade line segment is less than a preset angle. The lateral distance between the first cascade line segment and the second cascade line segment can be calculated to determine whether the lateral distance between the first cascade line segment and the second cascade line segment meets the preset lane line width specification. If the angular difference between the first cascade line segment and the second cascade line segment is less than the preset angle, and the lateral distance between the first cascade line segment and the second cascade line segment meets the preset lane line width specification (one of them), then it is determined that the first cascade line segment and the second cascade line segment meet the matching condition. If one of the above judgment results on the angular difference and the lateral distance between the first cascade line segment and the second cascade line segment is "no", then it is determined that the first cascade line segment and the second cascade line segment do not meet the matching condition.

[0107] Based on preset lane color specifications, a first color verification is performed on multiple line segment pairs. For each of the multiple line segment pairs, the color intensity distribution of the two original line segments included in the current line segment pair on color channels a and b is obtained. If the color intensity distribution of the two original line segments on color channels a and b both meet the preset lane color specifications, the current line segment pair is determined to have passed the first color verification. If the color intensity distribution of the two original line segments on color channels a and b does not meet the preset lane color specifications, the current line segment pair is determined to have failed the first color verification. The color intensity distribution meeting the preset lane color specifications means that the color intensity distribution meets one of the preset lane color specifications. Based on the multiple line segment pairs that pass the first color verification, multiple candidate lane lines are generated, with each line segment pair that passes the first color verification corresponding to a candidate lane line. For each line segment pair that passes the first color verification, a candidate lane line can be constructed based on the area between the current line segment pair.

[0108] For example, after a new line segment pair is generated in each iteration, the new line segment pair can be subjected to a first color verification. For example, the color intensity distribution of the two original line segments in the new line segment pair on the a color channel and the b color channel can be obtained; if the color intensity distribution of the two original line segments on the a color channel and the b color channel both meet the preset lane line color specifications (one of them), it is determined that the new line segment pair has passed the first color verification, and a candidate lane line is constructed based on the area between the new line segment pair that has passed the first color verification.

[0109] For ease of understanding, an example shown in FIG. 12 is provided here to illustrate the candidate lane line generation logic of step S140 . Step S140 may include steps S141 to S1418 shown in FIG. 11 .

[0110] Step S141 , sorting the multiple cascade line segments according to their direction angles to obtain a sequence of the cascade line segments.

[0111] Step S142: Calculate the length of the sequence of concatenated line segments and set i=0.

[0112] Where i is the number in the sequence, which starts at 0. When i = 0, segment 0 actually refers to the first segment in the sequence. i is an integer zero or greater. The total number of segments in the sequence (also known as the length of the sequence) is the maximum value of i plus 1.

[0113] Step S143: setting the i-th line segment in the sequence of cascaded line segments as the first cascaded line segment.

[0114] Step S144: determine whether the bprefered flag of the first cascade segment is true.

[0115] If the bprefered flag of the first concatenated line segment is true, the process proceeds to step S145. If the bprefered flag of the first concatenated line segment is false, the process proceeds to step S1414.

[0116] Step S145, k=i+1, set the bforward flag to true.

[0117] Where k is the number in the sequence, starting at 0. When k = 0, segment 0 actually refers to the first segment in the sequence. k is an integer zero or greater. The total number of segments in the sequence (also known as the length of the sequence) is the maximum value of k plus 1.

[0118] Step S146 , determining whether k is less than the sequence length of the concatenated line segments.

[0119] If k is less than the sequence length of the concatenated line segments, the process proceeds to step S148. If k is not less than the sequence length of the concatenated line segments, the process proceeds to step S147.

[0120] Step S147, set k=0, and set the bforward flag to false.

[0121] Step S148: setting the kth cascade line segment in the sequence of cascade line segments as the second cascade line segment.

[0122] Step S149: determine whether the matching flag (bmatchable flag) of the second cascade segment is true.

[0123] The bmatchable tag is used to indicate whether the second cascade line segment can be used to match the first cascade line segment.

[0124] If the bmatchable flag of the second concatenated line segment is true, it is determined that the second concatenated line segment can be used for subsequent matching operations with the first concatenated line segment, and the process proceeds to step S1410. If the bmatchable flag of the second concatenated line segment is false, it is determined that the second concatenated line segment cannot be used for subsequent matching operations with the first concatenated line segment, and the process proceeds to step S1416.

[0125] Step S1410 , calculating the direction angle difference between the first cascade line segment and the second cascade line segment, and determining whether the angle difference is less than a preset angle.

[0126] If the direction angle difference between the first cascade line segment and the second cascade line segment is less than the preset angle, the process proceeds to step S1411. If the direction angle difference between the first cascade line segment and the second cascade line segment is not less than the preset angle, the process proceeds to step S1416.

[0127] Step S1411 , determining whether the first cascade line segment and the second cascade line segment meet (one of) the preset lane line width specifications.

[0128] If the first cascade line segment and the second cascade line segment meet the preset lane line width specification (one of them), then go to step S1412; if the first cascade line segment and the second cascade line segment do not meet the preset lane line width specification (one of them), then go to step S1416.

[0129] Step S1412: Perform a first color verification on the first cascade line segment and the second cascade line segment to determine whether the color intensity distributions of the first cascade line segment and the second cascade line segment in the a color channel and the b color channel both meet (one of) the preset lane line color specifications.

[0130] If the color intensity distributions of the first and second concatenated line segments on color channels a and b both meet (one of) the preset lane color specifications, the process proceeds to step S1413. If the color intensity distributions of the first and second concatenated line segments on color channels a and b do not both meet (one of) the preset lane color specifications, the process proceeds to step S1416.

[0131] Step S1413: construct a candidate lane line based on the area between the first cascade line segment and the second cascade line segment.

[0132] Step S1414, i=i+1.

[0133] Step S1415 , determining whether i is less than the sequence length of the concatenated line segments.

[0134] If i is less than the sequence length of the concatenated line segments, the process returns to step S143. If i is not less than the sequence length of the concatenated line segments, multiple candidate lane lines are output.

[0135] Step S1416, k=k+1.

[0136] Step S1417: Determine whether the bforward flag of the kth cascade segment is true.

[0137] If the bforward flag of the kth cascade segment is true, the process returns to step S146 ; if the bforward flag of the kth cascade segment is false, the process proceeds to step S1418 .

[0138] Step S1418: Determine whether k is less than i. If k is less than i, return to step S148. If k is not less than i, return to step S1414.

[0139] Step S150: Merge multiple collinear candidate lane lines to obtain lane lines in the original image.

[0140] Collinear candidate lane lines among multiple lane line candidates are merged to obtain multiple long lane lines. Cascading collinear candidate lane lines to form long lane lines can avoid interference from short lane lines, thereby improving the accuracy of lane line detection. Short lane lines are caused by residual paint on outdated lane lines, or they are short lane lines that are meaningless for lane maintenance. For example, the angles between the multiple lane line candidates and the horizontal axis of the ground coordinate system can be calculated as the direction angles of the multiple lane line candidates; the multiple lane line candidates are sorted according to their direction angles to obtain a sequence of lane line candidates; and according to the sequence of lane line candidates, the collinear candidate lane lines among the multiple lane line candidates are merged in turn to obtain multiple long lane lines.

[0141] A second color verification is performed on the multiple long lane lines. The long lane lines that pass the second color verification are the lane lines in the original image. For example, for each of the multiple long lane lines, at least one pair of original line segments is found from the current long lane line. For each pair of original line segments, the color intensity distribution of the region between the two original line segments in color channels a and b is obtained. If the color intensity distribution meets (one of) the preset lane line color specifications, the current long lane line is determined to have passed the second color verification and is the lane line in the original image.

[0142] For example, to visualize the second color verification process, please refer to Figures 13 and 14. The original image involved in Figure 13 was acquired using a D435 camera, and the original image involved in Figure 14 was acquired using a Bosheng camera. An open source image labeling tool can be used to create a bounding box. The bounding box includes the pixel points of the candidate lane lines in multiple frames of original images (such as the Class I points shown in Figures 13 and 14), and the bounding box also includes background pixels (such as the Class II points in Figures 13 and 14). Based on the color of the pixel points of the candidate lane lines, the K-means clustering algorithm is used to cluster the color of the image points of each candidate lane line. As shown in Figures 13 and 14, the small circle C1 represents the cluster center, and the large circle C2 represents the cluster range for each cluster center. C1 and C2 can be determined according to user needs. After clustering, the color specification corresponding to each cluster range can be obtained. For each candidate lane line, it is possible to detect whether the color of each pixel within the current candidate lane line is consistent with the color specification corresponding to the cluster range where the current candidate lane line is located. If the colors are consistent, the pixel verification is determined to be successful; if they are not consistent, the pixel verification is determined to have failed. The number of pixels that have been detected (m) and the number of pixels that have been successfully verified (n) are calculated. If the number of pixels that have been detected is greater than a preset pixel number threshold (N, for example, N = 10) and the ratio of the number of pixels that have been successfully verified (n) to the number of pixels that have been detected (m) is greater than a preset probability, the current candidate lane line is determined to have passed the second color verification. If, after traversing all pixels within the current candidate lane line, the condition that "the number of pixels that have been detected is greater than a preset pixel number threshold (N, for example, N = 10) and the ratio of the number of pixels that have been successfully verified (n) to the number of pixels that have been detected (m) is greater than a preset probability" is still not met, the current candidate lane line is determined to have failed the second color verification.

[0143] For example, the merging of collinear candidate lane lines and the second color verification process can be implemented in an iterative manner. Please refer to Figure 15. Step S150 can include steps S151 to S1517.

[0144] Step S151: Determine whether the number of candidate lane lines in the sequence of candidate lane lines is greater than one.

[0145] If the number of the multiple candidate lane lines is greater than 1, the process proceeds to step S152. If the number of the multiple candidate lane lines is not greater than 1, the iteration ends.

[0146] Step S152: sorting the plurality of candidate lane lines according to their direction angles to form a sequence of candidate lane lines.

[0147] Step S153: determine whether the candidate merge flag (bfoundmerge flag) is true.

[0148] If the bfoundmerge flag is true, it is considered that a new candidate lane line has been merged, and the process proceeds to step S154. If the bfoundmerge flag is false, it is considered that no new candidate lane line has been merged, and the iteration ends.

[0149] Step S154, set i=0, and set the bfoundmerge flag to false.

[0150] Wherein, i is an integer zero or greater than zero, and is used to obtain the i+1th candidate lane line in the sequence of candidate lane lines.

[0151] Step S155: determine whether i is less than the length of the sequence of candidate lane lines.

[0152] If i is less than the length of the candidate lane line sequence, the process proceeds to step S156. If i is not less than the length of the candidate lane line sequence, the process proceeds to step S1516.

[0153] Step S156: Determine whether the i-th candidate lane line has been merged.

[0154] If the i-th candidate lane line has been merged, the process proceeds to step S1514. If the i-th candidate lane line has not been merged, the process proceeds to step S157.

[0155] Step S157, set j=i+1.

[0156] Wherein, j is an integer zero or greater than zero, and is used to obtain the i+1th candidate lane line in the sequence of candidate lane lines.

[0157] Step S158, determine whether j is equal to 1.

[0158] If j is equal to 1, go to step S1515; if j is not equal to 1, go to step S159.

[0159] Step S159: Determine whether the i-th candidate lane line and the j-th candidate lane line meet the merging condition.

[0160] If the i-th candidate lane line and the j-th candidate lane line meet the merging condition, the process proceeds to step S1510. If the i-th candidate lane line and the j-th candidate lane line do not meet the merging condition, the process proceeds to step S1512.

[0161] Step S1510: merge the j-th candidate lane line into the area after the i-th candidate lane line.

[0162] In an embodiment of the present application, setting the merging order of candidate lane lines can speed up iteration efficiency.

[0163] Step S1511: Set the bmerged flag of the j-th candidate lane line to true, and set the bfoundmerge flag to true. After step S1511 is executed, proceed to step S1515.

[0164] Step S1512, j=j+1.

[0165] Step S1513: Determine whether the difference between the number of candidate lane lines in the candidate lane line sequence and j is 1.

[0166] If the difference between the number of candidate lane lines in the candidate lane line sequence and j is 1, the process proceeds to step S1514. If the difference between the number of candidate lane lines in the candidate lane line sequence and j is not 1, the process returns to step S158.

[0167] Step S1514, set j=0.

[0168] Wherein, j is an integer of zero or greater than zero, and is used to obtain the j+1th candidate lane line in the sequence of candidate lane lines. After step S1514 is executed, the process returns to step S158.

[0169] Step S1515, i=i+1.

[0170] Step S1516: Remove the merged candidate lane lines from the sequence of candidate lane lines to obtain a new sequence of candidate lane lines.

[0171] Step S1517: Determine whether the number of candidate lane lines in the new candidate lane line sequence is less than 2.

[0172] If the number of candidate lane lines in the new candidate lane line sequence is less than 2, the iteration ends. If the number of candidate lane lines in the new candidate lane line sequence is not less than 2, the process returns to step S155.

[0173] In some embodiments, considering that some lane lines in actual operating scenarios may have patterns, pattern verification can be performed on lane lines that have passed the second color verification. The pattern on the lane lines that have passed the second color verification is detected to determine if it meets (one of) the preset pattern specifications. Lane lines that meet the preset pattern specifications are then used as lane lines in the original image. The preset pattern specifications are also a priori information that can be determined based on the actual operating scenario of the autonomous mobile device.

[0174] The lane detection method provided in the embodiments of the present application extracts lane line segments from the edge of a lane line using a gradient image calculated from the a and b color channels corresponding to the image captured by the camera. Lane lines in the original image are then detected based on the original line segments. This allows for illumination-invariant edge detection of the image. For challenging images with high levels of reflected light and low contrast, the lane detection method can extract relatively accurate original line segments, thereby improving the accuracy and robustness of lane detection. Furthermore, due to its low image quality requirements, the lane detection method can achieve accurate and reliable lane detection using a camera mounted at a low height, thereby reducing the height requirement for the autonomous mobile device itself and providing broad practical application. Furthermore, since the actual operating scenarios of autonomous mobile devices are typically specific and unlike cars, which operate in uncertain environments, prior information about the actual operating scenarios of the autonomous mobile device can be more easily acquired, such as preset lane width specifications, preset lane color specifications, and preset lane pattern specifications. This allows the lane detection method provided in the embodiments of the present application to fully utilize this prior information, improving the performance and accuracy of lane detection.

[0175] It should be noted that although the embodiments of the present application only mention the width specifications, color specifications and pattern specifications of the lane lines, it should be understood that if there are other specifications for the lane lines in the actual operation scenario, for example, lane line pattern specifications or lane line background color specifications, other specifications of the lane lines can be verified according to actual needs to improve the accuracy of lane line detection.

[0176] In order to keep the vehicle in the lane (i.e., between two lane lines) during stable driving, or to facilitate a safe and smooth transition from the starting lane to the target lane when switching lanes, in some embodiments, referring to FIG. 16 , after step S150 , the lane line detection method provided in the embodiments of the present application further includes steps S160 to S1110 of tracking the lane lines.

[0177] Step S160: Acquire lane lines in multiple consecutive frames of original images.

[0178] During the movement of the autonomous mobile device, each time it acquires a current frame of raw image output by the camera, it detects lane lines in the current frame of raw image according to the method described in steps S110 to S150. This allows the detection of lane lines in a continuous stream of raw images. When multiple consecutive frames of raw images are acquired, a reference lane line can be determined based on the multiple frames of raw image, and lane tracking can begin based on the reference lane line. The number of multiple frames of raw imagery can be set based on the actual operational scenario and the actual autonomous mobile device, and is not specifically limited here.

[0179] Step S170: Determine a reference lane line based on lane lines in multiple consecutive frames of original images.

[0180] A reference candidate lane line can be determined from lane lines in multiple frames of original imagery. For example, for each lane line in the multiple frames of original imagery, the length of the current lane line can be calculated. If the length of the current lane line is greater than a preset length, the current lane line is used as a reference candidate lane line. The preset length can be determined based on the actual operating scenario of the autonomous mobile device and is not specifically limited here.

[0181] If one or more reference lane candidate lines are obtained, it is possible to check whether the one or more reference lane candidate lines are detected in multiple frames of original imagery. The one or more reference lane candidate lines that are detected in multiple frames of original imagery are screened out, and the distances between the screened reference lane lines and the autonomous mobile device are calculated. The reference lane line is determined to be the reference lane line that is closest to the autonomous mobile device.

[0182] If one or more reference candidate lane lines are not obtained, or if no reference candidate lane lines are detected in multiple frames of original images, the process returns to step S110 to step S170 until one or more reference candidate lane lines are obtained.

[0183] By determining the lane lines in the continuous multi-frame original image as the reference lane lines, the lane lines with a length greater than a preset length, the closest distance to the autonomous mobile device, and the lane lines detected in the continuous multi-frame ab channel-based gradient image are selected. In this way, the lane lines with a longer length, a closer distance to the autonomous mobile device, and a relatively stable camera field of view can be selected as reference lane lines, which facilitates lane line tracking.

[0184] Step S180: performing lane line tracking based on the reference lane line.

[0185] During the movement of the autonomous mobile device, lane lines in the current frame raw image are obtained. For each lane line in the current frame raw image, whether the distance between the current lane line and the reference lane line meets the position requirement is detected. The position requirement may include that the lateral distance between the current lane line and the reference lane line is 0, in which case the current lane line and the reference lane line are collinear; whether the width of the current lane line is the same as the width of the reference lane line; and whether the color intensity distribution of pixels in the current lane line on the a color channel and the b color channel meets the preset lane line color specification. If the distance between the current lane line and the reference lane line meets the position requirement, the width of the current lane line is the same as the width of the reference lane line, and the color intensity distribution of pixels in the current lane line on the a color channel and the b color channel meets (one of) the preset lane line color specification, the current lane line is determined to match the reference lane line.

[0186] If there is a lane line in the original image of the current frame that matches the reference lane line, the lane line that matches the reference lane line will be used as the new reference lane line to update the reference lane line; if there is no lane line in the original image of the current frame that matches the reference lane line, the reference lane line will remain unchanged and will not be updated.

[0187] If a long update failure occurs after the last update of the reference lane line, for example, the time between the last update time and the current time is greater than a preset time, which can be determined based on the actual operating scenario of the autonomous mobile device, such as 5 seconds, or if the autonomous mobile device undergoes a large movement (for example, the autonomous mobile device crosses the lane, or the autonomous mobile device suddenly accelerates rapidly and cannot find a lane line that matches the reference lane line), the reference lane line will become invalid and it is necessary to redetermine the reference lane line according to the method of steps S110 to S170 to initialize the reference lane line and continue lane tracking based on the initialized reference lane line.

[0188] Step S190: Calculate the position and posture of the autonomous mobile device relative to the reference lane line.

[0189] Based on the reference lane line, a lane line coordinate system is constructed; the vehicle coordinate system and camera coordinate system of the autonomous mobile device are obtained; based on the lane line coordinate system, the vehicle coordinate system, and the camera coordinate system, a transformation matrix is ​​calculated between the vehicle coordinate system and the lane line coordinate system; based on the transformation matrix, the position and pose of the autonomous mobile device relative to the reference lane line are calculated. The lane line coordinate system can be constructed using the centerline of the reference lane line and the perpendicular bisector of the centerline. The origin of the lane line coordinate system is the center of the reference lane line. The vehicle coordinate system is a coordinate system with the center of the autonomous mobile device's rear axle as its origin, the orientation of the autonomous mobile device as its x-direction, and the left side of the autonomous mobile device as its y-direction.

[0190] For example, in the camera coordinate system xyz, the equation of the center line of the reference lane is: xsinθ-zcosθ+c=0, θ∈[-π / 2,π / 2)

[0191] Where x represents the coordinate of the centerline of the reference lane in the x-direction; z represents the coordinate of the centerline of the reference lane in the z-direction; θ represents the angle between the centerline of the reference lane and the x-direction; and c represents the distance from the camera to the origin of the centerline coordinate system.

[0192] Assume that the lane line coordinate system is x'-O'-y', and the coordinates of the camera in the camera coordinate system are (x c ,z c ,1), the camera’s pose in the centerline coordinate system is (t x ,0,α), the coordinates of the reference lane line in the lane line coordinate system are (xl,yl,1), then the transformation matrix between the lane line coordinate system and the camera coordinate system is:

[0193] We can get: 0 = x c cosα―z c sinα+t x 0=x c sinθ―z c cosθ+c

[0194] For any z c There are: x sinθ―ccosα+z c cos(θ+α)=0

[0195] You can get:

[0196] If θ>0, then t x =c

[0197] If θ≤0, then t x =-c

[0198] Assume that the reference pose of the camera coordinate system relative to the autonomous mobile device is (x b ,y b ,β b ), then:

[0199] Where L represents the lane line coordinate system; b represents the vehicle coordinate system; c represents the camera coordinate system, T Lb Refers to the transformation matrix from the lane line coordinate system to the vehicle body coordinate system; T LcRefers to the transformation matrix from the lane line coordinate system to the camera coordinate system; It refers to the inverse matrix of the transformation matrix from the vehicle coordinate system to the camera coordinate system, that is, the transformation matrix from the camera coordinate system to the vehicle coordinate system.

[0200] The position of the autonomous mobile device relative to the reference lane line (center line) is: (x b ,y b ,β b )=(t x ―x b cos(α―β b )+y b sin(α―β b ),―x b sin(α―β b )―y b cos(α―β b ),α― β b )

[0201] Step S1100 : When the position of the autonomous mobile device relative to the reference lane line satisfies a warning condition, a warning message is issued to prompt the autonomous mobile device to deviate from the lane.

[0202] If the horizontal coordinate of the autonomous mobile device's posture relative to the reference lane line is in the positive direction of the horizontal axis of the lane line coordinate system, and the angle of the posture is greater than a preset angle (e.g., 90 degrees), the posture is determined to indicate that the autonomous mobile device is approaching the reference lane line. Alternatively, if the horizontal coordinate of the posture is in the negative direction of the horizontal axis of the lane line coordinate system, and the angle of the posture is less than a preset angle (e.g., 90 degrees), the posture is determined to indicate that the autonomous mobile device is approaching the reference lane line. When the posture indicates that the autonomous mobile device is approaching the reference lane line, and the distance between the autonomous mobile device and the reference lane line is less than a preset minimum distance (set according to the actual operation scenario, such as 5 cm or 10 cm), the posture of the autonomous mobile device relative to the reference lane line meets the warning condition, and a warning message is issued to indicate that the autonomous mobile device has deviated from the lane.

[0203] For an example, please refer to Figure 17-20. In Figure 17-20, the coordinate system x'-O'-y' is the lane line coordinate system, and the coordinate system x c -O c -y c is the camera coordinate system, coordinate system x r -O r -y ris the vehicle coordinate system, and rectangle ABCD represents the autonomous mobile device. Lane keeping includes four situations as shown in Figures 17-20. In the situations shown in Figures 17 and 20, the autonomous mobile device is approaching the reference lane line. When |x b |Lane departure warning is issued when the preset minimum distance is less than the vehicle's lane departure distance.

[0204] Please refer to Figure 21, which shows a block diagram of the lane detection device provided in an embodiment of the present application. This lane detection device 200 can be applied to an autonomous mobile device. The lane detection device 200 includes an image acquisition module 210, a line segment extraction module 220, an iterative cascade module 230, a lane generation module 240, and a lane merging module 250.

[0205] The image acquisition module 210 is used to acquire the original image captured by the camera of the autonomous mobile device and calculate the gradient image based on the ab channel according to the original image. The specific working process of the image acquisition module 210 can be found in the above step S210.

[0206] The line segment extraction module 220 is used to extract line segments located on lane edges from the gradient image based on the ab channel to obtain multiple original line segments. The specific working process of the line segment extraction module 220 is shown in the above step S220.

[0207] The iterative concatenation module 230 is configured to concatenate collinear line segments in the plurality of original line segments using an iterative method to obtain a plurality of concatenated line segments. The specific working process of the iterative concatenation module 230 is described in the above step S230.

[0208] The lane line generation module 240 is used to generate multiple candidate lane lines from multiple cascaded line segments. The specific working process of the lane line generation module 240 can be found in the above step S240.

[0209] The lane line merging module 250 is used to merge multiple candidate lane lines to obtain lane lines in the original image. The specific working process of the lane line merging module 250 can be found in the above step S250.

[0210] In some embodiments, the lane line detection device 200 further includes a lane line tracking module, which is used to execute the above steps S260 to S280. For the specific working process of the lane line tracking module, please refer to the above step S250.

[0211] In some embodiments, the lane line detection device 200 further includes a lane keeping monitoring module, which is used to execute the above steps S290 and S2100. For the specific working process of the lane keeping monitoring module, please refer to the above steps S290 and S2100.

[0212] Those skilled in the art will clearly understand that the lane line detection device 200 provided in the embodiments of the present application can implement the lane line detection method provided in the embodiments of the present application. The specific working process of the above-mentioned device and modules can refer to the corresponding process of the method in the embodiments of the present application, and will not be repeated here.

[0213] In the embodiments provided in the present application, the coupling, direct coupling or communication connection between the modules shown or discussed may be an indirect coupling or communication coupling through some interfaces, devices or modules, and may be electrical, mechanical or other forms, and the embodiments of the present application do not impose specific limitations on this.

[0214] In addition, the functional modules in the embodiments of the present application may be integrated into a processing module, or each module may exist physically separately, or two or more modules may be integrated into a single module. The above-mentioned integrated modules may be implemented in the form of hardware or in the form of software functional modules.

[0215] Please refer to Figure 22, which shows a block diagram of the structure of an autonomous mobile device provided in an embodiment of the present application. The autonomous mobile device 300 may include a memory 310 and a processor 320. The memory 310 stores an application program, which is configured to execute the method provided in an embodiment of the present application when called by the processor 320.

[0216] The processor 320 may include one or more processing cores. The processor 320 utilizes various interfaces and circuits to connect various components within the autonomous mobile device 300. It is used to run or execute instructions, programs, code sets, or instruction sets stored in the memory 310, as well as to call and execute data stored in the memory 310, thereby executing various functions and processing data within the autonomous mobile device 300. The processor 320 may be implemented using at least one of the following hardware forms: a digital signal processing (DSP), a field-programmable gate array (FPGA), or a programmable logic array (PLA). The processor 320 may integrate one or a combination of a central processing unit (CPU), a graphics processing unit (GPU), and a modem. The CPU primarily processes the operating system, user interface, and application programs; the GPU is responsible for rendering and drawing display content; and the modem handles wireless communications. It is understood that the modem may not be integrated into the processor 320 but may be implemented separately via a communication chip.

[0217] The memory 310 may include random access memory (RAM) or read-only memory (ROM). The memory 310 may be used to store instructions, programs, codes, code sets, or instruction sets. The memory 310 may include a program storage area and a data storage area. The program storage area may store instructions for implementing an operating system, instructions for implementing at least one function, instructions for implementing the various method embodiments described above, and the like. The data storage area may store data created by the autonomous mobile device 300 during use.

[0218] The present application also provides a computer-readable storage medium having program code stored thereon, which is configured to execute the method provided by the present application when called by a processor. The computer-readable storage medium can be an electronic memory such as a flash memory, an electrically erasable programmable read-only memory (EEPROM), an erasable programmable read-only memory (EPROM), a hard disk, or a ROM.

[0219] In some embodiments, the computer-readable storage medium may include a non-transitory computer-readable storage medium (Non-TCRSM). The computer-readable storage medium has storage space for program codes that execute any of the method steps in the above method. These program codes can be read from or written into one or more computer program products. The program code can be compressed in an appropriate form.

[0220] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A lane line detection method, characterized in that: include: Acquire an original image captured by a camera of an autonomous mobile device, and calculate a gradient image based on ab channels according to the original image; From the gradient image based on the ab channel, extract the line segments located on the edge of the lane line to obtain multiple original line segments; Adopting an iterative method, cascading the collinear line segments among the multiple original line segments to obtain multiple cascaded line segments; Generate multiple candidate lane lines from multiple cascaded line segments; Merge multiple collinear candidate lane lines to obtain the lane lines in the original image.

2. The method according to claim 1, characterized in that The step of adopting an iterative method to cascade the collinear line segments in the plurality of original line segments to obtain a plurality of cascaded line segments comprises: The multiple original line segments are converted into a ground coordinate system, and the angles between the multiple original line segments and the horizontal axis of the ground coordinate system are calculated respectively as the direction angles of the multiple original line segments; Sort the multiple original line segments according to their direction angles to obtain a sequence of the original line segments; According to the sequence of the original line segments, the collinear line segments in the multiple original line segments are cascaded in sequence by adopting an iterative method to obtain multiple cascaded line segments.

3. The method according to claim 2, characterized in that Before the step of sequentially cascading the collinear line segments in the plurality of original line segments by an iterative method according to the sequence of the original line segments to obtain a plurality of cascaded line segments, the method further comprises: Determine the priority direction of the lane line according to the direction angles of multiple original line segments; From the multiple original line segments, an original line segment with a lane priority direction or an original line segment with a length greater than a length threshold is selected as a priority line segment, and a priority flag of the priority line segment is set to true; The step of sequentially cascading the collinear line segments in the plurality of original line segments by an iterative method according to the sequence of the original line segments to obtain the plurality of cascaded line segments comprises: According to the sequence of original line segments, an iterative method is used to sequentially cascade the colinear line segments in the plurality of original line segments to obtain a plurality of cascaded line segments, wherein each iteration involves the cascading of two line segments, the two line segments include a first line segment and a second line segment, and the priority mark of the first line segment is true and is not merged.

4. The method according to claim 3, characterized in that The step of sequentially cascading the collinear line segments in the plurality of original line segments by an iterative method according to the sequence of the original line segments to obtain the plurality of cascaded line segments comprises: In the first iteration, the first priority line segment is obtained from the sequence of original line segments as the first line segment involved in the first iteration; Find the second line segment that meets the cascade condition of the original line segment; If the second line segment is found, the first line segment and the second line segment are merged according to the number of original line segments respectively included in the first line segment and the second line segment to obtain the cascaded line segment obtained in this iteration; the priority flag of the cascaded line segment obtained in this iteration is set to true, and the cascaded line segment obtained in this iteration is used as the first line segment of the next iteration, and the next iteration is entered, and the step of searching for the second line segment that meets the original line segment cascade condition is repeatedly executed until the iteration end condition is met, and multiple cascaded line segments are obtained; If the second line segment is not found, the priority flag of the first line segment is set to false, and the next iteration is entered. The first line segment involved in the next iteration is re-determined from the priority line segments according to the sequence of the original line segments until the iteration end condition is met to obtain multiple cascaded line segments.

5. The method according to claim 4, characterized in that The step of merging the first line segment and the second line segment according to the number of original line segments respectively included in the first line segment and the second line segment to obtain the cascaded line segment obtained in this iteration comprises: If the number of original line segments included in the first line segment is greater than or equal to the number of original line segments included in the second line segment, the second line segment is merged into the first line segment to obtain a concatenated line segment obtained in this iteration; If the number of original line segments included in the first line segment is less than the number of original line segments included in the second line segment, the first line segment is merged into the second line segment to obtain the cascaded line segment obtained in this iteration.

6. The method according to claim 4 or 5, characterized in that: The step of searching for a second line segment that meets the original line segment cascade condition comprises: If the direction angle difference between the second line segment and the first line segment is less than the preset angle, and the lateral distance between the second line segment and the first line segment is less than the preset lane line width specification and less than the preset lateral distance error, and the front and rear distance between the second line segment and the first line segment along the line segment direction is less than the preset interval, it is determined that the second line segment that meets the original line segment cascade condition is found.

7. The method according to any one of claims 1 to 6, characterized in that: The step of generating a plurality of candidate lane lines from a plurality of cascaded line segments comprises: Get the preset lane line width specification and the preset lane line color specification; According to a preset lane width specification, a plurality of line segment pairs are selected from the plurality of cascaded line segments, each line segment pair including two cascaded line segments; According to the preset lane line color specification, the first color verification is performed on multiple line segment pairs; Based on the multiple line segment pairs that pass the first color verification, multiple candidate lane lines are generated, and each line segment pair that passes the first color verification corresponds to a candidate lane line.

8. The method according to claim 7, characterized in that The step of selecting a plurality of line segment pairs from a plurality of cascaded line segments according to a preset lane line width specification comprises: Calculate the angles between the multiple cascade line segments and the horizontal axis of the ground coordinate system respectively as the direction angles of the multiple cascade line segments; Sort the multiple cascade line segments according to their direction angles to obtain a sequence of the cascade line segments; According to the sequence of the cascaded line segments, a plurality of line segment pairs are selected in turn from the plurality of cascaded line segments, wherein the direction angle difference between the two line segments included in each line segment pair is less than a preset angle and the lateral distance meets the preset lane line width specification.

9. The method according to claim 7 or 8, characterized in that: The step of performing a first color verification on a plurality of line segment pairs according to a preset lane line color specification comprises: For each line segment pair among the plurality of line segment pairs, obtaining the color intensity distribution of two original line segments included in the current line segment pair on the a color channel and the b color channel; If the color intensity distribution meets the preset lane line color specification, it is determined that the current line segment pair passes the first color verification.

10. The method according to any one of claims 1 to 9, characterized in that: The step of merging a plurality of collinear candidate lane lines to obtain lane lines in the original image comprises: Merge the candidate lane lines that are on the same line among multiple candidate lane lines to obtain multiple long lane lines; A second color verification is performed on multiple long lane lines, and the long lane lines that pass the second color verification are the lane lines in the original image.

11. The method according to claim 10, characterized in that The step of merging the candidate lane lines that are on the same line among the multiple candidate lane lines to obtain multiple long lane lines includes: Calculate the angles between multiple candidate lane lines and the horizontal axis of the ground coordinate system as the angles of each candidate lane line. Angle of direction; Sort the multiple candidate lane lines according to their respective direction angles to obtain a sequence of the candidate lane lines; According to the sequence of candidate lane lines, collinear candidate lane lines among multiple candidate lane lines are merged in turn to obtain multiple long lane lines.

12. The method according to claim 10 or 11, characterized in that: The step of performing a second color verification on the plurality of long lane lines to obtain the lane lines in the original image comprises: For each of the plurality of long lane lines, find at least one pair of original line segments from the current long lane line; Obtaining color intensity distribution of a region between two line segments included in at least one pair of original line segments on color channel a and color channel b; If the color intensity distribution meets the preset lane line color specification, it is determined that the current long lane line passes the second color verification.

13. The method according to any one of claims 1 to 12, characterized in that: The step of obtaining the original image captured by the camera of the autonomous mobile device and calculating the gradient image based on the ab channel according to the a color channel and the b color channel corresponding to the original image comprises: Acquire a raw image captured by a camera of an autonomous mobile device; Convert the original image to Lab image; The GPU acceleration method of the DiZenzo operator is used to construct a gradient image according to the a color channel and b color channel of the Lab image, and a gradient image based on the ab channel is obtained.

14. The method according to claim 13, characterized in that When the autonomous mobile device is located on the ground, the distance between the camera and the ground is less than a specified height, and the direction of the camera forms an angle with the ground that is less than a specified angle.

15. The method according to any one of claims 1 to 14, characterized in that: After the step of merging a plurality of collinear candidate lane lines to obtain lane lines in the original image, the method further includes: Obtain lane lines in multiple consecutive frames of original images; Determine a reference lane line according to lane lines in multiple consecutive frames of original images; Lane line tracking is performed based on the reference lane line.

16. The method according to claim 15, characterized in that The step of determining the reference lane line according to the lane lines in the continuous multiple frames of original images comprises: From the lane lines in the continuous multiple frames of original images, the lane lines whose length is greater than the preset length and closest to the autonomous mobile device and are detected in the continuous multiple frames of gradient images based on the ab channel are determined as reference lane lines.

17. The method according to claim 15 or 16, characterized in that The step of tracking the lane line according to the reference lane line includes: During the movement of the autonomous mobile device, a lane line in the original image of the current frame is obtained; If there is a lane line in the original image of the current frame that matches the reference lane line, the lane line that matches the reference lane line is used as a new reference lane line to update the reference lane line; If there is no lane line matching the reference lane line in the original image of the current frame, the reference lane line is kept unchanged and is not updated.

18. The method according to claim 17, characterized in that After the step of obtaining the lane lines in the original image of the current frame, the method further includes: For each lane line in the lane line in the original image of the current frame, detect whether the distance between the current lane line and the reference lane line meets the position requirement; Check whether the width of the current lane line is the same as the width of the reference lane line; Check whether the color intensity distribution of the pixels in the current lane line on the a color channel and the b color channel meets the preset lane line color specification; If the distance between the current lane line and the reference lane line meets the position requirements, and the width of the current lane line is the same as the width of the reference lane line, and the color intensity distribution of the pixels in the current lane line on the a color channel and the b color channel meets the preset lane line color specifications, it is determined that the current lane line matches the reference lane line.

19. The method according to any one of claims 15 to 18, characterized in that: After the step of determining the reference lane line according to the lane lines in the continuous multiple frames of original images, the method further includes: Calculate the position and posture of the autonomous mobile device relative to the reference lane line; When the position and posture meet the warning condition, a warning message is issued to prompt the autonomous mobile device to deviate from the lane.

20. The method according to claim 19, characterized in that The step of calculating the position and posture of the autonomous mobile device relative to the reference lane line comprises: Construct a lane line coordinate system based on the reference lane line; Obtain the body coordinate system and camera coordinate system of the autonomous mobile device; According to the lane line coordinate system, the vehicle body coordinate system and the camera coordinate system, the transformation matrix between the vehicle body coordinate system and the lane line coordinate system is calculated; According to the transformation matrix, the position and posture of the autonomous mobile device relative to the reference lane line is calculated.

21. The method according to claim 19 or 20, characterized in that After the step of issuing a warning message for prompting the autonomous mobile device to deviate from the lane when the posture meets the warning condition, the method further includes: When the posture indicates that the autonomous mobile device is approaching a reference lane line and the distance between the autonomous mobile device and the reference lane line is less than a preset minimum distance, a warning message is issued to prompt the autonomous mobile device to deviate from the lane.

22. The method according to claim 20 or 21, characterized in that Before the step of issuing a warning message for prompting the autonomous mobile device to deviate from the lane when the posture meets the warning condition, the method further includes: If the horizontal coordinate in the posture is located in the positive direction of the horizontal axis of the lane line coordinate system, and the angle in the posture is greater than a preset angle, it is determined that the posture represents that the autonomous mobile device is approaching the reference lane line; or If the horizontal coordinate in the posture is located in the negative direction of the horizontal axis of the lane line coordinate system, and the angle in the posture is less than a preset angle, it is determined that the posture represents that the autonomous mobile device is approaching the reference lane line.

23. A lane line detection device, characterized in that: include: An image acquisition module is used to acquire an original image captured by a camera of an autonomous mobile device, and calculate a gradient image based on ab channels according to the original image; A line segment extraction module is used to extract line segments located on the edge of the lane line from the gradient image based on the ab channel to obtain multiple original line segments; Iterative cascading module, used to cascade the collinear line segments in multiple original line segments using an iterative method to obtain multiple cascaded line segments. Link segment; A lane line generation module, used to generate multiple candidate lane lines from multiple cascaded line segments; The lane line merging module is used to merge multiple co-linear candidate lane lines to obtain the lane lines in the original image.

24. An autonomous mobile device, characterized in that: include: A memory and a processor, wherein the memory stores an application program, and the application program is used to execute the method according to any one of claims 1 to 23 when called by the processor.

25. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program code, and the program code is used to execute the method according to any one of claims 1 to 23 when called by a processor.

Images