Method and device for detecting conical cylinder on driving lane, electronic equipment and storage medium

Abstract

The invention relates to a detection method and device for a conical cylinder on a driving lane, electronic equipment and a storage medium, and the method comprises the steps: collecting an image in the driving process of a vehicle in real time through a vehicle-mounted camera of the vehicle; acquiring a first pixel point corresponding to the lane line and a second pixel point corresponding to the cone from the image; according to the first distance between the first target pixel point and the second target pixel point, determining a second distance between the first target cone and the far end point of the lane line; the first target pixel point corresponds to a first target conical cylinder, and the first target conical cylinder is farther from the vehicle than the far-end point; according to the comparison result of the second distance and the lane width, whether a cone exists in front of the vehicle is determined. Through the method and the device, the problem that an unsafe strategy is decided by an automatic driving vehicle due to the fact that whether the cone at the turning is in the lane line or not cannot be identified by an automatic driving method in the prior art in the driving process of the vehicle is solved.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving technology, and in particular to a method and device for detecting cones on a driving lane, an electronic device and a storage medium. Background Art

[0002] The detection of the drivable area plays a very important role in autonomous driving. It is related to the route planning of the autonomous driving vehicle, ensuring safe navigation in various scenarios and the driving comfort of the passengers. In the existing technology, conventional autonomous driving methods are applicable to general scenarios, but the cones at some corners cannot be identified as being within the lane line. The existing autonomous driving methods cannot determine whether the cones are placed in the current vehicle driving path, which will cause the autonomous driving vehicle to decide on unsafe strategies. Summary of the invention

[0003] The present application provides a method and device for detecting cones on a driving lane, an electronic device and a storage medium to solve the problem in the prior art that the autonomous driving method cannot identify whether the cones at a turn are within the lane line during vehicle driving, causing the autonomous driving vehicle to decide on an unsafe strategy.

[0004] In a first aspect, the present application provides a method for detecting cones on a driving lane, comprising: collecting images of a vehicle in real time during driving by means of a vehicle-mounted camera; acquiring a first pixel corresponding to a lane line and a second pixel corresponding to a cone from the image; determining a second distance between a first target cone and a far end point of the lane line according to a first distance between the first target pixel and the second target pixel; wherein the second target pixel is the far end point of the lane line in the image; the first target pixel corresponds to the first target cone, and the first target cone is farther from the vehicle than the far end point; and determining whether there is a cone in front of the vehicle according to a comparison result of the second distance and the lane width.

[0005] Optionally, obtaining a first pixel point corresponding to the lane line and a second pixel point corresponding to the cone from the image includes: performing visual target detection on the image to obtain a visual target detection result, wherein the visual target detection result includes multiple detection frames, each detection frame includes the cone; performing semantic segmentation on the image after visual target detection to obtain a first pixel point corresponding to the lane line; cropping the detection frame to obtain a plurality of cropped images, sorting the plurality of cropped images, and selecting M cropped images with the highest sorting from the sorted results; the value of M is a positive integer; and inputting the M cropped images into a semantic segmentation model to obtain a second pixel point corresponding to the cone.

[0006] Optionally, determining the second distance between the first target cone and the far end point of the lane line based on the first distance between the first target pixel and the second target pixel includes: obtaining a depth value of the image through depth estimation, wherein the depth value represents an actual distance from the pixel in the image to the camera; determining a third distance from the first target pixel to the camera based on the depth value, and determining a fourth distance from the second target pixel to the camera; determining a first coordinate of the first target line pixel in the vehicle coordinate system based on the third distance and the fourth distance, and determining a second coordinate of the second target line pixel in the vehicle coordinate system; determining the first distance based on the first coordinate and the second coordinate, and determining the second distance in combination with the first distance and the camera extrinsic parameter.

[0007] Optionally, determining the third distance from the first target pixel to the camera according to the depth value includes: determining the fifth distance from each second pixel in the detection frame to the camera according to the depth value; obtaining the fifth distance values ​​corresponding to multiple second pixel points in the same detection frame, filtering out outliers and deleting them, and determining the average value of the filtered fifth distances as the third distance.

[0008] Optionally, before determining the second distance by combining the first distance with the camera extrinsic parameters, the method further includes: determining the height of the corresponding cone according to the third distance, the focal length of the camera, and the pixel height of the detection frame; and when the height of the cone is within a preset range, performing the operation of determining the second distance by combining the first distance with the camera extrinsic parameters.

[0009] Optionally, determining whether there is a cone ahead of the vehicle based on a comparison result of the second distance and the lane width includes: storing a first pixel point corresponding to the lane line in the image as a connected domain, and determining the current lane width based on the coordinates of each pixel point in the sorted connected domain and the coordinates of the midpoint of the front of the vehicle; when the second distance is greater than the lane width, determining that the cone is not in the current lane; when the second distance is less than or equal to the lane width, determining that the cone is in the current lane.

[0010] Optionally, the first pixel point corresponding to the lane line in the image is stored as a connected domain, including: marking two adjacent pixel points among the first pixel points corresponding to the lane line in the image as the same mark; storing the first pixel points corresponding to the same mark as the same connected domain, wherein different marks represent different lane line types corresponding to the pixel points, and the lane line types include left lane lines and right lane lines.

[0011] Optionally, the current lane width is determined according to the coordinates of each pixel point in the sorted connected domain and the coordinates of the midpoint of the front of the vehicle, including: collecting the coordinates of at least three pixel points at a longitudinal interval of N pixels from the connected domain corresponding to the left lane line, performing curve fitting, and obtaining a first third-order lane line curve; collecting the coordinates of at least three pixel points at a longitudinal interval of N pixels from the connected domain corresponding to the right lane line, performing curve fitting, and obtaining a second third-order lane line curve; and determining the sum of the distance from the midpoint of the front of the vehicle to the first third-order lane line curve and the distance from the midpoint of the front of the vehicle to the second third-order lane line curve as the current lane width.

[0012] In a second aspect, the present application provides a device for detecting cones on a driving lane, comprising: an acquisition device, used to acquire images of a vehicle in real time during driving through a vehicle-mounted camera; an acquisition module, used to acquire a first pixel point corresponding to a lane line and a second pixel point corresponding to a cone from the image; a determination module, used to determine a second distance between a first target cone and a far end point of the lane line based on a first distance between the first target pixel point and the second target pixel point; wherein the second target pixel point is the far end point of the lane line in the image; the first target pixel point corresponds to the first target cone, and the first target cone is farther away from the vehicle than the far end point; a detection module, used to determine whether there is a cone in front of the vehicle based on a comparison result of the second distance and the lane width.

[0013] Optionally, the acquisition module includes: a detection unit, used to perform visual target detection on the image to obtain a visual target detection result, wherein the visual target detection result includes multiple detection frames, and each detection frame includes the cone; a segmentation unit, used to perform semantic segmentation on the image after visual target detection to obtain a first pixel point corresponding to the lane line; a first processing unit, used to crop the detection frame to obtain a plurality of cropped images, sort the plurality of cropped images, and select M cropped images with the highest sorting from the sorted results; and a second processing unit, used to input the M cropped images into a semantic segmentation model to obtain a second pixel point corresponding to the cone.

[0014] In a third aspect, the present application provides an electronic device, comprising: at least one communication interface; at least one bus connected to the at least one communication interface; at least one processor connected to the at least one bus; and at least one memory connected to the at least one bus, wherein the processor is configured to: execute the method for detecting cones on a driving lane as described in any one of the embodiments of the first aspect above.

[0015] In a fourth aspect, the present application further provides a computer storage medium storing computer executable instructions, wherein the computer executable instructions are used to execute the method for detecting cones on a driving lane as described in any one of the embodiments of the first aspect above.

[0016] In the embodiment of the present application, after real-time image acquisition during vehicle driving is performed by the vehicle's onboard camera, if it is recognized that there are cones farther away from the far end point of the lane line, the distance between the cone and the far end point in the image can be determined in the actual environment, and then the distance between the cone and the far end point in the actual environment can be compared with the lane width to determine whether the cone is within the lane line. It can be seen that even if the lane line in the distance cannot be identified, and there are cones in the unidentifiable place, it is possible to determine whether the cone is within the lane line by the above-mentioned embodiment of the present application, so as to make an advance judgment in the process of autonomous driving, so that the vehicle can make safe driving decisions, thereby improving the safety of the autonomous driving process. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 A flowchart of a method for detecting cones on a driving lane provided in this application;

[0018] Figure 2 A schematic diagram of an image captured by a camera provided in an embodiment of the present application;

[0019] Figure 3 One of the optional flow charts of a method for detecting cones on a driving lane provided in an embodiment of the present application;

[0020] Figure 4 A schematic diagram of a cone in a detection frame in a visual target detection result provided in an embodiment of the present application;

[0021] Figure 5 A second optional flow chart of a method for detecting cones on a driving lane provided in an embodiment of the present application;

[0022] Figure 6 A third optional flow chart of a method for detecting cones on a driving lane provided in an embodiment of the present application;

[0023] Figure 7 A fourth optional flow chart of a method for detecting cones on a driving lane provided in an embodiment of the present application;

[0024] Figure 8 A schematic diagram of a drivable lane detection in a complex multi-cone scenario provided in an embodiment of the present application;

[0025] Fig. 9 A schematic diagram of the structure of a detection device for cones on a driving lane provided in this application;

[0026] Fig.10 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0027] The following will describe the embodiments of the present invention with reference to the accompanying drawings and preferred embodiments. Those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be understood that the preferred embodiments are only for illustrating the present invention, not for limiting the scope of protection of the present invention.

[0028] It should be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present invention, and thus the drawings only show components related to the present invention rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed arbitrarily, and the component layout may also be more complicated.

[0029] For ease of description, spatial relative terms may be used in the text to describe the relative position or movement of an element or feature as shown in the figure relative to another element or feature, such as "inside", "outside", "inner side", "outer side", "below", "below", "above", "above", "front", "back", etc. Such spatial relative terms are intended to include different orientations of the device in use or operation in addition to the orientation depicted in the figure. For example, if the device in the figure undergoes a position flip or a change in posture or a change in motion state, then these directional indications will also change accordingly, for example: an element described as "below other elements or features" or "below other elements or features" will subsequently be oriented as "above other elements or features" or "above other elements or features". Therefore, the example term "below..." can include both upper and lower orientations. The device can be oriented otherwise (rotated 90 degrees or in other directions) and the spatial relative descriptors used in the text are interpreted accordingly.

[0030] In order to solve the problem that the existing autonomous driving method cannot identify whether the cones at the corner are within the lane line during the vehicle driving process, causing the autonomous driving vehicle to decide on an unsafe strategy, the present application provides a method for detecting cones on the driving lane, such as Figure 1 As shown, the steps of the method include:

[0031] Step 101, using a vehicle-mounted camera to collect images of the vehicle in real time during its driving process;

[0032] In a specific example, Figure 2 The left side of the figure is an image captured in real time by the vehicle's onboard camera. From the image, we can see that there is a cone on the right lane line in the direction of the vehicle's travel. Figure 2 The right side of the image is a schematic diagram, where the solid lines are lane lines on both sides of the vehicle, the dotted lines are possible lane lines, and the solid origin is the cone in the image. Usually, the cones are placed on both sides of the lane.

[0033] Step 102, obtaining a first pixel point corresponding to the lane line and a second pixel point corresponding to the cone from the image;

[0034] Step 103, determining a second distance between the first target cone and the far end point of the lane line according to the first distance between the first target pixel and the second target pixel; wherein the second target pixel is the far end point of the lane line in the image; the first target pixel corresponds to the first target cone, and the first target cone is farther from the vehicle than the far end point;

[0035] A specific application scenario of the embodiment of the present application may be that a vehicle can recognize the cone at the far end but cannot recognize the lane line at the far end due to the ups and downs of the lane at a turn, such as Figure 2 As shown in the figure, the black solid origin is the identified cone, and the dotted line is the two possible lane line directions. Because the current road is curved and the bend is downhill, the lane line on the slope cannot be identified by the on-board camera. If the lane line on the slope is a light-colored dotted line, the cone at the far end is outside the lane line. If the lane line on the slope is a dark-colored dotted line, the cone at the far end is inside the lane line. In other words, the existing technology cannot identify whether the cone at the bend is inside the lane line, resulting in safety hazards in autonomous driving. Figure 2 For example, the first target pixel point may refer to Figure 2 The pixel corresponding to any of the three black solid origins on the upper left, and the second target pixel is Figure 2 The pixel point corresponding to the intersection of the solid line and the dotted line of the middle coordinate. For cones near the identifiable lane line, it is possible to directly determine from the image whether the cone is in the lane. Therefore, in the embodiment of the present application, the first target cone is farther from the vehicle than the far end point, which means that when the lane line at the far end is blocked for some reason and only the far end cone can be identified, the far end point is the end of the identifiable lane line, and the cone is a cone farther from the vehicle than the far end point.

[0036] Step 104: Determine whether there is a cone ahead of the vehicle based on the comparison result between the second distance and the lane width.

[0037] Through steps 101 to 104 of the above embodiment of the present application, after real-time image acquisition during the vehicle driving process is performed through the vehicle's onboard camera, if it is recognized that there is a cone farther away from the far end point of the lane line, the distance between the cone and the far end point in the image can be determined in the actual environment, and then the distance between the cone and the far end point in the actual environment is compared with the lane width to determine whether the cone is within the lane line. It can be seen that even if the lane line in the distance cannot be identified, and there is a cone in the unidentifiable place, it is possible to determine whether the cone is within the lane line through the above embodiment of the present application, so as to make an advance judgment during the automatic driving process, so that the vehicle can make a safe driving decision, thereby improving the safety of the automatic driving process.

[0038] In the embodiment of the present application, visual target level detection and semantic segmentation can be performed by a preset model. Therefore, for the method of obtaining the first pixel point corresponding to the lane line and the second pixel point corresponding to the cone from the image involved in the above step 102, as shown in FIG. Figure 3 As shown, the method may further include the following steps:

[0039] Step 301, performing visual target detection on the image to obtain a visual target detection result, wherein the visual target detection result includes a plurality of detection frames, each of which includes a cone;

[0040] Step 302, semantic segmentation is performed on the image after visual target detection to obtain a first pixel point corresponding to the lane line;

[0041] Step 303, cropping the detection frame to obtain a plurality of cropped images, sorting the plurality of cropped images, and selecting M cropped images with the highest sorting from the sorted results;

[0042] Step 304: input the M cropped images into a semantic segmentation model to obtain a second pixel point corresponding to the cone.

[0043] It should be noted that the whole image segmentation is not effective for small cones of distant targets, so the cone detection frame is cropped to obtain multiple cone cropped images, and then the image sizes are sorted from small to large, and the first M images are selected. Usually, since the cropped images are small, they need to be edge-filled to obtain N x N images, which are input into the semantic segmentation model (N is selected according to the selected segmentation model).

[0044] In an embodiment of the present application, visual target detection can be performed based on a target detection model, such as a YOLO model. If a cone is detected in an image through the YOLO model, a detection frame of the cone can be obtained, such as Figure 4As shown, if no cone is detected, the next frame of the image is detected. In addition, in the embodiment of the present application, semantic segmentation can be performed by a segmentation model, such as a U-Net model, through which lane lines and cones can be segmented from the image. Further, in a specific example, the first pixel point belonging to the lane line and the second pixel point of the cone can be marked with corresponding labels, such as label_lane for the first pixel point and label_cone for the second pixel point, and then different labels can be used to identify which pixels in the image are lane lines and which pixels are cones.

[0045] In the embodiment of the present application, the distance between each pixel in the image can be determined according to the image depth value and the corresponding coordinate system. Therefore, for the method of determining the second distance between the first target cone and the far end point of the lane line according to the first distance between the first target pixel and the second target pixel in the above step 103, as shown in FIG. Figure 5 As shown, the following steps may further be included:

[0046] Step 501, obtaining a depth value of an image by depth estimation, wherein the depth value represents an actual distance from a pixel in the image to a camera;

[0047] In a specific example of an embodiment of the present application, depth estimation refers to a method of inferring the distance from the surface of an object in a scene to a camera from a two-dimensional image. It is usually achieved by creating a depth value in a depth map, and the depth value indicates that the value of each pixel represents the actual distance from the surface of the object corresponding to the point in the image to the camera. Further, in an embodiment of the present application, the depth value of the image can be determined based on the Monodepth2 model of a deep network architecture for self-supervised learning, that is, the image is input into the depth estimation model to obtain the depth value of the image. In an embodiment of the present application, the depth value represents the distance between the pixel and the camera, that is, the depth value reflects the distance between the target object (lane line or cone) and the measuring equipment (on-board camera). Moreover, in an embodiment of the present application, the image is input into the Monodepth2 model to obtain the depth map of the image, and in the depth image, the value of each pixel represents the distance from the point to the camera, that is, the depth value. Therefore, the geometric depth information of the scene (cone or lane) can be truly and accurately reflected.

[0048] Step 502, determining a third distance from the first target pixel to the camera according to the depth value, and determining a fourth distance from the second target pixel to the camera;

[0049] In the embodiment of the present application, since the depth value refers to the distance from the pixel to the camera, the third distance from the first target pixel to the camera and the fourth distance from the second target pixel to the camera can be determined by the depth value.

[0050] Step 503, determining a first coordinate of a pixel point on the first target line in the vehicle coordinate system according to the third distance and the fourth distance, and determining a second coordinate of a pixel point on the second target line in the vehicle coordinate system;

[0051] Step 504, determining a first distance according to the first coordinate and the second coordinate, and determining a second distance by combining the first distance with the camera extrinsic parameter.

[0052] It can be seen that in the embodiment of the present application, a vehicle coordinate system is established with the vehicle as the center in the image, and the orientation of each pixel in the vehicle coordinate system can be further determined, and then the coordinates of each pixel in the vehicle coordinate system can be determined according to the depth value. Therefore, after determining the distance between the first target cone and the far end point of the lane line in the vehicle coordinate system, the distance between the first target cone and the far end point of the lane line in the actual environment can be calculated through the external parameter ranging of the on-board camera, and the distance includes both the distance between the first target cone and the far end point of the right lane line and the distance between the first target cone and the far end point of the left lane line.

[0053] It should be noted that extrinsic ranging means that when installing the camera, the camera parameters are combined and the camera is calibrated on the road surface. For example, marking lines are placed 10m, 20m, and nm in front of the camera. Then, in the current camera image, the pixel points corresponding to the positions of these marking lines have corresponding physical position distances, thereby calculating the coordinate conversion relationship between the image and the physical world. In other words, through camera extrinsic ranging, the distance relationship between each pixel in the image can be converted into the coordinate system of the real physical world.

[0054] In the embodiment of the present application, each detection frame may include multiple pixel points belonging to the cone. In order to make the distance between the cone and the camera more accurate, the distances between each pixel point and the camera are summed and then averaged in the present application. This can reduce the error of the third distance. Therefore, for the method of determining the third distance from the first target pixel point to the camera according to the depth value involved in the above step 502, as shown in FIG. Figure 6 As shown, the following steps may further be included:

[0055] Step 601, determining a fifth distance between each second pixel point in the detection frame and the camera according to the depth value;

[0056] Step 602, obtaining fifth distance values ​​corresponding to a plurality of second pixel points in the same detection frame, filtering outliers and deleting them, and determining an average value of the filtered fifth distances as the third distance.

[0057] It should be noted that before determining the third distance, the incorrectly detected detection frame needs to be deleted, that is, if there is no cone in the current detection frame, the incorrectly detected detection frame needs to be deleted. Then, the distance is measured for multiple pixel points belonging to the cone in each detection frame, that is, the distance is measured according to the depth value. Multiple distance measurement values ​​can be obtained for each detection frame. The distance from the cone in the detection frame to the camera can be obtained by averaging the multiple distance measurement values, thereby reducing the error caused by the distance from any pixel in the detection frame to the camera as the third distance. In addition, the screening and deletion of outliers can further improve the accuracy of the final third distance, so that cone detection can be performed more accurately in the future.

[0058] In the embodiment of the present application, the height of the corresponding cone can be determined according to the third distance, the focal length of the camera, and the pixel height of the detection frame. If the height of the cone is within the preset range, the second distance can be determined by combining the first distance and the camera external parameter. If the height of the cone is not within the preset range, it indicates that the current detection may be wrong, and the cone currently detected is not the actual cone, and the cone can be ignored.

[0059] In the specific example, the third distance is recorded as x, the focal length of the camera is known, the pixel height of the detection frame is known, and the actual height of the cone is obtained according to the pinhole imaging principle: The actual height of the cone is between 0.8±0.3m. If H is not within this range, it indicates that the detected cone is unavailable.

[0060] Generally, the lane width is the same whether it is in a straight line or at a curve. Therefore, in the embodiment of the present application, the lane width can be determined by the distance between the pixel points on the lane lines on both sides, and the lane width can be determined by the distance between two parallel pixel points on a straight line. Therefore, the method of determining whether there is a cone ahead of the vehicle according to the comparison result between the second distance and the lane width involved in 104 of the embodiment of the present application is as follows: Figure 7 As shown, the following method steps may be included:

[0061] Step 701, storing the first pixel point corresponding to the lane line in the image as a connected domain, and determining the current lane width according to the coordinates of each pixel point in the sorted connected domain and the coordinates of the midpoint of the vehicle head;

[0062] For this, store the connected components of the pixel points corresponding to the lane lines. The specific method can be as follows: Traverse the pixel points starting from the upper left point of the image. If the label of this point is label_lane and there is no marked rle value in the upper left of this point, then mark it as x and move to the next pixel; if the label of this point is label_lane and there is a marked rle x0 value in the upper left of this point, then reuse the x0 value. If the rle values in the upper left are different (x0 < x1), then reuse the smaller x0 value, and at the same time remember that x1 inherits x0 and move to the next pixel; if the label of this point is not label_lane, directly move to the next pixel until the entire image is traversed; In the second pass, still traverse from the upper left corner, check the x value of the rle of the pixel. If there is inheritance, update it; if there is no inheritance, keep it until the entire image is traversed. In this way, the pixel points with the same x value are a connected component of a lane line, and all the lane connected components form rle_lanes. The connected components store the pixel point information with the same label.

[0063] Under normal circumstances, lane lines are respectively set on both sides of the lane. Therefore, the connected components of the pixel points corresponding to the lane lines also respectively correspond to the left lane line and the right lane line.

[0064] Therefore, for the method of storing the first pixel points corresponding to the lane lines in the image as connected components involved in the above 701, it can further include:

[0065] Step 11, mark two adjacent pixel points among the first pixel points corresponding to the lane lines in the image with the same mark;

[0066] Step 12, store the first pixel points corresponding to the same mark as the same connected component, where different marks represent different lane line types corresponding to the pixel points, and the lane line types include the left lane line and the right lane line.

[0067] It can be seen that in the embodiments of the present application, the pixel points corresponding to the left lane line can be stored as the same connected component, and the pixel points corresponding to the right lane line can be stored as the same connected component, so that the corresponding pixel points can be determined from different connected components to determine the lane width when determining the lane width subsequently.

[0068] In the embodiments of the present application, in order to make the determination of the lane width more accurate, the method of determining the current lane width according to the coordinates of each pixel point of the sorted connected component and the coordinates of the midpoint of the vehicle's head involved in the above step 701 can further include:

[0069] Step 21, collect the coordinates of at least three pixel points at longitudinal intervals of N pixels from the connected component corresponding to the left lane line for curve fitting to obtain the first third-order lane line curve;

[0070] Step 22, collecting coordinates of at least three pixel points spaced N pixels apart in the longitudinal direction from the connected domain corresponding to the right lane line, and performing curve fitting to obtain a second curve;

[0071] In a specific embodiment, the value of N can be determined according to actual needs, for example, the value of N is 10, 8, 15, etc.

[0072] Step 23, the sum of the distance from the midpoint of the front of the vehicle to the first third-order lane line curve and the distance from the midpoint of the front of the vehicle to the second third-order lane line curve is determined as the current lane width.

[0073] It should be noted that the third-order lane curve corresponds to a third-order curve equation, and the third-order curve equation refers to an equation with a maximum degree of 3 containing unknowns, which can usually be written as ax 3 +bx 2 +cx+d=0. Therefore, in a specific example, the corresponding third-order curve can be determined based on the third-order curve equation. In the embodiment of the present application, the corresponding third-order lane curve is determined, because in the process of real-time image acquisition by the camera, the collected image may deviate from the real physical environment due to some reasons. For example, the lane line in the real physical environment is a straight line, but the lane line obtained after the camera acquires the image is tortuous, that is, it is inconsistent with the real physical environment. At this time, the pixel points in the sorted connected domain can be used for curve fitting to obtain a new curve corresponding to the lane line. The curve is more in line with the lane line in the real physical environment. Therefore, the distance between the curve obtained after fitting and the front of the vehicle can be used to obtain the lane width, which is more accurate than the lane width directly obtained based on the distance between the pixel points in the image and the front of the vehicle.

[0074] Step 702, when the second distance is greater than the lane width, determining that the cone is not in the current lane;

[0075] Step 703: When the second distance is less than or equal to the lane width, determine that the cone is in the current lane.

[0076] In a specific example, the second distance includes the distance between the first target cone and the far end point of the left lane line, and the distance between the first target cone and the far end point of the right lane line. As long as one of the distances is greater than the lane width, it indicates that the cone is not in the current lane. If the distance between the first target cone and the far end point of the left lane line is greater than the lane width, or the distance between the first target cone and the far end point of the right lane line is greater than the lane width, it indicates that the cone is not in the lane line. At this time, the automatic driving can continue to drive according to the current strategy without braking or decelerating due to the cone. If the distance between the first target cone and the far end point of the left lane line is less than or equal to the lane width, and the distance between the first target cone and the far end point of the right lane line is less than or equal to the lane width, the cone is in the lane line. At this time, the automatic driving needs to perform corresponding safety strategy control, such as braking, decelerating or changing lanes, because the cone is in the lane line.

[0077] The present application is explained below in conjunction with a specific implementation of an embodiment of the present application. The specific implementation provides a method for detecting drivable lanes in a complex multi-cone scene. Figure 8 , the method comprises the following steps:

[0078] Step 801, perform visual target detection on the picture collected in real time by the vehicle-mounted camera, and select the common YOLO model as the target detection model. If a cone is detected, obtain the detection frame of n cones. If no cone is detected, directly execute the next frame and skip the subsequent steps.

[0079] Step 802, perform visual semantic segmentation, and select the common U-Net model as the segmentation model. Obtain the semantic segmentation results of cones and lanes respectively. Label the pixels belonging to cones with label_cone, and label the pixels belonging to lanes with label_lane. Then store the pixels of label_lane as connected domains rle_lanes, remove rle_lanes that are too small or have abnormal shapes, and reorder them according to pixel coordinates. Find the two closest lane connected domains based on the position of the midpoint of the vehicle head, and store them as right_lane and left_lane respectively.

[0080] Step 803: select the Monodepth2 model based on self-supervised learning deep network architecture, which has good performance in occlusion scenes and motion scenes. Input the image into the depth estimation model to obtain the depth value of the image.

[0081] Step 804, traverse the cone detection box in step 801, if there is no label_cone pixel in the box, delete the cone detection box. Each cone detection box and its corresponding label_cone pixel in the box are regarded as a valid target object, such as Figure 2 Combined with the depth map, the distance measurement value of the label_cone pixel in each valid target object is calculated, the abnormal distance measurement data is eliminated, and the average value is calculated as the distance measurement value of the valid target object.

[0082] Step 805, obtain right_lane and left_lane in step 802, select the starting endpoint of the connected domain of the left and right lane lines, and through the depth value, know the current lane width and the distance value of the far end point of the left and right lane lines in the vehicle coordinate system.

[0083] Step 806, determining the relationship between the cone and the far end point of the lane line;

[0084] If the distance between the cone and the far end of the lane line is greater than the lane width, it is judged that the vehicle is not in the lane, and the path planning of the autonomous vehicle avoids braking and deceleration. If the distance between the cone and the far end of the lane line is less than or equal to the lane width, it is judged that the vehicle is in the lane or on the lane line, and the path planning of the autonomous vehicle should perform obstacle avoidance.

[0085] Corresponding to the above-mentioned method embodiments of the present application, the following is an apparatus embodiment of the present application, which can be used to execute the method embodiments of the present application.

[0086] Fig. 9 This is a schematic diagram of the structure of a detection device for cones on a driving lane provided in an embodiment of the present application. The device can be implemented as part or all of an electronic device through software, hardware, or a combination of both. Fig. 9 As shown, the device comprises:

[0087] The acquisition module 902 is used to acquire images of the vehicle in real time during driving through the vehicle's onboard camera;

[0088] An acquisition module 904 is used to acquire a first pixel point corresponding to the lane line and a second pixel point corresponding to the cone from the image;

[0089] A determination module 906 is used to determine a second distance between the first target cone and the far end point of the lane line according to a first distance between the first target pixel point and the second target pixel point; wherein the second target pixel point is the far end point of the lane line in the image; the first target pixel point corresponds to the first target cone, and the first target cone is farther from the vehicle than the far end point;

[0090] The detection module 908 is used to determine whether there is a cone ahead of the vehicle according to the comparison result between the second distance and the lane width.

[0091] Through the device of the embodiment of the present application, after the vehicle's on-board camera collects real-time images during the vehicle's driving process, if it is recognized that there are cones farther away from the far end point of the lane line, the distance between the cone and the far end point in the image can be determined in the actual environment, and then the distance between the cone and the far end point in the actual environment can be compared with the lane width to determine whether the cone is within the lane line. It can be seen that even if the lane line in the distance cannot be identified, and there are cones in the unidentifiable place, it is possible to determine whether the cone is within the lane line through the above-mentioned embodiment of the present application, so as to make an advance judgment in the process of automatic driving, so that the vehicle can make safe driving decisions, thereby improving the safety of the automatic driving process.

[0092] In an optional implementation of the embodiment of the present application, the acquisition module 904 in the embodiment of the present application may further include: a detection unit, used to perform visual target detection on the image to obtain a visual target detection result, wherein the visual target detection result includes multiple detection frames, and each detection frame includes a cone; a segmentation unit, used to perform semantic segmentation on the image after visual target detection to obtain a first pixel point corresponding to the lane line and a second pixel point corresponding to the cone; a first processing unit, used to crop the detection frame to obtain multiple cropped images, sort the multiple cropped images, and select M cropped images with the highest sorting from the sorted results; a second processing unit, used to input the M cropped images into a semantic segmentation model to obtain a second pixel point corresponding to the cone.

[0093] In an optional implementation manner of the embodiment of the present application, the determination module 906 in the embodiment of the present application may further include: an acquisition unit, used to acquire a depth value of an image through depth estimation, wherein the depth value represents the actual distance from a pixel in the image to the camera; a first determination unit, used to determine a third distance from a first target pixel to the camera according to the depth value, and to determine a fourth distance from a second target pixel to the camera; a second determination unit, used to determine a first coordinate of a first target line pixel in a vehicle coordinate system according to the third distance and the fourth distance, and to determine a second coordinate of a second target line pixel in a vehicle coordinate system; a third determination unit, used to determine a first distance according to the first coordinate and the second coordinate, and to determine a second distance in combination with the first distance and a camera external parameter.

[0094] In an optional implementation manner of an embodiment of the present application, the first determination unit in the embodiment of the present application may further include: a first determination subunit, used to determine the fifth distance of each second pixel point in the detection frame from the camera based on the depth value; a second determination subunit, used to obtain the fifth distance values ​​corresponding to multiple second pixel points in the same detection frame, filter out outliers and delete them, and determine the average value of the filtered fifth distances as the third distance.

[0095] In an optional implementation manner of the embodiment of the present application, the detection module in the embodiment of the present application may further include: a third processing unit, used to store the first pixel point corresponding to the lane line in the image as a connected domain, and determine the current lane width according to the coordinates of each pixel point in the sorted connected domain and the coordinates of the midpoint of the front of the vehicle; a fourth determination unit, used to determine that the cone is not in the current lane when the second distance is greater than the lane width; a fifth determination unit, used to determine that the cone is in the current lane when the second distance is less than or equal to the lane width.

[0096] In an optional implementation manner of an embodiment of the present application, the third processing unit in the embodiment of the present application may further include: a marking subunit, used to mark two adjacent pixel points in the first pixel point corresponding to the lane line in the image as the same mark; a first processing subunit, used to store the first pixel point corresponding to the same mark as the same connected domain, wherein different marks represent different lane line types corresponding to the pixel points, and the lane line types include left lane lines and right lane lines.

[0097] In an optional implementation manner of the embodiment of the present application, the third processing unit in the embodiment of the present application may further include: a second processing sub-unit, used to collect coordinates of at least three pixel points with a longitudinal interval of N pixels from the connected domain corresponding to the left lane line to perform curve fitting to obtain a first third-order lane line curve; a third processing sub-unit, used to collect coordinates of at least three pixel points with a longitudinal interval of N pixels from the connected domain corresponding to the right lane line to perform curve fitting to obtain a second third-order lane line curve; a third determination sub-unit, used to determine the sum of the distance from the midpoint of the front of the vehicle to the first curve and the distance from the midpoint of the front of the vehicle to the second curve as the current lane width.

[0098] like Fig.10 As shown, an embodiment of the present application provides an electronic device, including a processor 1001, a communication interface 1002, a memory 1003 and a communication bus 1004, wherein the processor 1001, the communication interface 1002, and the memory 1003 communicate with each other through the communication bus 1004.

[0099] Memory 1003, used for storing computer programs;

[0100] In one embodiment of the present application, the processor 1001 is used to execute the program stored in the memory 1003 to implement the steps of a method for detecting a cone on a driving lane provided by any of the aforementioned method embodiments, for example, including:

[0101] S1, collects images of the vehicle in real time during driving through the vehicle's onboard camera;

[0102] S2, obtaining a first pixel point corresponding to the lane line and a second pixel point corresponding to the cone from the image;

[0103] S3, determining a second distance between the first target cone and the far end point of the lane line according to the first distance between the first target pixel point and the second target pixel point; wherein the second target pixel point is the far end point of the lane line in the image; the first target pixel point corresponds to the first target cone, and the first target cone is farther from the vehicle than the far end point;

[0104] S4, determining whether there is a cone ahead of the vehicle based on a comparison result between the second distance and the lane width.

[0105] The role played by the electronic device in the embodiment of the present application in executing the above steps S1 to S4 is similar to the role played by the method for detecting cones on the driving lane in the above embodiment of the present application, and will not be repeated here.

[0106] An embodiment of the present application further provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of a method for detecting cones on a driving lane provided in any of the aforementioned method embodiments are implemented.

[0107] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0108] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a general hardware platform, and of course, by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0109] It should be understood that the terms used in the text are only for the purpose of describing specific example embodiments, and are not intended to be limiting. Unless the context clearly indicates otherwise, the singular forms "one", "an" and "said" as used in the text may also be meant to include plural forms. The terms "include", "comprise", "contain", and "have" are inclusive, and therefore specify the existence of stated features, steps, operations, elements and / or parts, but do not exclude the existence or addition of one or more other features, steps, operations, elements, parts, and / or combinations thereof. The method steps, processes, and operations described herein are not interpreted as necessarily requiring them to be performed in the specific order described or illustrated, unless the execution order is clearly indicated. It should also be understood that additional or alternative steps may be used.

[0110] The foregoing is merely a specific embodiment of the present invention, which enables those skilled in the art to understand or implement the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A method for detecting cones on a driving lane, characterized in that: include: The images of the vehicle in motion are collected in real time through the vehicle's onboard camera; Acquire a first pixel point corresponding to the lane line and a second pixel point corresponding to the cone from the image; Determine a second distance between the first target cone and the far end point of the lane line according to a first distance between the first target pixel point and the second target pixel point; wherein the second target pixel point is the far end point of the lane line in the image; the first target pixel point corresponds to the first target cone, and the first target cone is farther from the vehicle than the far end point; Whether there is a cone ahead of the vehicle is determined according to a comparison result between the second distance and the lane width.

2. The method according to claim 1, characterized in that Acquiring a first pixel point corresponding to the lane line and a second pixel point corresponding to the cone from the image, including: Performing visual target detection on the image to obtain a visual target detection result, wherein the visual target detection result includes a plurality of detection frames, each of which includes the cone; Performing semantic segmentation on the image after visual target detection to obtain a first pixel point corresponding to the lane line; The detection frame is cropped to obtain a plurality of cropped images, the plurality of cropped images are sorted, and M cropped images with the highest sorting order are selected from the sorted results; the value of M is a positive integer; The M cropped images are input into a semantic segmentation model to obtain a second pixel point corresponding to the cone.

3. The method according to claim 2, characterized in that Determining a second distance between the first target cone and the far end point of the lane line according to a first distance between the first target pixel point and the second target pixel point includes: Acquire a depth value of the image by depth estimation, wherein the depth value represents an actual distance from a pixel point in the image to a camera; Determine a third distance from a first target pixel to the camera according to the depth value, and determine a fourth distance from a second target pixel to the camera; Determine a first coordinate of the pixel point of the first target line in the vehicle coordinate system according to the third distance and the fourth distance, and determine a second coordinate of the pixel point of the second target line in the vehicle coordinate system; The first distance is determined according to the first coordinate and the second coordinate, and the second distance is determined by combining the first distance and a camera extrinsic parameter.

4. The method according to claim 3, characterized in that Determining a third distance from the first target pixel to the camera according to the depth value includes: Determine a fifth distance between each second pixel point in the detection frame and the camera according to the depth value; The fifth distance values ​​corresponding to multiple second pixel points in the same detection frame are obtained, outliers are screened out and deleted, and the average value of the screened fifth distances is determined as the third distance.

5. The method according to claim 4, characterized in that Before determining the second distance by combining the first distance with the camera extrinsic parameter, the method further includes: Determine the height of the corresponding cone according to the third distance, the focal length of the camera, and the pixel height of the detection frame; When the height of the cone is within a preset range, an operation of determining the second distance by combining the first distance with the camera external parameter is performed.

6. The method according to claim 1, characterized in that Determining whether there is a cone ahead of the vehicle according to a comparison result between the second distance and the lane width includes: The first pixel point corresponding to the lane line in the image is stored as a connected domain, and the current lane width is determined according to the coordinates of each pixel point in the sorted connected domain and the coordinates of the midpoint of the front of the vehicle; When the second distance is greater than the lane width, determining that the cone is not in the current lane; When the second distance is less than or equal to the lane width, it is determined that the cone is in the current lane.

7. The method according to claim 6, characterized in that Storing the first pixel point corresponding to the lane line in the image as a connected domain includes: Marking two adjacent pixel points in the first pixel point corresponding to the lane line in the image as the same mark; The first pixel points corresponding to the same mark are stored as the same connected domain, wherein different marks represent different lane line types corresponding to the pixel points, and the lane line types include left lane lines and right lane lines.

8. The method according to claim 7, characterized in that Determining the current lane width according to the coordinates of each pixel point of the sorted connected domain and the coordinates of the midpoint of the vehicle head includes: The coordinates of at least three pixel points are collected at intervals of N pixels in the longitudinal direction from the connected domain corresponding to the left lane line to perform curve fitting to obtain a first third-order lane line curve; The coordinates of at least three pixel points are collected at intervals of N pixels in the longitudinal direction from the connected domain corresponding to the right lane line to perform curve fitting, and a second third-order lane line curve is obtained; The sum of the distance from the midpoint of the front of the vehicle to the first third-order lane line curve and the distance from the midpoint of the front of the vehicle to the second third-order lane line curve is determined as the current lane width.

9. A device for detecting cones on a driving lane, characterized in that: include: The acquisition module is used to acquire images of the vehicle in real time during its driving process through the vehicle's onboard camera; An acquisition module, used to acquire a first pixel point corresponding to the lane line and a second pixel point corresponding to the cone from the image; A determination module, configured to determine a second distance between a first target cone and a far end point of a lane line according to a first distance between a first target pixel point and a second target pixel point; wherein the second target pixel point is a far end point of the lane line in the image; the first target pixel point corresponds to the first target cone, and the first target cone is farther from the vehicle than the far end point; The detection module is used to determine whether there is a cone ahead of the vehicle according to the comparison result of the second distance and the lane width.

10. The device according to claim 9, characterized in that The acquisition module comprises: A detection unit, configured to perform visual target detection on the image to obtain a visual target detection result, wherein the visual target detection result includes a plurality of detection frames, each of which includes the cone; A segmentation unit, used to perform semantic segmentation on the image after visual target detection to obtain a first pixel point corresponding to the lane line; A first processing unit is used to crop the detection frame to obtain a plurality of cropped images, sort the plurality of cropped images, and select M cropped images with top sorting from the sorted results; The second processing unit is used to input the M cropped images into a semantic segmentation model to obtain a second pixel point corresponding to the cone.

11. An electronic device, comprising: at least one communication interface; at least one bus connected to the at least one communication interface; at least one processor coupled to the at least one bus; At least one memory connected to the at least one bus, wherein the processor is configured to execute the method for detecting cones on a driving lane according to any one of claims 1-8.

12. A computer storage medium storing computer executable instructions, wherein the computer executable instructions are used to execute the method for detecting cones on a driving lane according to any one of claims 1 to 8.

Images